What you'll learn: why the access filter belongs inside the retrieval query, how a post-filter starves the result list and leaks signals (measured on the capstone), how to extend the same predicate to tenants, and why authorization is never the model's job.
At a glance
Figure 1. The same predicate before and after the search
How to read this: both lanes use the identical predicate; only the moment it runs differs, and that moment decides how many results survive and what the scores reveal.
- Run the access predicate inside the query, before scoring.
- A post-filter returns fewer than k results, and its scores and "removed" count describe hidden content.
- Role and tenant come from a verified token, never from the request body or the model.
- "Do not reveal partner prices" in a prompt is not access control; a context that never contained them is.
- Fail closed: an empty tenant or an unknown owner returns nothing.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply keeps a partner programme guide with partner-only prices next to its public datasheets. The first version of the assistant searched everything, then removed partner chunks from the result list.
In testing, questions about a partner product returned one or two sources instead of six, and a debug panel said "5 results hidden". The sales director asks whether this is safe. All numbers here come from the capstone's data and are illustrative.
What you will be able to do
- Explain the difference between a post-filter and an in-query filter and state the cost of each in one table.
- Measure starvation and score leakage of a post-filter with a test against the capstone's chunks.
- Write a tenant predicate that fails closed and prove it with a test.
- Show with a test that hidden text never reaches the model, whatever the user types.
Prerequisites
Search: keyword, vector, hybrid, and choosing a store for the BM25 index, and Prompts and untrusted input. The design view is in Data and retrieval architecture.
Core ideas
One predicate, two places
| Post-filter | In-query filter | |
|---|---|---|
| When the predicate runs | after the top k is chosen | before any chunk is scored |
| Scores | computed with hidden chunks in the statistics | computed over allowed chunks only |
| What the caller can infer | how many results were removed | nothing about hidden chunks |
| How it fails | silently, when someone forgets the second step | loudly, when the argument is missing |
The capstone's predicate is chunk_filter: it checks the visibility class and optionally drops superseded documents or restricts to given SKUs.
Verified: capstone/fde-product-discovery-copilot/app/retrieval/filters.py
def chunk_filter(
chunks: Mapping[str, Chunk],
visibilities: Collection[str],
exclude_superseded: bool = False,
skus: Collection[str] | None = None,
) -> Callable[[str], bool]:
wanted = set(skus) if skus else None
def allowed(chunk_ref: str) -> bool:
c = chunks[chunk_ref]
return (
c.visibility in visibilities
and not (exclude_superseded and c.status == "superseded")
and (wanted is None or bool(wanted.intersection(c.skus)))
)
return allowedThe index takes this function as where and applies it first, so its statistics cover allowed chunks only.
Verified: capstone/fde-product-discovery-copilot/app/retrieval/keyword.py
allowed = [i for i, item_id in enumerate(self.ids) if where is None or where(item_id)]
if not allowed:
return []
allowed_set = set(allowed)
avg_len = (sum(self.length[i] for i in allowed) / len(allowed)) or 1.0What a post-filter costs, measured
m3_l4_filtering.py builds the tempting version: search every chunk, take the top 6, then apply the same chunk_filter.
Verified: capstone/fde-product-discovery-copilot/lessons/M3-L4/m3_l4_filtering.py
def post_filter_search(
store: EvidenceStore, index: BM25Index, query: str, visibilities: Collection[str], top_k: int = 6
) -> PostFilterResult:
"""Take the top k of ALL chunks, then drop the ones the caller may not see."""
allowed = chunk_filter(store.chunks, visibilities)
ranked = index.search(query, top_k)
kept = [(ref, score) for ref, score in ranked if allowed(ref)]
return PostFilterResult([r for r, _ in kept], [s for _, s in kept], removed=len(ranked) - len(kept))Four public-user queries against the capstone's 174 chunks (5 are partner-only):
| Query | Post-filter returns | In-query returns |
|---|---|---|
| partner price LUM-VS-900 | 1 | 6 |
| partner programme vibration sensor | 1 | 6 |
| white label discount partner programme | 1 | 2 |
| Which vibration sensor has the widest frequency range you can sell us? | 6 | 6 |
Row 3 is short in-query because few allowed chunks match. Row 1 is pinned by this test:
Verified: capstone/fde-product-discovery-copilot/lessons/M3-L4/test_m3_l4_filtering.py
def test_post_filter_starves_the_result_list(svc: Services) -> None:
post = post_filter_search(svc.store, full_index(svc.store), QUERY, PUBLIC, top_k=6)
refs, _ = in_query_search(svc.store, QUERY, PUBLIC, top_k=6)
assert len(post.refs) == 1 and post.removed == 5 # asked for 6, got 1
assert len(refs) == 6 # the same predicate inside the query still fills all 6 slotsPredict first. For the first query, the post-filter reports removed = 5. What does that number tell a caller who is not allowed to see the partner guide?
Answer
It equals the number of chunks in the hidden guide (test_removed_count_equals_the_hidden_document_size: 5 removed, 5 partner-only). The caller learns that a five-chunk document matches, which is already a disclosure. Never show such a count.
A quieter signal: the visible chunk faq-licensing#7 scores 4.309962 under the post-filter and 4.384224 in-query, because hidden chunks changed the term statistics. In-query scores do not change when hidden documents are added (test_in_query_scores_do_not_change_when_hidden_documents_change).