Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Data and Retrieval Architecture for AI Systems.
Context
A B2B expense-management product has 600 customer companies (tenants). Each tenant uploads its own travel policy, approval rules and past finance tickets. An in-product assistant answers employees' questions, such as "What is our hotel limit in Berlin?", from that tenant's own documents.
All tenants share one vector index of 12 million chunks. Twenty large tenants hold about 400,000 chunks each (8 million in total, 67% of the index). The other 580 share the remaining 4 million, about 6,900 each. Within a tenant, some documents, such as executive travel rules, are restricted to named groups.
Problem statement
The first version searched the whole index and then dropped rows that did not belong to the caller's tenant, in application code. An internal red team tried 120 cross-tenant attempts (5 synthetic tenants, 20 ordered pairs, 6 entry points) and 4 of them (3.3%) returned another tenant's snippet. Separately, small tenants often got "I don't know" for questions their documents answered.
Measurable goal: 0 leaks in 120 attempts, on every release, and an answer for small tenants whenever their own data contains one.
Constraints
- Fixed: no tenant may see another tenant's data, including a snippet, a citation title or a cached answer.
- Fixed: group restrictions inside a tenant must hold.
- Fixed: tenants have contractual rights over where their data sits; a few may demand a dedicated index.
- Negotiable: the partitioning scheme and the cost of one extra index per large tenant.
- Limit: the platform team can operate a handful of index types, not 600 separate stacks.
Why the traditional approach fell short
The product already has keyword search per tenant, and it is safe because every query runs in a tenant-scoped database session. It cannot answer a paraphrased policy question. A shared index with an application filter is the simplest way to add semantic search, but it makes isolation depend on every code path remembering to filter. Two paths did not: a "related questions" endpoint and a response cache keyed by question text only. Isolation that depends on developer discipline fails at the first new feature.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Keep keyword search, no AI | Already tenant-scoped; no new risk | Kept as the fallback and for tenants who opt out of the assistant; does not handle paraphrases |
| B. Shared index, filter in the application after search | The first version | Rejected: leaks through unfiltered paths and loses recall |
| C. Filter enforced below the application, partitions for large tenants | The Retriever service owns the index credentials and builds the tenant filter from the signed token | Accepted |
The default is C. A separate stack per tenant (a silo) is stronger still, and stays available for tenants whose contract demands it.
Solution architecture
Figure 1. Where the tenant boundary is enforced
How to read this: the request carries no tenant field the caller can edit, and the only route to the index passes through the Retriever.
Figure 2. One question with the defence-in-depth check
How to read this: the filter is applied inside the search, and the check after it is a second lock that should never fire.
How AI solved it, step by step
- An employee of tenant A asks about hotel limits. The gateway verifies the session token and extracts the tenant id and user id from its signed claims.
- The Orchestrator forwards the question with that tenant scope. The request body has no tenant field to trust.
- The Retriever asks the Policy engine for the user's groups, then builds the filter: tenant equals A, and the chunk's group is public to the tenant or in the user's groups.
- The Retriever routes by tenant. A large tenant has its own partition. A small tenant's rows sit in the shared partition, where the filter selects about 6,900 rows and an exact search over them is cheap.
- The search returns chunks. The Retriever checks that every row is tagged with tenant A. Decision: any foreign row is dropped, the result is empty and security is alerted.
- The Model gateway drafts an answer from the passages, and the answer cites the tenant's own documents.
- The response cache key includes the tenant id, and the Trace store records tenant, user, filter and the ids returned.
Under the hood
The key step is where the filter runs. Pre-filtering restricts the candidates before ranking, so only the tenant's rows compete. Post-filtering ranks everything and then removes rows. The second loses recall as well as safety.
The pgvector README documents this for approximate indexes: "With approximate indexes, filtering is applied after the index is scanned", and, as its example, "If a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average." Its listed remedies are iterative index scans, partial indexes for a few distinct values, and partitioning for many. Applying the same arithmetic with an illustrative candidate list of 40, a large tenant (400,000 of 12 million rows, 3.3%) averages 40 x 0.033 = 1.3 rows. A typical small tenant (6,897 rows, 0.0575%) averages 40 x 0.000575 = 0.023 rows. This assumes the tenant is uncorrelated with similarity, which is roughly true for generic questions such as "expense limits" that every tenant's documents match.
Hence the tiering. Large tenants get partitions. Small tenants' scope is small enough for an exact scan: 6,897 vectors x 1,024 dimensions is 7,062,528 multiply-adds per query, so no approximate index is needed for them. Details of pre-filtering, post-filtering and partitioning are in Under the hood: Access control in retrieval.
What went wrong / failure modes
- A new endpoint skips the filter. Mitigation: application code has no index credentials; only the Retriever does, and it requires a tenant scope argument to run any query.
- Cache without the tenant in the key. A cached answer for tenant A is served to B. Mitigation: tenant id is part of every cache key, and cross-tenant tests cover the cache.
- Tenant taken from the request. A caller edits a field. Mitigation: the tenant comes only from verified token claims.
- Group restriction ignored. A user sees a restricted document of their own tenant. Mitigation: the group filter is part of the same pre-filter and is tested like the tenant filter.
- Partition mis-routing. A tenant is moved to a new partition and its old rows remain. Mitigation: a migration step deletes and verifies, and the post-search check would flag a foreign row.
Results
These are illustrative targets, not measurements. The red team's baseline was 4 leaks in 120 attempts (3.3%); the target is 0 in 120 on every release, and any non-zero result blocks the release. Recall for small tenants moves from an expected 0.023 own rows in a 40-row candidate list to a search over all of the tenant's rows. The cost is one partition per large tenant: 20 extra indexes to operate.
Tradeoffs and when NOT to use this approach
- Partitions cost operations effort. With 20 tenants, a shared index with a strict pre-filter is probably enough.
- An exact scan stops being cheap as a tenant grows; move it to its own partition when it passes an agreed size.
- A shared index leaves no physical boundary. Where the contract or the regulator demands one, use a silo per tenant and accept the cost.
- Do not rely on the post-search check as the control. It is a tripwire.
Design rules extracted
- Enforce tenant and group filters below the application, in the one service that holds the index credentials.
- Take the tenant from verified claims, never from the request body.
- Filter before ranking; post-filtering both leaks and loses recall.
- Include the tenant in every cache key.
- Test isolation adversarially: every tenant pair, every entry point, zero leaks as a release gate.
Assumptions
All numbers are illustrative.
- 600 tenants: 20 large at 400,000 chunks (20 x 400,000 = 8,000,000) and 580 small sharing 4,000,000 chunks (4,000,000 / 580 = about 6,897 each). Total 12,000,000.
- Shares: 400,000 / 12,000,000 = 3.33%; 6,897 / 12,000,000 = 0.0575%. Expected rows: 40 x 0.0333 = 1.33; 40 x 0.000575 = 0.023.
- The candidate list of 40 follows the default in the pgvector README example (https://github.com/pgvector/pgvector, read 2026-10-21); the post-filter arithmetic is this page's own application of that example, not a measurement. It assumes tenant is uncorrelated with similarity.
- Embeddings have 1,024 dimensions: 6,897 x 1,024 = 7,062,528 multiply-adds.
- Red team: 5 tenants give 5 x 4 = 20 ordered pairs, times 6 entry points = 120 attempts; 4 / 120 = 3.3%.
- The product, the tenants and the red-team result are invented.
Last verified: 2026-10-22