Data and Retrieval Architecture for AI Systems

Figure 6. HNSW search: greedy descent through the layers

How to read this: the query moves down the participants from left to right in time, each note is what happens on that layer, and only layer 0 uses the larger candidate list ef.

Memory. The paper gives the average per-vector cost of the graph as (Mmax0 + mL × Mmax) × bytes per link, with 4-byte links up to about four billion vectors, and reports typical M values between 6 and 48, or about 60 to 450 bytes per object excluding the vector data. Worked example (illustrative inputs): M = 16, Mmax0 = 32, and assuming Mmax = M = 16, mL = 1/ln 16 ≈ 0.361, so (32 + 0.361 × 16) × 4 ≈ 151 bytes per vector. For 1,000,000 vectors of 1,536 float32 values the vectors take 1,000,000 × 6,144 B = 6.144 GB and the graph about 0.151 GB. The vectors, not the graph, dominate. The same formula gives about 61 bytes at M = 6 and 434 bytes at M = 48, consistent with the paper's range. Real systems store extra per-row overhead, so treat this as a floor, not a quote for any product.

The tradeoff. The paper states that memory is proportional to M, that smaller M is generally better for lower recall and lower-dimensional data and larger M for high recall and high-dimensional data, that efConstruction trades build time for quality (efConstruction = 100 gave a reasonable index for 10 million SIFT vectors in about three minutes on their server, with little gain from going higher), and that the search-time ef controls recall against latency. It reports logarithmic search scaling, analysed under an exact-Delaunay-graph assumption and confirmed on low-dimensional data, and says that for high-dimensional data further analytical evidence is required. The pgvector README adds that HNSW has a better speed-recall tradeoff than IVFFlat but slower builds and more memory.

What this means for architects

Recall is a dial, not a property. Measure it on your own vectors by comparing approximate results with exact results, as the pgvector README advises, then set efSearch to the lowest value that meets your recall target. Raise it for queries that feed a reranker (more candidates) and lower it for autocomplete-style paths. Budget memory from the formula, and keep the index resident: the README says indexes need not fit in memory but usually perform better if they do. Expect slower inserts at high efConstruction and slow vacuum after many deletes (the README suggests reindexing first). Filters change all of this, see Access control in retrieval. Do not copy the paper's benchmark numbers into your plan.

Under the hood: IVF and product quantization

Intuition

Two separate tricks, often combined. An inverted file (IVF) divides the space into cells with k-means and, at query time, looks only in the few cells nearest the query. Product quantization (PQ) compresses each vector into a short code by splitting it into pieces and replacing each piece with the id of its nearest representative. IVF reduces how many vectors you look at; PQ reduces how many bytes each one takes.

Mechanism

IVF. The FAISS paper (Douze et al., arXiv 2401.08281) describes a coarse quantizer with K centroids (nlist); each vector is stored in the inverted list of its nearest centroid, and a query visits only the P nearest lists (nprobe). With balanced lists and an exhaustive coarse quantizer the number of distance computations is K + P × N/K, minimised at K = √(P × N), which the paper calls a rough approximation, since P must grow with K to hold accuracy and lists are never equal in size. In its experiments the best K was roughly 15 to 20 times √N (the paper notes the ratio depends on the data distribution). Worked example (illustrative): N = 10,000,000 and P = 16 give K = √160,000,000 ≈ 12,649 and 12,649 + 16 × 10,000,000 / 12,649 ≈ 25,298 distance computations, about 395 times fewer than the 10,000,000 of an exhaustive scan. IVF must be trained (k-means needs data), so the pgvector README says to create an IVFFlat index after the table has data, with lists of rows/1000 up to one million rows and √rows beyond, and probes starting at √lists.

Product quantization. In the original paper (Jégou, Douze and Schmid, TPAMI 2011) a D-dimensional vector is split into m sub-vectors of dimension D/m, and each sub-vector is quantized by its own k-means codebook of k* centroids. The code is m × log₂ k* bits, and the implied codebook has (k*)ᵐ centroids even though only m × k* × D/m = k* × D floating-point values are stored. Distances are estimated without decoding: for a query, a table of squared distances to every sub-codebook centroid is computed once, and each database vector then costs m table lookups and additions (the asymmetric version, which leaves the query unquantized and is more accurate than quantizing both). IVFADC combines the two: it encodes the residual between a vector and its coarse centroid, and, with k coarse centroids and balanced lists, a search scans about n × w / k entries for w visited lists.

Worked example (illustrative inputs): N = 10,000,000 vectors of d = 1,536 float32 values take 6,144 bytes each, 61.44 GB in all. With PQ at m = 192 sub-vectors of 8 dimensions and k* = 256 (8 bits each), a code is 192 bytes, so 1.92 GB, a 32-times reduction; at m = 96 it is 96 bytes, 0.96 GB, 64 times. The codebooks add k* × d × 4 = 1,572,864 bytes (about 1.57 MB), and 8-byte ids add 80 MB. The accuracy cost is real and data-dependent: the distance is estimated from the code, not computed from the vector. The paper reports that for a fixed number of bits, fewer sub-quantizers with more centroids each give lower quantization error, and that its recall measurements depend on m, k* and w.

Figure 7. IVF with product quantization: coarse search, code scan, exact re-rank

How to read this: the first two steps narrow the search to a few lists, the middle steps use compressed codes, and the last two steps correct the estimate with full vectors for a small shortlist.

What this means for architects

IVF and PQ are what you reach for when vectors no longer fit in memory, and the cost is a measured loss of recall. Decide the memory budget first, then pick m and nprobe to hit the recall target, and always evaluate recall against exact search on your data. Keep the original vectors (on cheaper storage) and re-score a shortlist, as the PQ paper does with its shortlist R (in its setup the full vectors sit in main memory, which the authors note limits the scale at which re-ordering can be used) and as the pgvector README's binary-quantization example does. Measure how much of the compression loss this recovers on your data. The FAISS paper also warns that when the vector distribution shifts a lot through additions and removals, IVF and PQ degrade because they were fitted to the old data, so schedule retraining. Compared with HNSW, IVF builds faster and uses less memory but, per the pgvector README, has a lower speed-recall tradeoff.

Under the hood: BM25 and hybrid search

Intuition

BM25 scores a document by how many of the query's words it contains, weighting each word by how rare it is across the collection, letting repeated occurrences count less and less, and discounting long documents. It needs the literal token in the text. That is its weakness (no synonyms) and its strength (an identifier such as "14.2" or "ERR-4012" either matches or does not).

Mechanism

Stanford's Introduction to Information Retrieval (Manning, Raghavan and Schütze, section 11.4.3) builds up to the form used for document scoring, equation 11.32 (equation 11.33 adds a weight for repeated query terms with a third parameter k₃, and 11.34 adds relevance-feedback estimates; neither is used here):

RSV_d = Σ over query terms t of [ log(N / df_t) × (k₁ + 1) × tf_td / ( k₁ × ((1 − b) + b × (L_d / L_ave)) + tf_td ) ]

where N is the number of documents, df_t the number containing term t, tf_td the count of t in document d, L_d the document length and L_ave the average length. The book says k₁ is a positive tuning parameter (0 gives a binary model, a large value approaches raw counts), b between 0 and 1 sets length scaling (1 is full scaling, 0 none), and that without tuning, reasonable values are k₁ between 1.2 and 2 and b = 0.75. Lucene's BM25Similarity defaults are k₁ = 1.2 and b = 0.75, and it computes idf as log(1 + (docCount − docFreq + 0.5) / (docFreq + 0.5)), so implementations differ from the book in the idf form and scores are not comparable across systems.

Worked example (illustrative, natural log): N = 1,000 documents, L_ave = 200 words, k₁ = 1.2, b = 0.75, query "indemnification cap". Suppose df = 40 for "indemnification" (idf = ln 25 = 3.219) and df = 300 for "cap" (idf = ln 3.333 = 1.204).

A wins: it is shorter and contains both terms. B repeats the rare word five times but is twice the average length, so its length factor (1 − b) + b × L_d/L_ave is 1.75. Saturation shows in the term-frequency factor at average length, (k₁ + 1) × tf / (k₁ + tf): 1, 1.375, 1.774, 1.964 and 2.148 for tf = 1, 2, 5, 10 and 50, approaching the ceiling k₁ + 1 = 2.2.

Figure 8. Hybrid retrieval: two searches in parallel, then fuse

How to read this: the par block shows the two branches running at the same time, and the final note is the fusion step described in the next section.

Why hybrid. Anthropic's Contextual Retrieval post describes running BM25 and embeddings together and merging with rank fusion, and reports in its own evaluation that embeddings plus BM25 beat embeddings alone, with a top-20 retrieval failure rate of 5.7% falling to 2.9% when contextual embeddings and contextual BM25 were combined. Those are the post's numbers on its datasets, not a prediction for yours.

What this means for architects

Add a lexical index when queries contain identifiers, defined terms or exact phrases (legal, compliance, code, parts catalogues). pgvector's README suggests combining it with Postgres full-text search. Check what the text analyser does to your identifiers, for example whether "14.2" stays one token, because BM25 can only match the tokens the analyser produces. Tune k₁ and b on your golden set if you can, since the book says they should be tuned on a development collection. Keep one set of statistics per index and do not mix scores from different systems. The fusion step is the next section, and the lexical score should not be added directly to a cosine score, since the two are on unrelated scales.

Under the hood: Reciprocal rank fusion

Intuition

Different retrievers produce scores that cannot be compared: a BM25 score of 6.7 and a cosine of 0.82 mean different things. Reciprocal rank fusion (RRF) ignores scores and uses only ranks: a document earns more the nearer the top it sits in each list, and documents found by several lists add up.

Mechanism

Cormack, Clarke and Büttcher (SIGIR 2009, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods") publish the formula. Given a set of documents D and a set of rankings R, each a permutation on 1..|D|:

RRFscore(d ∈ D) = Σ over rankings r ∈ R of 1 / (k + r(d)), with k = 60

The paper says k = 60 "was fixed during a pilot investigation and not altered during subsequent validation", that the pilot indicated k = 60 was near-optimal but "the choice was not critical", and explains the shape: highly ranked documents matter more, but the importance of lower-ranked ones does not vanish as it would with an exponential. Its pilot table of mean average precision against k ran from 0.2072 at k = 0 to 0.2145 at k = 60, peaked at 0.2147 at k = 80, and was 0.2098 at k = 500. Across their pilot and TREC experiments RRF beat Condorcet Fuse, CombMNZ and the best single system by 4% to 5% on average, though not in every comparison (CombMNZ scored higher in one TREC comparison and slightly higher on LETOR 3); those were not retrieval-augmented generation corpora. Elasticsearch documents the same formula, 1.0 / (k + rank) with ranks starting at 1, a rank_constant default of 60, and equal weight for every retriever.

Worked example (illustrative lists): dense ranking [d3, d1, d7, d2] and BM25 ranking [d1, d9, d3], k = 60.

The fused order is d1, d3, d9, d7, d2. d1 wins although the dense retriever ranked d3 first, because both retrievers found d1 and agreement adds up. The constant matters at the margin. Compare a document a ranked first in only one list with a document b ranked fourth in both lists (ignoring all others): at k = 60, a scores 1/61 = 0.016393 and b scores 2/64 = 0.03125, so b wins; at k = 1, a scores 1/2 = 0.5 and b scores 2/5 = 0.4, so a wins. A small k rewards a top rank in one list, a large k rewards agreement.

Figure 9. Fusing two rankings with reciprocal rank fusion

How to read this: the two indexes send ranked lists, the notes are the fusion arithmetic, and the last message is the fused list passed on.

What this means for architects

RRF needs no score calibration and no training, which is why it is the default hybrid fuser, and the cost is what it throws away. It treats all retrievers equally (you cannot weight BM25 above the vector side without changing the method), and it discards score magnitude: a near-perfect match and a poor match at the same rank earn the same. So the fused score is a poor signal for abstention; threshold a reranker score or a per-retriever score instead, calibrated on labelled queries. Depth matters: each list contributes only as many candidates as you fetch (Elasticsearch calls this the rank window size), and a document outside both windows cannot appear. Keep k at 60 unless the golden set shows a reason to move it.

Under the hood: Rerankers

Intuition

A bi-encoder embeds the query and each passage separately and compares the vectors; the passage vectors are computed ahead of time, so search is cheap. A cross-encoder feeds the query and one passage into the model together, so every query word can attend to every passage word, and outputs a relevance score. It can be more accurate because it sees both texts at once (an expectation, which you should test on your data), and it is far more expensive because nothing can be precomputed: one model run per query-passage pair.

Mechanism

The Sentence-BERT paper puts numbers on the cost of the cross-encoder form: finding the most similar pair among 10,000 sentences needs n(n − 1)/2 = 49,995,000 BERT inferences, about 65 hours on a V100 GPU, against about 5 seconds to embed the 10,000 sentences once and a further 0.01 seconds for the cosine comparisons. Nogueira and Cho (arXiv 1901.04085) show the intended use of a cross-encoder, as the second stage of a pipeline: a standard retriever such as BM25 returns a large candidate set (for example a thousand), the re-ranker scores each candidate, and the top ten or fifty go to the answer stage. Their re-ranker feeds the query as sentence A and the passage as sentence B, truncating the query to 64 tokens and the whole input to 512, takes the [CLS] vector into a single-layer network to produce a probability of relevance, scores each passage independently, and sorts by that probability. On the MS MARCO passage task, MRR@10 on the dev set was 16.7 for untuned Lucene BM25 and 36.5 for the BERT-large re-ranker over the BM25 candidates.

Cost is linear in the number of candidates: reranking 1,000 passages needs 20 times as many model runs as reranking 50. Anthropic's Contextual Retrieval post sits at the practical end of this: retrieve the top 150 chunks, rerank them, pass the top 20 to the model, and it notes that reranking more chunks improves performance but adds latency and cost. In its evaluation, adding reranking to contextual embeddings and contextual BM25 lowered the top-20 failure rate from 2.9% to 1.9% (a 67% reduction against the 5.7% baseline).

Figure 10. Two-stage retrieval: a cheap wide stage, then a costly narrow stage

How to read this: the first stage is cheap and wide, the alt block is the costly narrow stage with its failure branch, and the self-message at the end is the hand-off.

What this means for architects

A reranker is worth adding when the right passage is usually among the first-stage candidates but not near the top, which you can see by comparing recall at 50 with recall at 5 on the golden set. It cannot fix a first stage that never returned the passage; for that, repair chunking and hybrid retrieval first. Budget latency as candidates times per-pair cost, since model runs batch on a GPU but still grow with the list. A hosted reranker is another trust boundary: query and passage text leave your network, so access-filtered text only, and a timeout path (Figure 10). Mind the input limit of your reranker, 512 tokens in the cited setup, because chunks longer than that are cut. A reranker score is the usual abstention signal, but whether a threshold generalises is something to test on your labelled queries and not assume.

Under the hood: Chunking and its failure modes

Intuition

One chunk gets one vector and one citation. If a chunk mixes two topics, its vector sits between them and matches neither well. If it cuts a sentence from its subject, the vector cannot say what the sentence is about. Chunking is therefore a modelling decision about what the smallest retrievable unit should be, not a preprocessing detail.

Mechanism

The RAG paper split each Wikipedia article into disjoint 100-word chunks, 21 million in all, embedded each one and built a single index. Anthropic's Contextual Retrieval post states that chunks are usually no more than a few hundred tokens, that chunk size, boundary and overlap affect retrieval performance, and illustrates context loss: a chunk reading "The company's revenue grew by 3% over the previous quarter" does not say which company or period. Its remedy is to prepend a short generated context (usually 50 to 100 tokens) to each chunk before embedding and before building the BM25 index, and it describes using prompt caching to make that affordable. Sentence-BERT's default pooling averages the model's output vectors over the tokens, so a long chunk with several topics produces a blurred average (many models pool this way; check yours).

Figure 11. A chunker that keeps structure and attaches context

How to read this: splitting goes from coarse to fine, the alt block is the size decision, and every chunk leaves with its context header and metadata.

Failure modes to look for: a cut through a table row, clause, list or code block; a lost referent such as "the company" or "this clause"; chunks over the embedding model's input limit, which is truncated or rejected depending on the model; chunks too small to hold an answer; overlapping chunks that fill the top k with near duplicates; and several versions of one document chunked identically so that stale and current text compete. The last is the root of the policy-conflict case.

What this means for architects

Store, with every chunk, the document id, version, section path, offsets and the chunker version, so that a citation can be shown, a document deleted, and a chunker change rolled back. Treat chunk size and overlap as parameters tuned on the golden set by recall at k, not defaults. A change to the chunker or the embedding model is a re-index, so version both and roll out like a migration. Put dates and versions in the chunk's own text only if the answer needs them; otherwise keep them as filterable metadata. For long structured sources (contracts, manuals) chunk by structure; for short ones (FAQs) one chunk per question and answer is usually right. See RAG chunking strategies.

Under the hood: Access control in retrieval

Intuition

Access control in retrieval answers one question: which chunks may this caller see? Where the answer is applied relative to the nearest-neighbour search decides both what is returned and what is exposed along the way. Pre-filtering restricts the search to allowed chunks. Post-filtering searches everything, takes the top k, and then removes forbidden chunks.

Mechanism

What is documented depends on the engine. The pgvector README states that with approximate indexes "filtering is applied after the index is scanned", so a filtered query can return fewer rows than asked for: if a condition matches 10% of rows, HNSW with the default hnsw.ef_search of 40 returns about 4 matching rows on average. Its fixes are iterative index scans (hnsw.iterative_scan set to strict_order or relaxed_order, which keep scanning until enough rows pass the filter, bounded by hnsw.max_scan_tuples, default 20,000), a partial index when the filter has few distinct values, and partitioning when it has many; and for multi-tenant use it warns that a shared approximate index lets one tenant's vectors affect another's recall and speed. Elasticsearch's documentation says a filter inside the knn clause "is applied during approximate kNN search to ensure that k matching documents are returned", while a filter beside it in a bool query is applied after the vector search, "so the response can contain far fewer than five hits, or none at all"; when few documents match in a segment it can skip the graph and score them directly. An Elastic Search Labs post dated 2026-10-02 says Elasticsearch now decides per query whether to filter before or after, keeps pre-filter semantics with a retry and fallback, and is scheduled to enable this by default in 9.6. The FAISS paper describes filtering by a predicate on ids, and "vector-first" versus "metadata-first" search, where metadata-first scans the matching subset by brute force, acceptable when the subset is small. So the behaviour is a property of your index and database, which you must read and test; none of the above transfers to another product.

Figure 12. Pre-filter versus post-filter for a tenant with a small share of the data

How to read this: the alt block compares the two designs, the arithmetic in the post-filter branch is the expected survivors, and the last note is the exposure that post-filtering creates.

Why post-filtering loses recall: with 500 equal tenants a tenant matches 1/500 = 0.2% of rows, so a top-40 list holds 40 × 0.002 = 0.08 allowed rows on average, which is usually zero (the same arithmetic as the README's 10% example). Why it can leak is reasoning, not a documented behaviour of any one product: the forbidden chunks have already been retrieved and exist in the retrieval layer before they are removed, so any step placed before the filter, such as logging, a cache, a call to an external reranker, or an error message that reports counts or scores, handles data the caller may not see. Postgres row-level security is one way to put the rule in the database: the documentation says policies restrict, per user, which rows normal queries can return, that with security enabled and no policy the default is deny, that superusers and roles with BYPASSRLS always bypass it, and that table owners normally do too unless FORCE ROW LEVEL SECURITY is set. The documentation read for this page does not say how policies combine with approximate vector index scans, so test that combination.

What this means for architects

Derive the filter from the verified identity inside the Retriever or below it, never from model output or a client field (the missing-boundary lesson of the helpdesk case in the diagramming page). Prefer pre-filtering semantics and prove them with a recall test per tenant size, not only a security test. Make the service's database role something that is not the table owner or a bypass role. Write tests that try to cross tenants (a query as tenant A that should match a tenant B chunk must return nothing, including through the lexical index, the cache and the reranker), and run them on every index change. Treat a permission change as a fast-path update: access revocation must reach the index much sooner than a content edit.

SET hnsw.iterative_scan = relaxed_order;

SELECT id, content
FROM chunks
WHERE tenant_id = $1
ORDER BY embedding <=> $2
LIMIT 8;

The statement is the README's filtered-query pattern with iterative scans so that a selective tenant filter still returns rows. relaxed_order can return results slightly out of distance order, and the README describes a materialised-CTE way to restore strict order. The README says iterative scans start with pgvector 0.8.0, so check the setting exists in your installed version.

Under the hood: Incremental indexing and freshness

Intuition

The index is a copy, and copies go stale. Freshness is the lag between a change in the Document store and the moment queries reflect it, and it has three parts: how fast events are noticed, how fast documents are processed, and how reliably failures are retried. The same machinery handles deletes, permission changes and a model change.

Mechanism

The RAG paper measured what staleness costs. It built an index from a December 2016 Wikipedia dump and compared it with the December 2018 index on 82 world leaders who had changed in between, asking "Who is {position}?": RAG answered 70% correctly with the 2016 index for 2016 leaders and 68% with the 2018 index for 2018 leaders, but only 12% (2018 index, 2016 leaders) and 4% (2016 index, 2018 leaders) when the index and the question's era did not match. The authors conclude that replacing the non-parametric memory updates the world knowledge without retraining. The index technology sets the rules for how you update: the FAISS paper says HNSW can add vectors on the fly, that the library's API can remove and update vectors by id (while its HNSW graph index does not support removal or mutation), and that when the vector distribution changes a lot techniques fitted to it, including IVF and PQ, degrade; pgvector's HNSW index can be created on an empty table while IVFFlat needs data first, the README recommends CREATE INDEX CONCURRENTLY in production so writes are not blocked, and notes that vacuuming an HNSW index can take a while, so reindex first.

Figure 13. Life of an index entry for one chunk version

How to read this: each box is a state of one chunk version, each arrow is a transition with its trigger, and the black dot and ringed dot are start and end.

The ordering rule that makes it safe: write the new version, then mark the old one superseded, then commit the event offset. A crash between any two steps repeats work and does not lose it, which is why every step must be idempotent (keyed by document, version and chunk number, as in Figure 2).

What this means for architects

Define a freshness objective in terms a user can feel, for example "99% of changes searchable within N minutes", measure it as the age of the oldest unprocessed change, and alert on it. N is yours to set; the paper gives no figure to copy. Separate lanes by urgency: deletes and permission revocations get a fast path, bulk re-embeds a slow one. Reconcile on a schedule by comparing document versions in the Document store with those in the indexes, since events can be lost. For a model or chunker change, build a new index beside the old, fill it from the Document store, evaluate on the golden set, then switch an alias, and keep the old one until rollback is no longer wanted. Put a dead-letter queue and an owner on it: a silent failure is a stale answer.

Decision framework

1. If the exact search meets the latency budget on your corpus (for example, a single tenant's partition), then use exact search with a filter, because recall is perfect and there is nothing to tune; revisit when p95 latency or memory breaks the budget.
2. If queries contain identifiers, clause numbers or defined terms, then run a lexical index beside the vector index and fuse with RRF, because embeddings blur exact strings and BM25 needs the literal token; revisit when the golden set shows the lexical side adds nothing.
3. If recall at 50 is good but recall at 5 is poor, then add a reranker; if recall at 50 is itself poor, then fix chunking and hybrid retrieval first, because a reranker only reorders what it is given; revisit when the first-stage candidate set changes.
4. If any caller sees only part of the corpus, then derive a filter from the verified identity and apply it inside the search (pre-filtering semantics), with partitions for large tenants, because post-filtering returns too few results and exposes forbidden text to earlier steps; revisit when the engine or its version changes.
5. If you change the embedding model, the chunker or the index parameters, then build a new index beside the old and switch after a golden-set comparison, because vectors from different models are not comparable and a bad change should be reversible; revisit when re-embedding time exceeds your change window.
6. If sources can disagree, then keep version, effective date and status as metadata, prefer the superseding document, and show both when neither supersedes the other, because the retriever cannot decide truth; revisit when a conflict is reported by a user.

Comparison of the main options

Failure scenarios to design for

1. The embedding model changes, or a stale index is queried with a new model. Defined behaviour: the index records its model id and the Retriever refuses to search on a mismatch and raises an alert; a new index is built in parallel and switched in after the golden-set comparison.
2. A filtered query returns too few results, or none. Defined behaviour: the Retriever uses pre-filtering semantics or iterative scans up to a cap, and if fewer than the minimum number of passages remain it returns an abstain result instead of padding with unfiltered text; a per-tenant-size recall test runs on every index change.
3. The lexical index and the vector index diverge because one write failed. Defined behaviour: a chunk version counts as indexed only when both writes succeed, the offset is committed after both, and a scheduled reconciliation compares the two and the Document store and alerts on a mismatch.
4. The reranker times out or is unavailable. Defined behaviour: the Retriever falls back to the fused order with a shorter candidate list and marks the trace; for high-risk use cases it returns an abstain result instead, because the abstention threshold cannot be applied without the reranker score.
5. Two sources conflict or a source is stale. Defined behaviour: the Retriever keeps the superseding version, or returns both with dates and sources for the answer to state, and flags passages past their review date.
6. Evidence is weak. Defined behaviour: when no passage reaches the calibrated threshold the Orchestrator does not call the model and tells the user nothing was found, as in the retrieval-failure sequence of the diagramming page.
7. A permission is revoked but the index still serves the chunk. Defined behaviour: permission changes travel on a fast lane, the filter metadata is updated first, and the staleness of permissions is measured separately from content freshness.

Worked example

Size a retrieval tier for a multi-tenant assistant and check three decisions with arithmetic.

Assumptions (all illustrative): 2,000,000 documents averaging 10 chunks each, so N = 20,000,000 chunks; vectors of d = 1,024 float32 values (4,096 bytes); 500 tenants of equal size; HNSW with M = 16, Mmax0 = 32 and Mmax = 16, so 151 bytes of graph per vector by the paper's formula; product quantization at m = 128 sub-vectors of 8 bits; first stages fetch 50 candidates each; embedding throughput of 2,000 chunks per second; an average of 300 tokens per chunk.

Memory. Vectors: 20,000,000 × 4,096 B = 81.92 GB. HNSW graph: 20,000,000 × 151 B ≈ 3.02 GB. PQ codes: 20,000,000 × 128 B = 2.56 GB (32 times smaller than the raw vectors, with the originals kept elsewhere for re-scoring).

Filtering. One tenant holds 20,000,000 / 500 = 40,000 chunks, 40,000 × 4,096 B = 163.84 MB. A shared HNSW index post-filtered by tenant leaves 40 × (1/500) = 0.08 allowed results per top-40 on average, so most queries return nothing. A per-tenant partition holds only 40,000 vectors, and an exact scan of it is 40,000 × 1,024 = 40.96 million multiply-adds per query, with perfect recall and no index parameters. The decision is therefore a partition per large tenant and a mandatory pre-filter for small ones.

Reranking. Fusing two lists of 50 gives at most 100 distinct candidates; reranking all of them is 100 model runs per query, against 1,000 runs if the first stage returned a thousand, a 10-times difference in model work.

Re-embedding. A model change means 20,000,000 chunks at 2,000 per second, 10,000 seconds or about 2.8 hours, and 20,000,000 × 300 = 6.0 billion tokens to embed (price it with the cost estimator). That is why the plan is a parallel index and an alias switch rather than an in-place update.

A hybrid retrieval function with RRF

The function below fetches two ranked lists with an identity-derived filter, fuses them with the published RRF formula and reranks with a fallback. The identity, vector_index, lexical_index and reranker objects stand for your own components; the assertions reproduce the worked example above, and the code was run with stub components to check it.

from collections import defaultdict

RRF_K = 60  # the constant published by Cormack et al. (SIGIR 2009)

def rrf(rankings, k=RRF_K):
    """rankings: ranked lists of chunk ids, best first. Returns [(id, score)], best first."""
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking, start=1):  # ranks start at 1
            scores[chunk_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: (-kv[1], kv[0]))  # ties: by id

def hybrid_search(query, identity, vector_index, lexical_index,
                  reranker=None, depth=50, final_n=8):
    allowed = identity.allowed_filter()  # built from the verified identity, never from model output
    dense = vector_index.search(query, filter=allowed, k=depth)    # chunk ids, best first
    lexical = lexical_index.search(query, filter=allowed, k=depth)
    fused = [cid for cid, _ in rrf([dense, lexical])][:depth]
    if reranker is not None:
        try:
            fused = reranker.rank(query, fused)  # reorders only, cannot add candidates
        except TimeoutError:
            pass  # fall back to the fused order and record it in the trace
    return fused[:final_n]

# the worked example from the lesson
fused = rrf([["d3", "d1", "d7", "d2"], ["d1", "d9", "d3"]])
assert [d for d, _ in fused] == ["d1", "d3", "d9", "d7", "d2"]
assert abs(fused[0][1] - (1 / 62 + 1 / 61)) < 1e-12

The reranker can only reorder the fused list, so a timeout degrades order and not correctness, but the function does not abstain: add a threshold check on the reranker score before using it in a high-risk path.

Common mistakes

• Adding a vector store and calling it retrieval. Without a lexical side, versions, filters and a golden set, the first exact-match or stale-document complaint has no fix.
• Mixing embedding models in one index. Vectors from different models are not comparable, and the search still returns results, so the error is silent.
• Post-filtering for access control. It returns too few results for small tenants and processes forbidden text before removing it.
• Adding scores from different retrievers. A BM25 score and a cosine similarity are on unrelated scales; fuse ranks, or calibrate on labelled data.
• Treating a reranker as a fix for bad recall. It reorders candidates and cannot add the passage that was never retrieved.
• Never measuring recall of the approximate index. The pgvector README tells you to compare approximate against exact search; without it the efSearch or nprobe setting is a guess.

Try it

Time: 30 to 40 minutes. Use any small corpus you have (a folder of manuals, a wiki export) and at least 20 questions for which you know the right passage.

1. Split the corpus into chunks with ids. For each question record the id of the right chunk.
2. Build two rankings per question: one from a vector search (any embedding model and exact cosine over the vectors) and one from BM25 (any library that implements it with k₁ = 1.2 and b = 0.75).
3. Fuse them with the rrf function above and compute recall at 5 for vector only, BM25 only and the fused list.
4. Pick one question where the fused result differs from the vector-only result and write down which retriever supplied the right chunk and why the other missed it.

Done when you have a three-row table of recall at 5 for vector, BM25 and fused retrieval over at least 20 questions, and a one-paragraph explanation for one question where fusion changed the top result.

Key takeaways

• An embedding is a learned summary of semantic closeness, blunt on exact strings; cosine, dot product and L2 agree only on unit-length vectors, and a model change forces a re-index.
• HNSW is a layered graph with three dials: M (memory and recall), efConstruction (build quality) and efSearch (recall against latency). Memory is roughly (Mmax0 + mL × Mmax) × 4 bytes per vector, small next to the vectors themselves.
• IVF limits how many vectors you scan and product quantization limits the bytes per vector, at a measurable accuracy cost; keep originals and re-score a shortlist.
• BM25 needs the literal token and scores by rarity, saturation and length; RRF fuses ranks with 1 / (60 + rank) and needs no calibration, but discards score magnitude.
• A cross-encoder reads query and passage together: accurate, linear in candidates, and unable to add a missing passage.
• Pre-filtering versus post-filtering is engine-specific; read and test your engine, and design for pre-filtering semantics and partitions for large tenants.
• Treat the index as a derived copy with versions, an ordered write sequence, a dead-letter path, reconciliation and a freshness objective.

Self-check

Further reading

• Yu. A. Malkov and D. A. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs: layers, the insert and search algorithms, the parameters and the memory formula.
• Matthijs Douze et al., The Faiss library: IVF, product quantization, HNSW in practice, filtering and index updates.
• G. V. Cormack, C. L. A. Clarke and S. Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR 2009.
• Manning, Raghavan and Schütze, Introduction to Information Retrieval, section 11.4.3 Okapi BM25.
• pgvector README: index types, parameters, filtering, iterative scans and multi-tenancy notes.

Last verified: 2026-10-22

Next

The next page covers agents, tools and the Model Context Protocol: Agents, tools and MCP.