Pseudocode.
at ingest: for each chunk -> vector = embed(chunk text); store (chunk ref, vector, metadata)
at query: q = embed(question); candidates = nearest(q, k, filter on metadata)
decide: embedding model, vector size, distance, exact or approximate index
Two facts matter for the decision. Anthropic's write-up notes that embeddings "can miss crucial exact matches" while BM25 is effective for unique identifiers and technical terms. And pgvector, a PostgreSQL extension, searches exactly by default and offers approximate indexes that "trade some recall for speed"; with those, filters are applied after the index scan, so a restrictive filter can return fewer rows than asked (as of 2026-11-02, from the pgvector README).
Hybrid search: fuse the rankings
Reciprocal rank fusion (RRF) scores each document by adding 1 over (k plus its rank) across rankings, with k set to 60 in the original paper (ranks start at 1). In practice a document missing from a ranking simply adds nothing; the paper itself ranks every document. It uses ranks only, so a BM25 score and a cosine similarity need no common scale.
Verified: capstone/fde-product-discovery-copilot/lessons/M3-L3/m3_l3_fusion.py
def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
"""score(d) = sum over rankings of 1 / (k + rank), ranks start at 1 (Cormack et al., SIGIR 2009).
A document missing from a ranking adds nothing. Ties keep first-seen order."""
scores: dict[str, float] = {}
for ranking in rankings:
for position, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + position)
return sorted(scores.items(), key=lambda item: -item[1])
The widget fuses the first four documents of two real rankings for the capstone's first golden question. The first list is the normal BM25 order. The second is a second, deliberately different keyword view (titles and headings only), standing in for a vector list because the capstone has none.
The starting output ranks compat-opcua first (score 0.03279, ranked first in both lists), ds-gw-200 second (0.03200) and faq-humid-environments third (0.01613, found only by the second list at rank 2). A document at rank 2 in one list beats a document at rank 3 in one list. Try k = 1 and watch the gaps widen.
Worked example
Fusion on all 23 questions that expect documents, scored by document recall at 3 (first three distinct documents), from the lesson code:
Fusion changed three questions:
- q01 improved from 0.33 to 0.67: in the widget example
faq-humid-environments entered the top three.
- q07 improved from 0.5 to 1.0.
- q34 fell from 1.0 to 0.0: the weaker list pushed unrelated documents above the datasheet.
Two gains and one loss is a net of minus 0.7 points. Fusion is not magic: it helps when the second list finds what the first misses and is not much noisier. Varying k from 1 to 1000 left recall at 94.2 percent, because both lists are short.
Assumptions: the question set is in-sample (the same author wrote it and tuned the pipeline), one question is 4.3 points, and the weak ranker is a stand-in, not a real vector index. A real embedding model would behave differently, so the finding is about method, not about vectors.
Always report which questions changed, not only the average. Two gains and a loss average to almost nothing and hide that fusion improved both known misses and broke a question that worked.
Choosing a store
Figure 3. A store decision, in the order to ask
flowchart TD
accTitle: Choosing a search store
accDescr: Start by measuring misses. If questions use the documents' own words, keep keyword search. If measured misses come from different wording, add vector search and fuse the rankings. Then choose where to run it: in the database you already operate if it supports the search and the filters you need, otherwise a dedicated search service.
A["Measure misses on a question set"] --> B{"Do misses come from different wording?"}
B -- no --> C["Keep keyword search, fix chunking or data"]
B -- yes --> D["Add vector search, fuse with keyword"]
D --> E{"Do you already run a database with vector support?"}
E -- yes --> F{"Does it filter inside the query as you need?"}
E -- no --> G["Evaluate a dedicated search service"]
F -- yes --> H["Use the database you operate"]
F -- no --> G
How to read this: the first test is about your measured misses, and a product name does not appear until the last two questions.
At the customer: "Which vector database should we use?" A good answer starts with the measured misses on their own questions, says what a vector index would and would not fix, and names the operational cost of each option. The product comes last.
Build it
Work in capstone/fde-product-discovery-copilot. The fusion experiment is lessons/M3-L3/m3_l3_fusion.py; the tests are in the same folder.
PYTHONPATH=. ./.venv/Scripts/python lessons/M3-L3/m3_l3_fusion.py
./.venv/Scripts/python -m pytest lessons/M3-L3 -q
The real output of the first command:
chunk text (BM25) doc recall@3 94.9%
titles and headings (BM25) doc recall@3 63.8%
fused (RRF, k=60) doc recall@3 94.2%
changed by fusion (id, text, headings, fused): [('q01', 0.33, 0.67, 0.67), ('q07', 0.5, 1.0, 1.0), ('q34', 1.0, 0.0, 0.0)]
[('vibration', 13, 165, 2.509), ('opc', 37, 165, 1.488), ('gateway', 64, 165, 0.945), ('lum-gw-200', 26, 165, 1.835)]
The second list is built by HeadingIndex, a BM25 index over title and heading text only. Replace it with a function that returns a ranked list of document ids and the rest of the code stays the same; that is where a vector ranking would plug in.
Exercise
Decision exercise. Support wants two things: "find the manual for error code E-17" and "find the thing that makes the pump shake". Options: (A) BM25 only; (B) vector search only; (C) keep BM25, measure the misses on 30 real queries, and add vector search with fusion only if misses come from wording. Choose and justify.
Model answer
Choose C. The error code is a literal identifier, which BM25 handles and embeddings can miss, so B risks the query that works today. The second query is a paraphrase, which BM25 cannot see, so A leaves a known gap. Measuring first shows how often each kind occurs; if wording misses are rare, the added system is not worth operating. If they are common, add vector search behind the same Retriever and fuse.
Task: in a copy of the lesson file, write a third ranking function (for example BM25 over SKUs only) and fuse all three. Report document recall and which questions changed. Acceptance criteria: the original tests still pass; a test pins your numbers; your report says whether the third list helped and by how many questions. Done when the report and the test are in your lesson folder and the whole capstone suite is green.
Hint 1: what should the third list know?
Pick a view of the corpus the first two lists do not use. The SKU list on each chunk is metadata, so a query that names a product should favour documents that list it.
Hint 2: how to avoid fooling yourself
Compare questions, not just the mean. A list that only repeats the first one adds nothing, and a noisy list can lose a question the first list had right.
Common mistakes
- Buying a vector database before measuring which questions fail and why.
- Reading a high BM25 score as "the answer is here"; it only means the words matched.
- Adding a second ranking without checking per-question changes, so a loss hides in the average.
- Fusing scores from different scales instead of ranks.
- Choosing the store from the vendor's feature list instead of from your filters and operations.
Knowledge check
1. Why does a rare query word count for more than a common one in BM25?
The idf term is large when few chunks contain the word, so a match on it narrows the candidates sharply. In this corpus "vibration" appears in 13 of 165 chunks and scores 2.509, while "gateway" appears in 64 and scores 0.945. A match on the rarer word is stronger evidence.
2. Why does keyword search return results for a paraphrase that shares no meaningful words?
BM25 ranks every chunk that shares at least one non-stopword with the query, however weak. It cannot say "no good match" on its own, so weak matches look like answers. A threshold or an evidence check in the pipeline has to decide when to refuse.
3. Why does reciprocal rank fusion use ranks instead of scores?
BM25 scores and vector similarities live on different scales and distributions, so adding them needs calibration. Ranks are comparable across systems, and 1 over (k plus rank) rewards documents that appear high in several lists without any tuning of scales.
4. Fusion lifted two questions and dropped one. What do you report?
Report the mean (94.9 to 94.2 percent), the three changed questions with before and after values, and the cause of the loss, a weaker list pushing unrelated documents upward. Also state that the set is in-sample. A reader can then judge the trade instead of trusting an average.
5. When would you keep keyword search and not add vectors?
When measured misses are mostly not wording problems: missing documents, chunking, stale versions or access rules. A vector index cannot fix those, and it adds an index to build, secure and operate. Add it only after the question set shows wording misses are frequent.
You can now
- Explain BM25's idf and length normalisation using real counts from the corpus.
- Show a paraphrase miss and say why it returns weak results rather than none.
- Compute and read an RRF fusion, and report which questions it changed.
- Walk a customer through a store decision that starts from measured misses.
Further reading
- Cormack, Clarke and Buettcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods", SIGIR 2009, PDF: the method, with k = 60 fixed in a pilot.
- pgvector README: exact and approximate search, index types and filtering behaviour.
- Anthropic, Contextual Retrieval: BM25 and embeddings used together.
- PostgreSQL documentation, Full text search: introduction.
Last verified: 2026-11-02
Next
Metadata filters and tenant-aware access control