Add a second-stage reranker behind keyword search, measure a small deterministic toy reranker on the capstone's golden questions, and learn when to use one.
What you'll learn: what a second-stage reranker does, why it can only reorder what the first stage found, and how to measure one. You build and test a small deterministic toy reranker and run it over the capstone's golden questions with real numbers. The toy is not a model. A real reranker model is out of scope here, and nothing below says how well one would work.
At a glance
Figure 1. Two stages: cheap recall, then a second look
How to read this: stage one decides which chunks are even considered; stage two only changes their order, and the context takes the first few.
A reranker re-scores a short list of candidates with a slower, more careful scorer.
It cannot find a chunk the first stage did not return: the pool size sets a ceiling.
Judge a reranker by the questions it changes, not by a demo or an average.
A weak reranker is worse than none: the toy built here loses three questions on its own.
The cost is per candidate, so pool size is a cost and latency knob as well as a quality one.
Scenario
Composite scenario, not a real company. A vendor tells Lumen Industrial Supply that adding "a reranker" in front of the copilot's search will improve relevance. The sales team nods. The FDE has the golden set and the evidence store, and a reasonable doubt: the capstone's search already puts most expected documents in the first three. The FDE decides to test the idea with a toy reranker first, to learn what to measure before any vendor's model is considered. All figures come from the capstone's data.
What you will be able to do
Describe the two-stage retrieval pattern and say what limits a reranker's gain.
Compute the best recall any reordering of a pool could reach.
Measure a reranker per question and list the questions it gained and lost.
Diagnose a loss to a signal the reranker ignored.
Decide from failure classes whether a reranker is the right fix.
Reranking in theory and its place in a retrieval design are in Data and retrieval.
Core ideas
Two stages and a ceiling
Anthropic's Contextual Retrieval article describes reranking as a filtering technique so that only the most relevant chunks reach the model. Its recipe:
retrieve a larger set of candidates first (the article used the top 150);
pass each with the query through a reranking model that scores it;
keep the top few (the article used 20).
It notes that reranking adds a small latency, and that there is a trade-off between reranking more chunks for quality and fewer for latency and cost. It recommends testing settings on your own use case.
Stage
Job
Cost
Cannot do
One: BM25 search
find candidates cheaply
scores every allowed chunk
judge subtle relevance
Two: reranker
reorder the candidates
scores each candidate
add a chunk stage one missed
That last cell is the ceiling. If the right document is not in the pool, no reordering can bring it back. On the capstone's golden set, questions that expect documents are 23 of 36, and the best document recall at 3 that any reordering of a pool could reach depends only on the pool size:
Pool (chunks from stage one)
Ceiling for document recall at 3
3 or 6
94.9% (same as plain BM25)
10
97.1%
15
97.1%
20
98.6%
30
100%
How to read this: the ceiling is a property of stage one's pool. At a pool of 3 or 6 the toy cannot gain at all, and the first real room to improve appears at 10.
Predict first. The toy reranker scores each of the top 10 chunks by how many query words appear in its text, plus how many appear in its title and heading, and it ignores the BM25 score entirely. Does document recall at 3 go up, stay the same or go down compared with plain BM25 (94.9 percent)? Decide, then open the answer.
Answer
It goes down, to 81.9 percent. BM25 already weighs rare words and chunk length, and it indexes the chunk's SKUs. A scorer that counts only word overlap throws those signals away. A second stage must add information the first stage lacks, not replace what it has.
The toy reranker
The toy is a deterministic score over three features: the BM25 score scaled by the best score in the pool, the share of query words found in the chunk text, and the share found in its title and heading. Ties keep the first-stage order. The weights are constructor arguments, so the same class is both the blended version (0.5, 0.3, 0.2) and the features-only version (0, 0.6, 0.4).
defscore(self, query: str, hit: Hit, best: float) -> float:
terms = query_terms(query)
ifnot terms:
return0.0
body = set(tokenize(hit.chunk.text))
head = set(tokenize(f"{hit.chunk.title}{hit.chunk.heading}"))
cover = sum(t in body for t in terms) / len(terms)
title = sum(t in head for t in terms) / len(terms)
returnself.w_bm25 * (hit.score / best if best else0.0) + self.w_cover * cover + self.w_title * title
defrerank(self, query: str, hits: Sequence[Hit]) -> list[Hit]:
ifnot hits:
return []
best = max(h.score for h in hits)
scored = [(self.score(query, h, best), position, h) for position, h inenumerate(hits)]
scored.sort(key=lambda item: (-item[0], item[1]))
return [h for _, _, h in scored]
A test in the lesson folder builds two constructed chunks to show the mechanism a reranker exists for: one stuffed with the query's first word and scored slightly higher by BM25, one that actually answers the question. The toy puts the answering chunk first. Another test pins the tie rule. The tie rule matters: if equal scores were sorted in the wrong direction, two equal chunks would swap places for no reason, and the per-question comparison would show movement that is not real.
NOTE
Constructed chunks show that the mechanism works. They say nothing about how often it helps on real questions. That is what the next section measures.
Measure it per question
The measurement takes the golden questions that expect documents, asks stage one for a pool of 10 chunks (public callers see public documents, superseded documents excluded), reorders the pool, and counts the first three distinct documents. It records recall at 3, the mean reciprocal rank (one over the rank of the first expected document, averaged), and the questions whose top three changed.
Ranking
Doc recall at 3
Mean reciprocal rank
Gained
Lost
BM25 order (baseline)
94.9%
0.906
none
none
Toy, features only
81.9%
0.826
none
q17, q19, q34
Toy, blended with BM25
94.9%
0.906
none
none
The blended version reordered the top three documents on six questions (q05, q07, q10, q13, q17, q19) and changed nothing that the metric can see. In no variant did any question gain. The features-only version changed twelve questions' top three and lost three.
Spot the flaw. A demo shows a reranker applied to the first three results of a search and reports "the right document moved from position 3 to position 1". The team concludes that reranking improves recall at 3. What is wrong?
Answer
Reordering the top three cannot change which three documents are in the top three, so recall at 3 cannot move; only position-sensitive metrics can. The pool must be larger than the number of results you keep, and the claim must be tested on questions chosen before the demo, with the lost questions listed next to the gained ones.
Worked example
Take q19, "Does the LUM-CT-200 offer an OPC UA server?". The expected document is the CT-200 datasheet.
BM25's top three documents are compat-opcua, ds-ct-200 and ds-gw-410.
The features-only toy returns ds-gw-200, ds-gw-100 and compat-opcua: the datasheet falls out.
Why? The product code is the strongest cue in the question, and BM25 sees it because the chunk's index text includes its SKUs. The toy looks only at chunk text, title and heading, so a gateway chunk full of "OPC UA server" scores higher by overlap.
NOTE
A reranker is judged by what it does with the signals the first stage already used.
One more loss fits the same pattern: q17 asks about a named product ("Which protocols does the LUM-GW-410 support?"). The third, q34, asks which vibration sensor has the widest frequency range and names no product code at all.
A test in the lesson folder shows a repair: add a bonus when a SKU in the query is listed on the chunk.
With a bonus of 0.4, q17 and q19 return and only q34 remains lost.
Recall at 3 rises from 81.9 to 90.6 percent, still below BM25's 94.9.
A second stage that must be patched this much to stop losing should keep the first stage's score as an input, as the blended version does.
Now the diagnosis that decides whether to build anything. The capstone's two document misses sit at different places:
In a plain search for the top 30 chunks, q07's expected FAQ is at document rank 6 (it is the tenth chunk), so it is inside a pool of 10 and a good reranker could lift it.
q01's expected datasheet is at document rank 14, outside any pool of 10, so only a bigger pool or a different first stage could help.
The ceiling table says the same thing: 97.1 percent at a pool of 10 means one expected document in the whole set is within reach.
Assumptions: document recall is in-sample (the same author wrote the questions and tuned the pipeline); 23 questions make one question worth 4.3 points; the toy is a stand-in for the idea, not for any model; the weights were chosen by hand after trying a few settings, so the two shown are not a tuned optimum.
Cost: with a pool of 10, a rerank scores 10 candidates per question, 230 scorings for the 23 questions here. A real reranking model or service has a per-candidate price and latency to take from its provider's documentation.
At the customer: "Will a reranker improve our search?" A good answer asks first which failures they have (retrieval, where the document is not in the pool, or ordering, where it is) and shows how to tell them apart on their own questions. It then proposes a pool size, a measurement per question, and a stop rule: if the second stage gains nothing the first stage can't, do not add a service.
Build it
The code is in capstone/fde-product-discovery-copilot/lessons/M3-L8/. Run it from the capstone folder with PYTHONPATH=. ./.venv/Scripts/python lessons/M3-L8/m3_l8_rerank.py. The real output:
bm25 order (baseline) recall@3 94.9% recall@1 66.7% MRR 0.906
toy, features only recall@3 81.9% recall@1 58.0% MRR 0.826
toy, blended with bm25 recall@3 94.9% recall@1 66.7% MRR 0.906
ceiling for a pool of 10: 97.1%
toy, features only {'gained': [], 'lost': ['q17', 'q19', 'q34'], 'top3_changed': ['q01', 'q04', 'q05', 'q07', 'q10', 'q13', 'q15', 'q16', 'q17', 'q19', 'q21', 'q34']}
toy, blended with bm25 {'gained': [], 'lost': [], 'top3_changed': ['q05', 'q07', 'q10', 'q13', 'q17', 'q19']}
Recall at 1 is capped below 100 percent because many questions expect two or three documents. The tests in test_m3_l8_rerank.py pin every number on this page: the baseline row, the two toy rows, the lost questions, q19's before and after, the ceilings for pools of 3, 6, 10, 15, 20 and 30, and the SKU-bonus repair. The rest of the capstone is unchanged: the toy is not wired into the pipeline.
WARNING
Do not wire a reranker into the pipeline because it passes a demo. The default stays BM25 until a measurement on the golden set, with the lost questions listed, says otherwise.
Exercise
In a new file under lessons/M3-L8/, write a reranker with a fourth feature: a bonus when a SKU named in the question is in the chunk's SKU list. Keep the features-only weights for the other three. Measure it with measure and changes.
Acceptance criteria:
your reranker reports which questions it gained and lost, compared with BM25;
a test pins its recall at 3 and its lost questions;
a comment explains in one sentence why any question is still lost.
Done when the file and its test pass, the whole capstone suite passes, and you can say in two sentences whether you would ship your reranker over plain BM25.
Hint 1: finding the SKUs in the question
The tokenizer keeps product codes whole and lowercase, so a query token that starts with lum- is a SKU. Compare it with the chunk's SKUs in lowercase.
Hint 2: what to check if q34 stays lost
Print the top three documents before and after for q34 and read the question. Ask which cue it contains: a product code, or a property of the product. A SKU bonus cannot help a question that names none.
Explain it back. In three sentences, explain to a customer why you measured a toy reranker before looking at any vendor's. A good explanation says that a second stage can only reorder what the first stage returned, that you learned which failures a reranker could even address, and that you can now judge any candidate by the questions it gains and loses.
Common mistakes
Reranking a pool no bigger than the number of results you keep, then expecting recall to move.
Replacing the first stage's score with a new one instead of adding to it.
Reporting an average and not the questions gained and lost.
Adding a reranker for retrieval failures, where the document never entered the pool.
Tuning weights by hand on the golden set and calling the result an accuracy estimate.
Forgetting that cost and latency grow with the pool size.
Knowledge check
1. Why can a reranker not fix a question whose document is at rank 14 when the pool is 10?
The reranker only sees the candidates stage one returned. A document outside the pool is never scored, so no ordering can bring it into the top results. A bigger pool or a better first stage is the fix, and the ceiling calculation shows this before any reranker is built.
2. The features-only toy lost three questions and gained none. What does that say about the design?
It replaced the first stage's signals, including the product codes BM25 indexes, with a weaker word-overlap score. A second stage should add information the first lacks. Blending the BM25 score back in removed all the losses, but it also gained nothing, so the toy did not earn its place.
3. A recall at 3 is unchanged but six questions have a different top three. Is that an improvement?
Not by itself. The metric cannot see a change in which documents fill the lower slots when the expected documents were already in place. It might matter if the third document is shown to the user, so inspect those questions, but do not call it a gain without a metric that moves.
4. How do you tell whether a miss is a retrieval failure or an ordering failure?
Search deeper than the context size and find where the expected document sits. If it is inside the pool but below the cut, it is an ordering failure and a reranker may help. If it is outside any reasonable pool, it is a retrieval failure and needs chunking, query or first-stage changes.
5. What does a pool of 10 cost compared with a pool of 30?
A reranker scores each candidate, so a pool of 30 does three times the scoring work of a pool of 10 for every query, with the corresponding latency and, for a hosted model, price. The ceiling rises from 97.1 to 100 percent here, so the question is whether that extra 2.9 points is worth three times the work.
You can now
Explain the two-stage pattern and its ceiling.
Build a small deterministic reranker and test its tie and mechanism behaviour.
Measure a reranker per question and list what it gained and lost.
Trace a loss to the signal the reranker ignored.
Decide whether a miss calls for a reranker or for a better first stage.
Further reading
Anthropic, Contextual Retrieval: reranking as a second stage, with its latency and cost trade-off.
Lesson 8 of 10 · Forward Deployed Engineer: RAG, Search and Data
Add a second-stage reranker behind keyword search, measure a small deterministic toy reranker on the capstone's golden questions, and learn when to use one.
In the rest of this lesson
Worked example
Build it
Exercise
Common mistakes
Knowledge check
You can now
Enrolling is free. It unlocks every lesson in Forward Deployed Engineer: RAG, Search and Data, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.