Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Data and Retrieval Architecture for AI Systems.
Context
The knowledge team of a commercial law firm maintains a library of 6,500 precedent agreements and negotiated contracts. Each is split into clauses, about 80 per agreement, so the index holds 6,500 x 80 = 520,000 clause-level passages. Associates ask two kinds of question. Some are paraphrases ("which of our supply agreements let the buyer walk away on short notice?"). Others name something exact: a clause number ("clause 14.3(b) of the 2019 supply agreement") or a defined term with a fixed capital-letter meaning ("Permitted Disclosure", "Change of Control").
A first assistant used embeddings only: retrieve the five nearest clauses, then ask a model to answer from them.
Problem statement
An evaluation set of 200 answerable questions, 60 of which name a clause number or defined term and 140 of which are paraphrases, showed the split. The right clause was in the top five for 21 of the 60 exact questions (35%) and 119 of the 140 paraphrases (85%), which is 140 of 200 (70%) overall. A further 30 questions had no answer in the library; the assistant answered 22 of them anyway (73.3%).
The worst failure is the confident wrong one. Asked whether clause 14.3(b) of the 2019 supply agreement allows assignment without consent, the retriever returned an assignment clause from a different agreement, similar in meaning but not the one asked about, and the model answered from it.
Measurable goal: the right clause in the top five for at least 90% of both groups, and at most 3 of the 30 unanswerable questions answered.
Constraints
- Fixed: every answer quotes the clause and names agreement, clause number and version.
- Fixed: a lawyer is accountable for the conclusion; the tool finds and reads clauses.
- Fixed: confidential agreements stay inside the firm's approved environment.
- Negotiable: a few seconds of latency, and the number of candidates reranked.
- Limit: the team has no search specialist; the lexical component must come from an existing engine.
Why the traditional approach fell short
Boolean keyword search finds "14.3(b)" and "Change of Control" exactly, and misses "terminate for convenience" when the text says "exit on notice". Embedding-only retrieval has the opposite profile: it matches meaning and treats a clause number as noise, so "14.3(b)" is near "14.3(c)" and near "12.1" of another agreement. Clause libraries tagged by hand cover the types someone anticipated.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Keyword and boolean only, no AI | Add clause-number and defined-term fields to the existing search | Kept as one half of the final design and as the fallback; exact, but blind to paraphrase |
| B. Embeddings only | The first assistant | Rejected: misses exact terms and does not know when it is wrong |
| C. Hybrid lexical plus vector, rank fusion, reranker, abstention | Run both searches, merge by rank, rerank the top 20, answer only on strong evidence | Accepted |
The default is C. Some teams skip the reranker to save latency; keep it only if the evaluation shows it lifts the 90% target, because it adds a model pass for each candidate.
Solution architecture
Figure 1. Hybrid retrieval components
How to read this: two searches feed one merged list, a reranker orders it, and the orchestrator decides between answering and abstaining.
Figure 2. One question with the abstain path
How to read this: the two searches run side by side, and two checks, the score threshold and the quote check, stand between the retriever and the lawyer.
How AI solved it, step by step
- An associate asks whether clause 14.3(b) of the 2019 supply agreement allows assignment without consent.
- The Orchestrator extracts exact items: the clause number "14.3(b)" and the agreement. It passes the question and these items to the Retriever.
- The lexical search looks for the clause number and terms. The vector search looks for the meaning. Decision: both always run, because neither alone covers both question types.
- The Retriever fuses the two ranked lists by rank, which avoids having to compare a lexical score with a similarity score.
- The Reranker reads the question and each of the top 20 passages together and reorders them.
- The Orchestrator checks the evidence. Decision: if the best score is under the threshold, or the clause the question names is not among the passages, it replies "no matching clause found" and offers keyword search.
- Otherwise the Model gateway drafts a short answer that quotes the clause.
- A check confirms that every quoted span appears verbatim in a retrieved passage; an answer that fails is withheld. The associate sees agreement, clause number, version and the quote.
Under the hood
BM25 scores a passage by how often query terms appear in it, how rare each term is across the library, and the passage's length against the average. Two parameters tune it: Elasticsearch's similarity documentation gives the defaults k1 = 1.2 (how quickly repeated occurrences stop adding score) and b = 0.75 (how strongly length matters). Anthropic's contextual retrieval post says BM25 is "particularly effective for queries that include unique identifiers or technical terms", with the example of the error code "TS-999". A clause number and a defined term behave like those identifiers. Whether "14.3(b)" survives your analyser as one token is a configuration question to test, or index clause numbers in a separate exact-match field.
Fusion uses reciprocal rank fusion. Cormack, Clarke and Büttcher define RRFscore(d) as the sum over rankings of 1 / (k + r(d)), "where k = 60 was fixed during a pilot investigation". Worked example with k = 60: clause C (the one asked about) is 7th in the vector list and 1st in the lexical list; clause A is 1st and 2nd. Score(A) = 1/61 + 1/62 = 0.0325; score(C) = 1/67 + 1/61 = 0.0313; clause B, 2nd in the vector list only, scores 1/62 = 0.0161. Fused order: A, C, B. C moved from 7th to 2nd, and the reranker can promote it to 1st. Anthropic publishes that on its own datasets a reranked, contextual-embedding-plus-BM25 pipeline cut the top-20 retrieval failure rate by 67% (5.7% to 1.9%); treat that as a pointer to test, not a promise. The mechanism details are in Under the hood: BM25 and hybrid search.
What went wrong / failure modes
- Tokenisation splits the clause number. "14.3(b)" becomes fragments that match many clauses. Mitigation: test it on the evaluation set; index clause numbers as exact fields.
- Defined terms in lowercase. The same words used in their ordinary sense match too. Mitigation: boost capitalised defined terms and check them against the agreement's definitions clause.
- Right clause, wrong version. A superseded amendment ranks first. Mitigation: index the operative version and show its date.
- Threshold set too high. The system abstains on answerable questions. Mitigation: set it on the evaluation set, track the abstain rate, and always offer keyword search.
Results
These are illustrative targets, not measurements. On the 200 answerable questions the target is 54 of 60 exact questions (90%) and 126 of 140 paraphrases (90%) with the right clause in the top five, which is 180 of 200 (90%), against 140 of 200 (70%) now. On the 30 unanswerable questions the target is at most 3 answered (10%), against 22 (73.3%). The cost is a second search per query and one reranker pass per candidate: 20 passes for each question.
Tradeoffs and when NOT to use this approach
- Two indexes mean two things to keep in sync. If questions are almost all paraphrases, vectors alone may be enough.
- The reranker adds latency and cost per query. Skip it if the fused list already meets the target.
- Abstention frustrates users when the threshold is wrong. Measure the abstain rate against answerable questions.
Design rules extracted
- Exact identifiers need lexical search; paraphrase needs vectors; legal text has both, so run both.
- Merge by rank, not by raw score, because the two scores are not comparable.
- Define when the system must say "no matching clause" and test it with questions that have no answer.
- Verify that every quote appears verbatim in the passage it cites.
- Keep the evaluation set split by question type so one average does not hide a weak group.
Assumptions
All numbers are illustrative.
- 6,500 agreements x 80 clauses = 520,000 passages.
- Evaluation set: 200 answerable questions (60 exact plus 140 paraphrase) and 30 unanswerable. Baseline: 21 / 60 = 35%, 119 / 140 = 85%, (21 + 119) / 200 = 70%; 22 / 30 = 73.3%. Targets: 54 / 60 = 90%, 126 / 140 = 90%, 180 / 200 = 90%; 3 / 30 = 10%.
- Worked fusion example uses k = 60 from Cormack, Clarke and Büttcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods", SIGIR 2009 (read 2026-10-21 from http://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf). The ranks of clauses A, B and C are invented.
- BM25 defaults are Elasticsearch's, as stated at https://www.elastic.co/docs/reference/elasticsearch/index-settings/similarity and in https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables (read 2026-10-22); other engines may differ.
- The BM25 quotation and the 67% figure come from https://www.anthropic.com/news/contextual-retrieval (read 2026-10-21); the 67% is Anthropic's result on its own datasets.
- Top 20 for reranking and the score threshold are chosen for illustration. The firm, the agreements and the clause are invented. Not legal advice.
Last verified: 2026-10-22