Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Data and Retrieval Architecture for AI Systems.
Context
A home and motor insurer has 140 customer-service agents. They answer questions such as "Is a burst pipe covered if the house was empty for 30 days?" while a customer waits on the phone. The answers live in about 2,000 documents: policy wordings, endorsements that amend them, internal circulars and an agent FAQ. Many documents exist in several versions, because a customer is bound by the wording in force on their policy's start or renewal date, and a claim is judged by the wording in force on the date of loss.
A policy assistant retrieves passages from these documents and has a model write a short answer with citations. It went live to agents after a pilot that tested only the newest documents.
Problem statement
A specialist audit of 200 sampled answers found two kinds of error. In 31 answers (31 / 200 = 15.5%) the cited passage came from a superseded version. In 9 answers (4.5%) two documents that were both in force disagreed, and the assistant quietly picked one. Together that is 40 of 200, or 20%.
The cause is visible from the data: a clause that survives unchanged through three versions produces three near-identical chunks. All three land in the top five results, and the model sees three answers that look equally good. If it picks one at random, it is wrong in 2 of 3 cases (66.7%) whenever the question concerns the newest version.
Measurable goal: at most 2 wrong-version answers in a fresh sample of 200 (1%), and every genuine conflict shown to the agent instead of resolved silently.
Constraints
- Fixed: old versions must stay answerable, because claims on old losses need the wording of that date.
- Fixed: an agent must see which document, version and effective date backs each answer.
- Fixed: the assistant gives guidance to agents; it does not make coverage decisions.
- Negotiable: which documents are indexed first, and the escalation route for conflicts.
- Limit: the policy team can fix metadata for about 50 documents a week.
Why the traditional approach fell short
Keyword search over the document system returns every version of a clause, ranked by term frequency, with no notion of which one applies. The agent FAQ is written once and drifts: it still quoted a 60-day unoccupancy limit after a wording cut it to 30 days. A plain retrieval-augmented system inherits the keyword problem: embeddings measure similarity of text, and two versions of a clause are, by design, similar.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Controlled lookup, no AI | A structured page keyed by product and date, with the key clauses written once by the policy team | Kept as the fallback for the 20 most asked topics; accurate, but covers only what the team has written |
| B. Delete old versions from the index | Only current documents are searchable | Rejected: claims on older losses need the older wording |
| C. Version metadata, in-force filter, conflict surfacing | Every chunk carries version and dates; retrieval filters to what is in force on the as-of date; disagreements are shown | Accepted |
The default is C because the failure is a data-model problem, and filtering fixes it before the model sees anything.
Solution architecture
Figure 1. Ingestion with metadata and a quarantine path
How to read this: nothing reaches the index without version and date fields, and incomplete documents wait in a queue instead of being searched.
Figure 2. One question with the conflict path
How to read this: the filter runs inside the search, before the model; the nested alternatives show the empty, the conflicting and the clean outcome.
How AI solved it, step by step
One question, end to end.
- An agent asks, "Is a burst pipe covered if the house was empty 30 days?" The client app sends the product code and the as-of date from the policy screen (the date of loss, for a claim).
- The Orchestrator passes the question and the filter to the Retriever. Decision: the filter is built from system data, not from the model's guess.
- The Retriever searches only chunks of that product where the effective-from date is on or before the as-of date and the effective-to date is empty or later.
- Superseded versions fall out of the result set. If two in-force documents remain (a wording and its endorsement), their authority level decides which is primary.
- If no passages are left, the assistant says so and offers the policy team. Decision: no answer beats an old one.
- The model gateway receives the passages and is asked for a short answer plus a yes or no on whether the passages disagree, with the quotes if they do.
- On a conflict the agent sees both passages with their versions and dates, and a ticket goes to the policy team. On no conflict the agent sees one answer with document, version and date.
- The Trace store records the filter, the passages and the outcome.
Under the hood
The key step is what each chunk knows about itself. A chunk cut from the middle of a wording says "unoccupied for more than 60 days" and nothing about which wording, which version or which product. Anthropic's post on contextual retrieval states the problem generally: "traditional RAG solutions remove context when encoding information", with an example chunk about revenue growth that "on its own doesn't specify which company it's referring to or the relevant time period." Here the missing context is version and date.
The fix has two parts. First, every chunk carries structured fields (document id, version, effective-from, effective-to, supersedes, authority level, section path) so the filter can act on them without the model. Second, a short context line such as "Home wording v3, effective 2024-01-01, section 7.2 Unoccupancy" is placed in front of the chunk text before embedding and before lexical indexing, so the chunk's own text reflects where it came from. Chunk boundaries matter too: a definition split from its clause misleads on its own. The failure modes of chunking and how to check them are in Under the hood: Chunking and its failure modes.
What went wrong / failure modes
- Wrong as-of date. The assistant filters correctly on the wrong date, for example today instead of the date of loss. Mitigation: show the as-of date in every answer and make it an explicit field the agent can change.
- Missing metadata on legacy documents. Scanned or old files have no effective dates. Mitigation: they go to the review queue, not the index; the queue is worked at 50 documents a week and the backlog is reported.
- Broken supersedes chain. A new version is published without closing the old one, so both look in force. Mitigation: publishing a version sets the previous version's effective-to date in the same step, and a nightly check lists products with two in-force versions of one document.
- Conflict check misses a disagreement. The yes or no is a model judgement and can be wrong. Mitigation: sample 20 answers a week against a specialist and treat the check as a safety net, not the control.
Results
These are illustrative targets, not measurements. In the audit, 40 of 200 answers (20%) were affected. The target is at most 2 of 200 (1%) wrong-version answers and all 9 audited conflicts shown to the agent. With 140 agents asking 30 questions a day there are 4,200 questions a day. If 25% of them return passages from two or more documents and so need the conflict check, that is 1,050 extra model calls a day, and the filter adds no model call. If 4.5% of questions meet a genuine conflict, that is 189 conflict displays a day; each distinct conflicting pair is ticketed once, so the policy team sees pairs, not 189 tickets.
Tradeoffs and when NOT to use this approach
- Metadata work is real work. If the corpus is 30 documents that change twice a year, option A (a controlled page) is cheaper and safer.
- Filtering assumes the as-of date is known. If agents cannot tell which date applies, the tool moves the error into a field.
- Do not use this where a wrong answer causes a regulated outcome without a human check.
Design rules extracted
- Version, effective dates and supersession belong on every chunk, and the filter runs before generation.
- Never delete old versions that older cases still need; select by the as-of date.
- When two sources of equal authority disagree, show both with their dates; do not choose silently.
- Documents without trustworthy metadata wait in a queue, not in the index.
- Make the as-of date visible and editable in the answer.
Assumptions
All numbers are illustrative.
- 2,000 documents; about 600 are superseded versions and 1,400 are current. Chunking is not specified.
- Audit: 200 sampled answers; 31 superseded (31 / 200 = 15.5%), 9 silent conflicts (9 / 200 = 4.5%); 31 + 9 = 40 (20%).
- Three near-identical chunks and a random pick gives 2 / 3 = 66.7% wrong. This assumes the model has no reason to prefer the newer text.
- 140 agents x 30 questions = 4,200 a day; 25% needing the check = 1,050 calls; 4.5% x 4,200 = 189 conflict displays.
- The policy team capacity of 50 documents a week, the 20 controlled topics and the 20-answer weekly sample are chosen for illustration.
- Anthropic's contextual retrieval post (https://www.anthropic.com/news/contextual-retrieval, read 2026-10-21) is used only for the quoted statement and example; the insurer, the documents and the clauses are invented.
- Not legal or regulatory advice.
Last verified: 2026-10-22