Patterns: Retrieval, Hybrid Search and Documents

What you'll learn: three patterns for answering from documents (retrieval with citations, hybrid retrieval, and a document-processing pipeline): when to use each, how it fails, how to evaluate it and what it costs.

At a glance

Figure 1. The answer patterns around the policy library

How to read this: the Retriever searches only current versions in the Document store, the Orchestrator gives the model the passages with their ids, and a Worker keeps the store correct in the background.

• Retrieval with citations answers from the current text and shows where each claim came from.
• Start with keyword search; add vectors only when your golden set (fixed test questions with known right answers) shows misses.
• A document pipeline decides which version is current before any search.

Scenario

Composite scenario, not a real company. At Lumen Industrial Supply, a fictional distributor, agents answer 3,360 policy questions a month (illustrative) from 62 documents. Nine exist in several versions, and an old version is still searchable on the intranet. The policy owner's demand: never misquote or invent policy. A return question needs P-RET-01 version 3 (30 days), not the superseded version 2 (14 days).

What you will be able to do

• Choose among cited retrieval, hybrid retrieval and a document pipeline.
• Name the metadata that makes "current version only" enforceable.
• Say how each pattern fails and how to measure it.

Prerequisites

• Patterns: extract, route and explain, the previous page.
• AI Architect data and retrieval for the architecture.

Core ideas

Pick a pattern from the requirement

Figure 2. Which answer pattern fits

How to read this: add the pipeline before search whenever versions exist.

Pattern 4: Retrieval with citations

• When to use: answers must come from changing documents and the user must be able to check the source.
• When not to: the answer is a field in a system (fetch it), or a few short, stable pages fit in the prompt.
• Structure: the Retriever to Document store to Model gateway path in Figure 1.
• Failure modes: a superseded passage retrieved; an answer saying more than the passage; a citation that does not support the claim; an answer where it should refuse.
• How to evaluate: golden questions with the expected policy id and version; judge retrieval (right passage returned?) apart from the answer (cited and supported?); include questions the library cannot answer.
• Cost profile: one model call with the question plus a few passages; passages are the main cost.
• Minimal build: metadata per document (policy id, version, effective date, status); search filtered to current versions inside the query; three to five passages passed in with ids; code checks every cited id was supplied; refuse below a relevance floor.
• Used in the Order and Policy Assistant: yes: the core of policy answers and the defence against the version 2 trap.

Pattern 5: Hybrid retrieval

• When to use: users ask by exact terms (a policy id) and by meaning, and keyword search alone misses on your golden set.
• When not to: keyword search already meets the target; vectors add an index, embedding cost and a second thing to evaluate.
• Structure: the Retriever in Figure 1: two ranked lists merged into one.
• Failure modes: a merge that buries a good exact hit; near-identical versions scoring alike; an index that lags the documents.
• How to evaluate: run the same golden set with keyword alone and with hybrid; adopt it only if the gain justifies the cost.
• Cost profile: embeddings for every document and query, plus a vector index to host and refresh.
• Minimal build: keyword first; list its misses; add vector search; merge the lists with a simple rule such as reciprocal rank fusion; rerun the golden set (search for RAG).
• Used in the Order and Policy Assistant: not in the MVP (minimum viable product, the first release). Policy questions use ids and fixed phrases, so keyword search starts; vectors only if evaluation shows a need.

Pattern 6: Document-processing pipeline

• When to use: documents arrive or change and must be parsed, checked, labelled and stored before anything uses them.
• When not to: a small one-off set you can load by hand, or content that is already structured data.
• Structure: the Worker to Document store path in Figure 1.
• Failure modes: a superseded version indexed as current; a parser dropping tables; an unnoticed failed run; missing metadata hiding a document.
• How to evaluate: document counts in and out match, a spot check of parsed text against the source, and a test that fails if a superseded document is returned.
• Cost profile: off the request path; mostly engineering and storage; model calls only to extract fields, once per change.
• Minimal build: a loader, a parser, a required-metadata check, a version rule (the latest version whose effective date has passed is current, older ones are marked superseded) and a report of changes (ingestion).
• Used in the Order and Policy Assistant: yes, small: load the 62 documents, mark superseded versions, report the nine multi-version policies. No model is needed to start.

In the interview: "How do you stop the assistant quoting an outdated policy?" A strong answer puts the version decision in metadata, filtered inside the search, with a test that fails if a superseded version is returned.

Worked example

Assumptions: illustrative question and passages. P-RET-01 version 3 says 30 days with a 15 percent restocking fee for opened non-defective items; version 2 (superseded) says 14 days.

Spot the flaw. A team indexes all 62 documents including old versions and writes in the prompt: "If several versions match, use the most recent one." What is wrong?