Context Assembly and Citations for RAG Answers

Figure 2. Where a citation is checked

How to read this: the two self-arrows on the orchestrator are the only places a citation is trusted or dropped, and neither involves the model.

Verified: capstone/fde-product-discovery-copilot/app/discovery.py

def _validated(out: ComposeOutput, cands: list[Candidate], ctx: Context) -> list[Recommendation]:
    """Code, not the model, decides what survives: known SKU, real chunk ids that mention it."""
    by_sku = {c.product.sku: c.product for c in cands}
    chunks = {c.ref: c for c in ctx.chunks}
    recs: list[Recommendation] = []
    for draft in out.recommendations:
        product = by_sku.get(draft.sku)
        if product is None or any(r.sku == draft.sku for r in recs):
            continue
        refs = [r for r in dict.fromkeys(draft.citations) if r in chunks and is_about(product, chunks[r])][:3]
        if refs:  # no citation, no claim
            recs.append(Recommendation(sku=product.sku, name=product.name, why=draft.why, citations=refs))
    return recs

Three rules: a draft for an unknown SKU is dropped; a citation not in the context is dropped (this removes invented ids such as ds-gw-200#9); a citation survives only if is_about says the chunk names the product or belongs to its own document. If nothing survives, the draft is dropped.

Refusing what the documents cannot answer

Before retrieval, the gate asks: is the request about this domain, and what share of its topic words appear in the documents and catalogue this caller may see?

Verified: capstone/fde-product-discovery-copilot/app/retrieval/support.py

def coverage(text: str, vocabulary: set[str]) -> float:
    terms = significant_terms(text)
    if not terms:
        return 1.0
    return sum(t in vocabulary for t in terms) / len(terms)

min_coverage is 0.6. A request below it is refused with "I cannot answer this from the documents."

The question is "Do you sell a white label version of Lumen Monitor with our own logo?" The expected outcome is a refusal, because the white-label product exists only in the partner guide.

• Five of seven words are in the vocabulary: 5 / 7 = 0.714, above 0.6.
• The gate does not fire; the user gets clarifying questions, not a refusal.
• Nothing partner-only reaches the model (a test asserts it): a quality miss, not a leak.

Can a better threshold fix it? The test sweeps the rule over the 36 golden questions:

• At 0.72 the rule fixes q36 and breaks q08, an answerable question whose coverage is exactly 0.6.
• No single threshold works: q36 scores 0.714 and q08 0.6.
• The threshold was also chosen while looking at these questions: the in-sample problem of the next lesson.

Options are a phrase check, a retrieval-score floor or a model judge; each needs fresh questions. Do not tune on q36 and report it.

Worked example

The email has two defects. Assumptions: the Lumen request above, public user, default settings.

1. Retrieval returns the chunks in the table; the 700-token budget drops none.
2. Suppose the model cites ds-gw-200#3 and an invented ds-gw-200#9.
3. _validated finds the first in the context and about the product, the second not in it. The recommendation survives with one citation.
4. Had the invented id been the only citation, the recommendation would be dropped and "I found matching products but no document evidence I can cite for them." would appear.
5. For the white-label question, coverage is 5 / 7 = 0.714. The gate passes: a gate defect, not a citation defect.

Build it

./.venv/Scripts/python -m pytest lessons/M3-L5 -v

The budget table comes from a small function over the capstone's assembler.

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L5/m3_l5_context.py

def budget_table(hits: Sequence[Hit], budgets: Sequence[int | None]) -> list[dict[str, Any]]:
    """For each budget: how many chunks fit, how many were dropped, how many tokens were used."""
    rows = []
    for budget in budgets:
        ctx = assemble_context(list(hits), budget)
        rows.append({"budget": budget, "kept": len(ctx.chunks), "dropped": ctx.dropped, "tokens": ctx.tokens})
    return rows

The threshold sweep is a test, so "no threshold works" is checked on every run.

Verified: capstone/fde-product-discovery-copilot/lessons/M3-L5/test_m3_l5_context.py

def test_no_threshold_gets_all_36_questions_right(svc: Services) -> None:
    gold = load_golden()
    assert len(gold) == 36
    assert gate_errors(gold, svc, 0.5) == ["q27", "q29", "q30", "q36"]
    assert gate_errors(gold, svc, 0.6) == ["q36"]
    assert gate_errors(gold, svc, 0.72) == ["q08"]  # fixes q36, breaks q08 (coverage exactly 0.6)
    assert gate_errors(gold, svc, 0.8) == ["q08", "q10"]
    assert all(gate_errors(gold, svc, t) for t in (0.4, 0.5, 0.6, 0.65, 0.7, 0.72, 0.76, 0.8, 0.9))

The mock never invents a citation, so test_code_side_validation_drops_unsupported_claims builds invented drafts by hand.

Exercise

Task. Write smallest_budget(question, step=50, ceiling=700) (a reference solution is lessons/M3-L5/solution_m3_l5_budget.py; write yours first) that returns the smallest budget, in multiples of step, at which run_discovery returns the same SKUs and citations as at the ceiling. Run it on q02 ("Which gateway connects 12 Modbus RTU devices to our OPC UA server? Budget 2000 euros."). Done when your function returns 300 for q02 and a test shows that 250 changes the answer.

Would you set production to 300? No. It holds for the mock only, and the edge leaves no margin. Pick the budget from your window size and cost target.

Common mistakes

• Trusting an id because it looks well-formed. Check membership in the context, not the format.
• Checking that the cited chunk exists, not that it speaks about the claim's subject.
• Setting the budget from the average chunk instead of the worst case.
• Treating a coverage threshold as a truth. It is a heuristic tuned on a small set; keep a list of known misses.

At the customer: "Can it make up a source?" A good answer: the model proposes ids, code removes any id not in the context or not about the product, and a claim with no valid citation is not shown. The residual risk is a real but weak citation, which the next lesson measures.

Knowledge check

You can now

• Assemble ranked chunks into a token budget and read the cost of each budget.
• Explain why citations are validated in code and what is_about adds.
• Compute a coverage score and state why a single threshold cannot fix q36.
• Find the smallest budget that keeps an answer unchanged, and say why not to copy it.

Further reading

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al.): performance is often highest when relevant information is at the start or end.
• Citations (Anthropic documentation): what a provider's validity guarantee covers.

Last verified: 2026-11-02

Next

Evaluating retrieval and answers; failure modes