What you'll learn: how to write an evaluation question set by hand, with answerable, unanswerable, access-controlled and vague questions, and how to run a tested coverage check that fails when a kind is missing or mislabelled.
At a glance
Figure 1. Labelling one question for the golden set
How to read this: you label a question by what you find when you search, not by what you hope the system will do.
- A golden set lists questions with the outcome you expect, written before you see any score.
- Write it by hand from real requests; a generated set only measures the system's own habits.
- Include four kinds: answerable, unanswerable, access-controlled and ambiguous.
- Prove "unanswerable" by searching, and let a coverage check stop drift into easy questions.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply is a fictional distributor with about 30 product documents, some partner-only. A sales engineer pastes eight customer questions from old emails into a spreadsheet. Each has a clear datasheet answer. The team reports 8 of 8 correct, and the head of sales engineering asks, "how good is it?" None is about warranty, partner prices or a vague request.
What you will be able to do
- Write golden questions by hand, each with a kind and an expected outcome.
- Prove an unanswerable question is unanswerable by searching the corpus.
- List exactly what an access-control question must not leak.
Prerequisites
Metadata filters and tenant-aware access control, Evaluating retrieval and answers; failure modes and AI Architect: Evaluation.
Core ideas
Four kinds of question, one table
Figure 2. The capstone's 36 questions by kind
| Kind | Count | Expected outcome | What it tests |
|---|---|---|---|
| discovery | 12 | answer | A vague request becomes products plus evidence |
| factual | 7 | answer | One fact from one datasheet |
| conflict | 2 | answer, with a disagreement flagged | Two documents contradict each other |
| superseded | 1 | answer from the current document only | An old datasheet must not be cited |
| ambiguous | 4 | clarify | Ask a question instead of guessing |
| unanswerable | 6 | refuse | The documents are silent |
| access_control | 4 | refuse (3), answer (1) | What a public caller must not get |
How to read this: 23 questions expect an answer, 9 a refusal and 4 a clarifying question, so 13 of 36 test the paths a demo never shows.
How you label a question
Figure 3. Which outcome to expect
How to read this: the order of the three questions matters, because a partner-only answer to a vague question is still a vague question.
Here is one real entry, the warranty question. No document has warranty text, and the entry says so.
{
"id": "q29",
"kind": "unanswerable",
"role": "public",
"question": "What warranty period applies to the LUM-CT-200?",
"expected_outcome": "refuse",
"expected_skus": [],
"expected_doc_ids": [],
"forbidden_doc_ids": [],
"forbidden_skus": [],
"expect_conflict": false,
"why": "Warranty terms are not documented."
}Prove the silence
An unanswerable question claims the corpus is silent.
Verified: capstone/fde-product-discovery-copilot/lessons/M6-L1/m6_l1_golden_coverage.py
def silent_terms(terms: list[str], data_dir: Path = ROOT / "data") -> list[str]:
"""Terms that appear in NO document and NO product record: the evidence that a question is unanswerable."""
corpus = " ".join(p.read_text(encoding="utf-8") for p in (data_dir / "docs").glob("*.md"))
corpus += (data_dir / "products.json").read_text(encoding="utf-8")
return [t for t in terms if t.lower() not in corpus.lower()]Access control: the answer exists, but not for this caller
Question q36 asks about a white-label version of Lumen's monitoring software. The partner guide covers it, so a public caller must get a refusal. The entry lists that guide and the partner SKUs as forbidden, or the grader cannot tell a refusal from a leak.
At the customer: a reviewer asks, "How do you know a public user cannot get partner content?" Point at named golden questions and the leak counter in the results, not at a claim that the filter is correct.
Worked example
Assumptions:
- Counts come from the capstone's
data/eval/golden_set.json. - The customer and the emailed questions are fictional.
The kinds add up: 12 + 7 + 2 + 1 + 4 + 6 + 4 = 36. Refusals and clarifications: 6 unanswerable + 3 access-control refusals + 4 ambiguous = 13. The share is 13 / 36 = 36%, above the 25% minimum.
A system that answers everything scores 8 of 8 on the emailed questions, but only 23 of 36 on this set: it also answers q29 and the 12 other questions that expect a refusal or a clarification.
Build it
The rules sit in one table.
Verified: capstone/fde-product-discovery-copilot/lessons/M6-L1/m6_l1_golden_coverage.py
REQUIRED_KINDS: dict[str, tuple[tuple[str, ...], int]] = {
"discovery": (("answer",), 10),
"factual": (("answer",), 5),
"conflict": (("answer",), 2),
"superseded": (("answer",), 1),
"ambiguous": (("clarify",), 3),
"unanswerable": (("refuse",), 5),
"access_control": (("refuse", "answer"), 4),
}
MIN_NON_ANSWER_SHARE = 0.25 # refuse + clarify questions, as a share of the whole setPer question, coverage_problems checks the label against the evidence: a refusal with expected evidence contradicts itself, an answer with none cannot be graded, and an access-control question with nothing forbidden cannot catch a leak.
Verified: capstone/fde-product-discovery-copilot/lessons/M6-L1/m6_l1_golden_coverage.py
has_evidence = bool(item.get("expected_skus") or item.get("expected_doc_ids"))
if outcome in ("refuse", "clarify") and has_evidence:
problems.append(f"{qid}: expects '{outcome}' but lists expected evidence; nothing should be found")
if outcome == "answer" and not has_evidence:
problems.append(f"{qid}: expects an answer but lists no expected SKU or document")
if kind == "access_control" and not (item.get("forbidden_doc_ids") or item.get("forbidden_skus")):
problems.append(f"{qid}: an access-control question must list what must not leak")Real output, run from capstone/fde-product-discovery-copilot:
./.venv/Scripts/python -m pytest lessons/M6-L1 -v
lessons\M6-L1\test_m6_l1_golden_coverage.py ......... [100%]
============================== 9 passed in 0.78s ==============================Spot the flaw. A colleague adds this entry. Which defect will the coverage check report, and which will it miss?
{
"id": "q40",
"kind": "unanswerable",
"role": "public",
"question": "What warranty period applies to the LUM-CT-200?",
"expected_outcome": "answer",
"expected_skus": ["LUM-CT-200"],
"expected_doc_ids": ["ds-ct-200"],
"forbidden_doc_ids": [],
"forbidden_skus": [],
"expect_conflict": false,
"why": "The controller datasheet should say."
}Answer
The check reports the kind and outcome mismatch ("q40: a 'unanswerable' question should expect ('refuse',), not 'answer'"). It misses that the listed datasheet never mentions warranty, so the right entry is refuse with empty evidence, as in q29.
Exercise
Copy data/eval/golden_set.json to a scratch file and add six wireless-product questions: two unanswerable, one access-control, one ambiguous, two answerable.
Acceptance criteria:
- Every new question has all ten fields and a unique id.
silent_termsreturns every term you call missing for each unanswerable question.- The access-control question lists forbidden documents or SKUs that exist.
coverage_problemsreturns an empty list for the extended set.
Done when the extended set passes and you can name, for each new question, the file you searched to choose its label.
Hint 1: start from the documents, not the questions
Open compat-wireless.md and the wireless datasheets. Note what they omit.
Hint 2: the silent term is the noun, not the question
For "How long does the LUM-VS-200 take to ship to Chile?" test "ship" and "Chile" with silent_terms. A term not returned appears in some document, so pick another question.
Common mistakes
- Writing only questions the system gets right. Keep a quarter expecting a refusal or a clarification.
- Labelling unanswerable by feel. Search, and keep the search as a test.
- Forgetting what must not leak. A leak then looks like a correct answer.
- Editing questions after seeing results. Never reword a question to hide a miss.
Knowledge check
1. Why must every unanswerable question be checked against the documents?
Unanswerable is a claim about the corpus, and you can be wrong. If a document does answer it, the system is penalised for being right.
2. What does an access-control question need that an ordinary question does not?
The list of documents and SKUs that must not reach the caller. The grader counts leaks from it; without it a partner-only answer to a public user looks correct.
3. A golden set has 40 questions and all of them expect an answer. What is wrong?
It cannot test the refusal or clarifying paths, so a system that answers everything scores perfectly. The coverage check rejects a set where fewer than a quarter expect a refusal or a clarification.
4. Why is the capstone's set a regression check and not an accuracy estimate?
One author wrote the questions and tuned the pipeline while looking at results, so they are in-sample, and 36 is small. A score shows whether a change made these cases worse, not how unseen requests fare.
5. Question q36 is a known miss. Is it a bad question?
No. It is a correct label the system fails: it asks clarifying questions where it should refuse a public caller. Keeping it unchanged keeps the set honest.
You can now
- Write a hand-labelled golden question with a kind, outcome and evidence.
- Prove a question is unanswerable with a search kept in a test.
- List what must not leak for an access-control question.
- Run a coverage check and say why the set is a regression check, not an accuracy figure.
In the interview: "How would you build an evaluation set for a RAG assistant with no production data?" A good answer: collect real requests, write them by hand, add refusal, access-control and vague cases on purpose, prove the unanswerable ones, check coverage in code, freeze the set before measuring, and say it is small and in-sample.
Further reading
- Anthropic, Create strong empirical evaluations, on task-specific evals, as of 2026-11-02.
- pytest, How to use fixtures.
- AI Architect: Evaluation, golden sets as a regression gate.
Last verified: 2026-11-02