Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Evaluating AI systems.
This page is about evaluation design for an internal procedure-lookup assistant in an invented scenario. It is not medical, clinical or regulatory advice, and it is not guidance for patient care or for choosing a clinical system.
Context
A regional hospital group gives its ward and laboratory staff an assistant that answers questions about internal procedures: specimen labelling, patient-transfer paperwork, equipment cleaning logs. It retrieves passages from the procedure library and writes a short answer with the passage ids it used. It gives no diagnosis or dosing advice, and the staff member remains responsible for following the official procedure.
Problem statement
After launch, staff reported answers that sounded certain and were wrong, such as a labelling rule that the library had replaced a year earlier. The team's only dashboard showed a mean faithfulness score of 0.93, so by that measure the assistant was healthy. "Hallucination" is used here in the everyday sense of a confident wrong answer; the audit splits the causes, and most were not invented text.
A human audit of 200 sampled questions found 38 confident wrong answers (19%, 95% interval 14.2% to 25.0%). The target: measure retrieval and generation separately, bring confident wrong answers to at most 5%, and make the assistant say "I could not find a current procedure" when the evidence is weak.
Constraints
- Fixed: questions and answers contain no patient data; the system prompt forbids clinical advice.
- Fixed: every answer must cite passage ids that were actually retrieved.
- Limit: two procedure owners could label 200 questions with the answering passage.
- Limit: a person must be able to read why the assistant declined.
Why the traditional approach fell short
Keyword search over the intranet returns every version of a procedure with equal rank and leaves the staff member to pick. A rules or FAQ list covers top questions and goes stale as procedures change. A manual review of a few answers catches wording errors but cannot tell whether the answer came from the current document, since the reviewer rarely sees which passage the model saw.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Curated "current procedures" page and better intranet search (non-AI) | Kept | Cheap and exact; becomes the destination of the abstention message |
| B. Keep the faithfulness-only dashboard and tune the prompt | Rejected | Faithfulness is judged against the retrieved text, so a faithful answer from the wrong passage scores well |
| C. Retrieval metrics plus generation metrics, an evidence check with abstention, a sampled judge | Accepted | Locates the failure and stops answering when evidence is weak |
| D. Put the whole library in every prompt | Rejected | Superseded versions stay in the context, so the model still has to choose |
Solution architecture
Figure 1. Two measurement points and an evidence check
How to read this: the two arrows leaving the Eval service point at the part each metric measures, so a drop is traced to retrieval or to generation.
Figure 2. One question, with the abstain path
How to read this: the alt block is the decision to answer or decline, made before the model is called.
How AI solved it, step by step
- A nurse asks how long a labelled specimen may wait before processing. The Client app sends it to the Orchestrator.
- The Orchestrator asks the Retriever for the top 5 passages, restricted to current versions by metadata.
- The Retriever returns passages with ids, version and effective date.
- Decision: the evidence check needs at least one current passage above a relevance threshold tuned on the golden set. If none qualifies, the Orchestrator abstains and links the curated page.
- Otherwise the Model gateway sends the question and passages to the provider, and the model answers with the ids it used.
- The Orchestrator rejects any answer citing an id that was not retrieved.
- The Trace store keeps the ids, versions and outcome. Nightly, the Eval service reruns the golden set for recall and MRR, and judges a sample of 100 traces for faithfulness.
- A weekly human check reads 30 answers and 30 abstentions, to notice declined questions that had an answer.
Under the hood
The Ragas paper (Es et al.) defines three reference-free scores, computed with LLM prompts. Faithfulness: an LLM splits the answer into statements S, checks each against the retrieved context, and F = |V| / |S| for the V statements supported. Answer relevance: an LLM writes n questions from the answer, and AR is the mean cosine similarity of their embeddings to the original question. Context relevance: an LLM extracts the sentences from the context that are needed, and CR is those over all sentences in the context. On the paper's WikiEval set, agreement with human annotators in pairwise comparisons was 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance, and the authors found context relevance the hardest.
Read the definitions as an architect. Faithfulness asks whether the answer follows from the context it was given, not whether the context was right. A faithful answer built on a superseded passage scores 1.0. Context relevance is computed from the retrieved text alone, so it cannot see the passage that was missed. Only a labelled set with the gold passage ids can count that, which is why recall at k and the ranking metrics sit beside these scores. The mechanism is in Under the hood: Generation metrics.
What went wrong / failure modes
- Faithful but wrong. Of the 38 confident wrong answers, 29 were faithful to a wrong or superseded passage and the judge scored them 1.0. Mitigation: retrieval metrics against gold passage ids, and a current-version filter.
- Unsupported claims (hallucination in the narrow sense). The other 9 contained statements in no retrieved passage. Mitigation: the faithfulness judge on a sample, and a citation check.
- Over-abstention. A threshold set too high declines answerable questions. Mitigation: track the abstention rate and read declined questions weekly.
- Judge error. A judge can misgrade, especially on long contexts. Mitigation: compare 60 judged answers with a human, here 54 agree (90%, interval 79.9% to 95.3%), and see the summarisation case for agreement beyond chance.
- Gold set drift. Procedures change. Mitigation: owners re-label when a document version changes.
Results
All results are illustrative targets, not measurements.
| Measure | At audit | Target |
|---|---|---|
| Recall at 5 | 152 of 200 (0.76) | At least 180 of 200 (0.90, lower bound 0.85) |
| MRR at 5 | 0.63 | Report it; watch the trend |
| Confident wrong answers | 38 of 200 (19%) | At most 10 of 200 (5%) |
| Questions without a current passage | Answered anyway | Declined with a link to the curated page |
Tradeoffs and when NOT to use this approach
- Abstention has a cost. Every declined question sends someone to a human or a page. Tune the threshold against both error types.
- Small libraries. With a few dozen procedures, a curated page may beat retrieval.
- Clinical advice. This design suits procedure lookup. A system whose answers guide patient care needs clinical governance beyond this page.
- Judge-only quality. Do not let an uncalibrated judge decide releases.
Design rules extracted
- Measure retrieval and generation separately; a generation score cannot see a retrieval miss.
- Keep gold passage ids for a labelled set, and report recall at k and a rank metric every night.
- Decide when the system must abstain, before the model is called, using metadata and a tuned threshold.
- Treat a judge as a model to calibrate against humans, then sample.
- Cite ids and verify them in code.
Assumptions
All numbers are illustrative.
- Audit set: 200 questions, one gold passage each. Confident wrong: 38 of 200 = 0.19; Wilson 95% (z = 1.96): 0.1417 to 0.2500. Of the 38, 29 faithful to a wrong passage (judged 1.0) and 9 with unsupported claims (29 + 9 = 38). All 38 fall among the 48 questions whose gold passage was outside the top 5; the other 10 were not confidently wrong (the answer was right or hedged), since this audit predates the abstention check.
- Rank of the gold passage: rank 1 for 110 questions, rank 2 for 20, rank 3 for 12, rank 4 for 6, rank 5 for 4, outside the top 5 for 48. Recall at 5 = (110 + 20 + 12 + 6 + 4) / 200 = 152 / 200 = 0.76. Reciprocal rank is 1 / rank of the first relevant passage and MRR is its mean over questions, as the TREC-8 question answering track scored it (Voorhees and Tice, LREC 2000): (110 x 1 + 20 / 2 + 12 / 3 + 6 / 4 + 4 / 5) / 200 = 126.3 / 200 = 0.6315.
- nDCG at 5, from the Stanford IR book: DCG = sum over positions m of (2^grade - 1) / log2(1 + m), divided by the DCG of the ideal order. Grades: 3 current section, 2 supporting, 1 superseded, 0 irrelevant. Retrieved grades [1, 0, 3, 0, 2] give 1 + 0 + 3.5 + 0 + 1.1606 = 5.6606; the ideal [3, 2, 1, 0, 0] gives 7 + 1.8928 + 0.5 = 9.3928; nDCG = 0.6026.
- Target: 180 of 200 = 0.90, Wilson 0.8506 to 0.9343. Confident wrong at most 10 of 200, Wilson 0.0274 to 0.0896.
- Faithfulness mean 0.93 and the judge sample (54 of 60 agree, Wilson 0.7985 to 0.9534) are assumed. The hospital group, procedures and thresholds are invented.
Last verified: 2026-10-22