What you'll learn: how the capstone computes recall@3 and groundedness, how to read its results table without overclaiming, and how to sort every miss into a failure class with a decision tree, using the capstone's five real misses (q01, q02, q07, q09, q36).
At a glance
Figure 1. The evaluation loop
How to read this: metrics tell you that something is wrong, classification tells you where, and the loop changes one thing at a time so the next table can be compared with this one.
- Recall@3 asks whether the expected documents are among the first three the model was given.
- Groundedness asks whether each recommendation is a real, cited product with numbers found in the evidence.
- The capstone's five misses fall into four classes (retrieval, ordering twice, selection, gate): no single fix helps them all.
- All numbers are in-sample regression numbers on 36 questions the author wrote, not an accuracy estimate.
Scenario
Composite scenario, not a real company. The Lumen sales director sees a results table with 94.2% document recall and 100% groundedness and asks whether the assistant is 94 percent accurate. A teammate wants to report it that way. Another wants to fix the one wrong refusal by raising a threshold. Neither has checked what the numbers measure or what the misses have in common. All figures are illustrative.
What you will be able to do
- Compute recall@3 for a question by hand and reproduce the table's 94.2% from per-question values.
- State what groundedness measures and why it reads 100% for every strategy here.
- Classify any miss with the decision tree, and name who owns the fix.
- Explain the results table to a customer in three sentences without overclaiming.
Prerequisites
Metadata filters and tenant-aware access control for the leak count, and Context assembly and citations for the gate and context order. Metric theory (judges, nDCG, confidence intervals) is in Evaluation.
Core ideas
What the capstone measures
| Metric | Question it answers |
|---|---|
| Doc recall@3 | Are the expected documents among the first 3 in the context? |
| SKU recall@3 | Are the expected products among the first 3 recommendations? |
| Groundedness | Is each recommendation real, cited, with numbers found in the evidence? |
| Access leaks | Did partner-only content reach the model or answer? |
| Stale citations | Did the answer cite a forbidden superseded document? |
| Refusal accuracy | Did it refuse exactly when it should? |
Recall is a share of what was expected, so it needs a labelled set.
Verified: capstone/fde-product-discovery-copilot/eval/metrics.py
def recall_at_k(expected: list[str], ranked: list[str], k: int = 3) -> float | None:
"""Share of expected items found in the first k ranked items. None when nothing was expected."""
if not expected:
return None
return len(set(expected) & set(ranked[:k])) / len(expected)For q01 the expected documents are ds-vs-100, compat-opcua and faq-humid-environments; the first three in its context are ds-vs-100, ds-gw-200 and compat-opcua, so 2 / 3 = 66.7%. For q02 (expected ds-gw-200 and compat-opcua) the first three are ds-gw-200, ds-gw-300 and ds-gw-500: 1 / 2 = 50%.
The table's 94.2% is the mean over the 23 questions with expected documents. Twenty score 100%; q01, q02 and q07 lose 1/3, 1/2 and 1/2: (23 - 1.333) / 23 = 94.2%. SKU recall: q09 scores 50%, so (23 - 0.5) / 23 = 97.8%.
Reading the results table
These rows are copied from eval/results/latest.md (strategy B is production):
| Strategy | Doc recall@3 | SKU recall@3 | Groundedness | Refusal accuracy | Access leaks | Stale citations |
|---|---|---|---|---|---|---|
| baseline | 94.2% | 95.7% | 100.0% | 94.4% | 5 | 2 |
| A | 94.2% | 97.8% | 100.0% | 97.2% | 0 | 2 |
| B | 94.2% | 97.8% | 100.0% | 97.2% | 0 | 0 |
- Recall did not move. The strategies differ in safety and staleness, not in finding documents.
- Groundedness is 100% everywhere. The mock composes only from its context, so the metric cannot tell strategies apart.
- The safety counts moved. The in-query filter removes all 5 leaks; excluding superseded documents removes the 2 stale citations.
- The set is small and in-sample. One question is 2.8 percentage points of refusal accuracy.
With a real model every number would need measuring again.
Spot the flaw. A status note reads: "Doc recall is 94.2% and groundedness is 100%, so the assistant is 94 percent accurate and safe to launch. After raising the refusal threshold to 0.72, refusal accuracy will be 100%."