flowchart TD
accTitle: Honesty checklist for one claim
accDescr: For each claim, check four things in order. Is the number copied from a results file? Does it show the count and the total? Does it say what it was measured on, including the mock provider and the in-sample set? Is every miss behind it listed? If any answer is no, fix the claim or remove it.
A["One claim in the draft"] --> B{"Number copied from a results file?"}
B -- no --> X["Fix or remove the claim"]
B -- yes --> C{"Count and total shown?"}
C -- no --> X
C -- yes --> D{"Says what it was measured on?"}
D -- no --> X
D -- yes --> E{"Related misses listed?"}
E -- no --> X
E -- yes --> OK["Keep the claim"]
How to read this: a claim that fails any check is fixed before it is quoted.
Worked example
Assumptions: every number is copied from eval/results/latest.md; the customer is fictional.
- Method. 36 hand-written questions (23 answer, 9 refuse, 4 clarify), three strategies, the mock provider.
- Results (strategy B, which production runs). Document recall@3 94.2%, SKU recall@3 97.8%, refusal accuracy 35 of 36, clarify accuracy 4 of 4, access leaks 0 questions (baseline: 5), stale citations 0 (baseline: 2), conflicts flagged 2 of 2. Groundedness is 100%, which reflects the mock.
- Misses q01, q02, q07, q09. The expected document missed the first three (q01: absent; q02, q07: fourth); in q09 one of two licences was not recommended.
- Miss q36. A white-label request got clarifying questions instead of a refusal. Its coverage score, 0.714, is above the 0.6 threshold. Nothing partner-only leaked.
- Limits. In-sample, small, mock provider, keyword search.
- Recommendation (judgement). A public-only pilot on B; before any quality claim, add real requests and a real provider.
Build it
The harness lists the misses; you supply the reasons.
Verified: capstone/fde-product-discovery-copilot/eval/run_eval.py
misses = [r for r in result["strategies"].get("B", {"rows": []})["rows"] if r["outcome"] != r["expected_outcome"]
or (r["sku_recall_at_3"] is not None and r["sku_recall_at_3"] < 1)
or (r["doc_recall_at_3"] is not None and r["doc_recall_at_3"] < 1)]
for r in misses:
lines.append(f"- {r['id']} ({r['kind']}): outcome `{r['outcome']}` (expected `{r['expected_outcome']}`), "
f"SKU recall {_pct(r['sku_recall_at_3'])}, doc recall {_pct(r['doc_recall_at_3'])}")
Strategy B's five misses:
- q01 (discovery): outcome `answer` (expected `answer`), SKU recall 100.0%, doc recall 66.7%
- q02 (discovery): outcome `answer` (expected `answer`), SKU recall 100.0%, doc recall 50.0%
- q07 (discovery): outcome `answer` (expected `answer`), SKU recall 100.0%, doc recall 50.0%
- q09 (discovery): outcome `answer` (expected `answer`), SKU recall 50.0%, doc recall 100.0%
- q36 (access_control): outcome `clarify` (expected `refuse`), SKU recall n/a, doc recall n/a
Template:
EVALUATION REPORT: <system>, <date>
Decision this report informs:
METHOD
Questions: <n>, written by <who>, frozen on <date>. Kinds: <counts>
Strategies compared: <names and switches>
Provider: <mock or real, model id from docs>; search: <type>
Scoring: <file>; reproduce with: <command>
RESULTS (copied from <file>)
<metric>: <count> of <total> (<rate>), interval <low> to <high>
Safety counts: leaks <n>, stale citations <n>
MISSES (one line each: id, what happened, why)
LIMITS
In-sample: <how it was tuned>. Size: <n>. Provider: <stand-in?>. Other: <list>
RECOMMENDATION (labelled as judgement) and what to measure next
Spot the flaw. Find the problems.
Our assistant reaches 97.2% accuracy on a 36-question test set and is ready for production. Documents are found 94.2% of the time, answers are 100% grounded, there are no access leaks, and latency averages 16 ms.
Answer
"Accuracy" is refusal accuracy, with no count. In-sample and mock go unmentioned. "100% grounded" is a property of the mock, "no access leaks" covers 36 public-role questions and is not an audit, and 16 ms is plumbing. No misses are listed.
Exercise
Write the one-page report for strategy B, using python -m eval.run_eval --out <scratch folder>.
Acceptance criteria:
- Every number appears in the scratch
latest.md or latest.json, as a count with its total.
- All five misses have a short reason.
- Limits name in-sample, size, the mock and keyword search.
Done when a colleague finds every number in the files and no claim fails the checklist.
Hint 1: reasons come from the rows
In latest.json, see context_docs.
Hint 2: the in-sample sentence is specific
Name what you tuned.
Common mistakes
- Leading with one percentage. Give the count and sample size.
- Omitting the misses. Readers who find them distrust the rest.
- Hiding the mock. Name the provider in the method.
- Judgement as a result. Label the recommendation.
Knowledge check
1. Why show "35 of 36" and not only "97.2%"?
The count shows the sample size and that the result is one question from perfect.
2. Why list the misses by name in a report for a customer?
No misses suggests a set tuned to pass, and a miss the customer finds costs more trust than one you disclosed.
3. Which limits must the capstone's report state, and why?
In-sample, small, mock provider and keyword search. In-sample is optimistic, 36 questions are noisy, and the mock makes groundedness uninformative.
4. Why copy the numbers from the results file instead of retyping them?
Retyping can change a result unnoticed; copying lets anyone match report to file.
5. The customer wants "ready for production" in the summary. What do you write?
Write what the evidence supports: B suits a public-only pilot, with 0 leaks on the golden set. Name what is unmeasured.
You can now
Further reading
Last verified: 2026-11-02
Next
Module M6 ends the Build course. The AI Solution Architect course starts with The discovery workshop, where you work with the customer before a line of code is written.