Baseline vs Strategy Evaluation with a Golden Set

Why doc recall is the same everywhere

All three use the same keyword ranking. A filter removes partner-only chunks and a flag marks superseded ones, but neither reorders the rest. Doc recall is identical for 36 of 36 questions.

So the strategies differ on safety and staleness, not retrieval quality.

Why groundedness is 100% under the mock

The mock provider is a rule-based stand-in for a model. It copies sentences from its context, so no claim falls outside it. The metric works (a test forges a number and the flag turns false); the mock never forges.

Treat 100% as a statement about the mock.

Is one question a difference?

Refusal accuracy is 34 of 36 for the baseline and 35 of 36 for A and B: one question, q33. A Wilson interval puts a range around each rate, and unlike the normal approximation it cannot go below zero (NIST handbook).

Figure 4. Is this gap real?

How to read this: safety counts skip the statistics, and everything else needs an interval before you call it a win.

The widget starts with 94.4% for the baseline (interval 81.9% to 98.5%) and 97.2% for A and B (85.8% to 99.5%). The intervals overlap, so the refusal gain is within noise. Leaks are different: zero is a requirement, not a rate, so 5 to 0 matters even though 34 to 35 does not.

At the customer: asked "Is strategy B better?", say what it fixes (5 leaks, 2 stale citations, 2 unflagged conflicts), what it does not (document recall) and that the set is small and written by the author who tuned the system.

Worked example

Assumptions:

• Scores come from the capstone harness with the mock provider.

A question passes when "refused or not" matches the label. The baseline misses q33 (answered, expected refuse) and q36 (clarified, expected refuse), so 36 - 2 = 34. A and B miss only q36, so 35. In fractions: 34 / 36 = 0.944 and 35 / 36 = 0.972.

The baseline's 5 leaks are three ordinary discovery questions (q08, q10, q12) and two access-control questions (q33, q34): no adversarial question needed, only an unfiltered index.

Build it

The lesson code works on the dictionary evaluate returns. compare_refusals adds the intervals.

Verified: capstone/fde-product-discovery-copilot/lessons/M6-L2/m6_l2_compare.py

def compare_refusals(result: dict[str, Any]) -> list[dict[str, Any]]:
    """One line per strategy; the first strategy is the baseline. 'within noise' means the intervals overlap."""
    lines: list[dict[str, Any]] = []
    base: tuple[float, float] | None = None
    for name, data in result["strategies"].items():
        passes, n = refusal_passes(data["rows"])
        low, high = wilson_interval(passes, n)
        base = base or (low, high)
        verdict = "baseline" if not lines else ("within noise" if low <= base[1] and high >= base[0] else "different")
        lines.append({"strategy": name, "passes": passes, "n": n, "rate": passes / n, "lower": low, "upper": high, "verdict": verdict})
    return lines

fingerprint hashes every number that must repeat, after without_latency drops the one column that varies.

Verified: capstone/fde-product-discovery-copilot/lessons/M6-L2/m6_l2_compare.py

def fingerprint(result: dict[str, Any]) -> str:
    """A hash of every number that must repeat exactly. Latency is excluded because it never does."""
    text = json.dumps(without_latency(result), sort_keys=True)
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]

Printing compare_refusals for a fresh run gives this real output:

baseline 34/36 94.4% 81.9% to 98.5% baseline
A 35/36 97.2% 85.8% to 99.5% within noise
B 35/36 97.2% 85.8% to 99.5% within noise

Exercise

Add a strategy C: B with the context budget off. Run all four through evaluate, then use questions_that_moved(result, "B", "C") and the metrics to say what the budget does here.

Acceptance criteria:

• C has context_budget=False and the other switches as in B.
• You report which metrics changed and which questions moved.
• You say whether the data justifies keeping the budget.

Done when your answer names its evidence and what 36 short questions cannot show.

Common mistakes

• Comparing averages only. Diff the rows.
• Changing the scorer between runs. The numbers stop being comparable.
• Reading a one-question gap as a win. On 36 questions one question is 2.8 percentage points.
• Averaging safety away. A rate hides a single leak. Report leaks as counts.

Knowledge check

You can now

• Say what each results column measures.
• Name the questions each strategy moved.
• Say why recall and groundedness cannot separate these strategies.
• Use a Wilson interval to judge a small gap.

In the interview: "How would you compare two retrieval strategies for a customer?" A good answer: fix the golden set and scorer first, run a simple baseline, change one idea per strategy, compare per question, add an interval for small samples, report safety failures as counts, and say what the experiment cannot show.

Further reading

• NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.4.1, on the Wilson interval.
• Anthropic, Create strong empirical evaluations, as of 2026-11-02.
• AI Architect: Evaluation, on regression gates and sample sizes.
• Optional next step: Model-graded evaluation, when answers are free text and exact match cannot score them.

Last verified: 2026-11-02

Next

Latency, tokens, cost and failure capture