Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Evaluating AI systems.
Context
A regional public agency handles benefit appeals. Each case holds a long thread of letters, forms and call notes. A summarisation feature gives the caseworker a brief of at most 150 words: the facts, dates, amounts, deadlines and what the citizen is asking for. The caseworker reads the brief and opens the source when anything matters. The agency wants to move to a cheaper model.
Problem statement
Nobody could say whether the cheaper model was as good, because nobody agreed what "good" was. Caseworkers wanted short briefs, the legal team wanted every deadline, and the communications team wanted plain language. When two caseworkers rated the same 100 summaries pass or fail, they agreed on 71 (Cohen's kappa 0.37, defined below), so even the human labels were unreliable.
The target: a written rubric, human agreement above kappa 0.8, a judge whose agreement with those humans is measured, and a model-change decision that rests on numbers with intervals.
Constraints
- Fixed: case content contains personal data, so only approved providers receive it, through the model gateway, with redaction.
- Fixed: the caseworker stays responsible; the brief links to its source.
- Limit: reviewers can label about 100 summaries per round.
Why the traditional approach fell short
A template brief built from form fields (claimant, dates, amount, deadline) is exact but cannot read a letter in which the deadline sits in a sentence. Extractive tools that pick the first and last paragraphs miss the middle of long threads. Overlap scores against a reference summary reward wording, not whether a deadline is present. Manual reading gives a verdict per reader, and readers disagreed.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Structured fields plus the first and last paragraph (non-AI) | Kept | The comparison baseline and the fallback when the model is down |
| B. Ship on demo impressions and thumbs up | Rejected | No definition of good, no regression signal |
| C. Rubric, human labels, calibrated judge, pairwise checks for model changes, spot checks | Accepted | Gives a repeatable measure and shows how far to trust it |
| D. Humans review every summary | Rejected | Removes the saving the feature exists for |
Solution architecture
Figure 1. Evaluation loop around the summariser
How to read this: the Eval service reaches the model only through the same gateway as production, and human labels are the only thing that flows back into the labelled set.
Figure 2. A model change, judged in both orders
How to read this: each pair is judged twice with the order swapped, and the judge's verdicts are only used while a human sample still agrees with it.
How AI solved it, step by step
- Cost review proposes a cheaper model. The Worker asks the Eval service to compare it with the current one.
- The Eval service loads 100 sources that were never used to write or tune the judge prompt.
- Both models summarise each source through the Model gateway.
- The judge grades each summary on four yes or no criteria, reasoning first: key facts present, no unsupported statement, the citizen's request stated, at most 150 words in plain language. A summary passes only if all four pass.
- The judge then compares each pair in both orders. A win counts only if the same summary wins in both orders; otherwise the pair is a tie.
- Decision: the Eval service reports the rubric pass rate and the win share among decided pairs, each with a 95% interval.
- Reviewers label 30 judged-pass and 30 judged-fail summaries; the Eval service recomputes kappa against them, weighting the two groups back to their shares of the judge's verdicts because the sample is not random.
- If kappa has dropped below 0.67, the judge is suspended and humans label the round. The Trace store keeps model, rubric and judge-prompt versions.
Under the hood
Cohen's kappa corrects raw agreement for chance: kappa = (p_o − p_e) / (1 − p_e), where p_o is the observed agreement and p_e the agreement expected if each rater labelled at random with their own label frequencies (scikit-learn's documentation, citing Cohen 1960). It is 1 for complete agreement and 0 for no agreement beyond chance. The Stanford IR book gives rules of thumb for relevance judges: above 0.8 good, 0.67 to 0.8 fair, below 0.67 a dubious basis for evaluation, with cutoffs depending on purpose. Its worked example pools label frequencies; this page uses each rater's own.
Worked case: the judge and the consensus labels agree on 84 of 100 (p_o = 0.84). The judge passes 74 and the humans 66, so p_e = 0.74 x 0.66 + 0.26 x 0.34 = 0.5768 and kappa = (0.84 − 0.5768) / (1 − 0.5768) = 0.62. An 84% agreement hides that the judge passes 12 of the 34 summaries humans failed.
Judges are models with documented biases. The MT-Bench paper (Zheng et al.) reports over 80% agreement between a strong judge and humans on chat questions, and describes position bias, verbosity bias and self-enhancement bias; for the last, it says its data could not settle the question. Its mitigation for position is calling the judge twice with the order swapped and counting a win only when both orders agree. That agreement was measured on chat, not on this agency's briefs, so agreement is measured locally. The mechanism is in Under the hood: Calibrating a judge against human labels.
What went wrong / failure modes
- Lenient judge. First calibration: kappa 0.62 and a 12-of-34 false-pass rate. Mitigation: itemised yes or no criteria, reasoning before the verdict, and recalibration on a held-out set.
- Overfitted judge prompt. Tuning on the calibration set inflates its agreement. Mitigation: confirm on 100 summaries the prompt has never seen.
- Order effects. In this run 40 of 100 pairs ended as ties because the verdict changed with the order or both orders said tie. Mitigation: swap and count only agreeing orders. In the paper's test, the strongest judge was consistent in 65.0% of cases.
- Verbosity. Judges can favour longer text, yet the rubric caps length at 150 words. Mitigation: check length in code.
- Judge from the generating family. Self-preference may favour its own outputs. Mitigation: use a different model to judge than to generate, as a code comment in Anthropic's evaluation guide recommends.
Results
All results are illustrative targets, not measurements.
| Measure | Start | Target |
|---|---|---|
| Human agreement | 71 of 100, kappa 0.37 | 91 of 100, kappa 0.82 after the rubric |
| Judge versus humans | kappa 0.62, 12 of 34 false passes | kappa 0.77, 7 of 34 false passes, on a held-out set |
| Candidate model, rubric pass rate | Unknown | 80 of 100 (interval 71.1% to 86.7%), bar: lower bound at least 70% |
| Candidate versus current, pairwise | Opinion | 36 wins to 24 among 60 decided pairs (60%, interval 47.4% to 71.4%) |
The reading: the candidate is not shown to be worse or better.
Tradeoffs and when NOT to use this approach
- Small volume. A few hundred summaries a month can be read by people; skip the judge.
- Decisions about people. When a summary drives a benefit decision, a judge cannot replace a human reading the source.
- No stable rubric. If stakeholders will not agree on criteria, calibration has nothing to measure.
- Close models. With 40% ties, pairwise checks cannot separate them; collect more pairs.
Design rules extracted
- Write the rubric first, and measure human agreement with kappa before building a judge.
- A judge is a model: calibrate it against human labels on data it was not tuned on, and recheck it on a schedule.
- Compare models in pairs, in both orders, and report an interval, not a winner.
- Keep checkable criteria, such as length, in code.
- Keep humans sampling the judge's passes and fails.
Assumptions
All numbers are illustrative.
- Round 1, two raters, 100 summaries: both pass 50, rater 1 only 12, rater 2 only 17, both fail 21. p_o = 0.71; rater 1 passes 62, rater 2 67; p_e = 0.62 x 0.67 + 0.38 x 0.33 = 0.5408; kappa = 0.1692 / 0.4592 = 0.37. Round 2 with the rubric, a fresh 100: 52, 5, 4, 39. p_o = 0.91; passes 57 and 56; p_e = 0.5084; kappa = 0.82.
- Judge against consensus, 100 summaries: both pass 62, judge only 12, human only 4, both fail 22. p_o = 0.84, p_e = 0.5768, kappa = 0.62. Revised judge, held-out 100: 63, 7, 3, 27. p_o = 0.90; judge passes 70, humans 66; p_e = 0.564; kappa = 0.77. False passes 7 of 34.
- Pairwise, 100 sources judged in both orders: candidate wins both orders 36, current wins both 24, order-dependent or tied (ties) 40; 36 + 24 + 40 = 100, decided pairs 60. Rubric pass rates: candidate 80 of 100, current 82 of 100.
- Wilson 95% interval, z = 1.96: 80 of 100 gives 0.7112 to 0.8666; 36 of 60 gives 0.4737 to 0.7143. The 70% bar, the 0.67 kappa floor for the judge (the textbook's rule of thumb, applied as a gate), the 30 + 30 review sample and the sample sizes are choices, not standards. Dropping ties from the pairwise share is a choice. Intervals are two-sided 95%, so the 70% bar is checked against a limit that is effectively a 97.5% one-sided bound, stricter than the one-sided 95% bounds used for gates in the lesson. Kappa here uses each rater's own label frequencies, the same definition as the lesson.
- The agency, the rubric and all counts are invented.
Last verified: 2026-10-22