Evaluating AI Systems: Metrics, Judges and Gates

• Swap positions. "A conservative approach is to call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders." Inconsistent results are a tie. Swapping doubles the judge calls, so do it always for release decisions and on a sample for monitoring. Random position assignment is "effective at a large scale with the correct expectations".
• Few-shot examples raised GPT-4's consistency from 65.0% to 77.5%, but "high consistency may not imply high accuracy", they may add new biases, and the longer prompts made API calls 4 times more expensive.
• Chain-of-thought and reference-guided judging cut maths failures from 14/20 to 6/20 and 3/20. With chain-of-thought the judge often "makes exactly the same mistake as the given answers".
• Length and family controls. OpenAI's guidance names position and verbosity bias and recommends controlling for response length; Anthropic's example recommends a different judge model from the generator.

Figure 5. The position-swap mitigation

How to read this: a preference counts only if it survives swapping the order, so a pure position effect becomes a tie rather than a false win.

What this means for architects

• Measure your own judge. The published rates are for old models. Run every pair in both orders and report the share of flips, and pad a few answers with redundant text to see whether the verdict moves. Both cost a small multiple of one run.
• Bias that cancels and bias that does not. A constant bias applied equally to baseline and candidate largely cancels in an A/B comparison. A bias correlated with the change does not: a prompt edit that lengthens answers looks better to a verbosity-biased judge. This is the course's reasoning, not a measured result, and it is why you control length and spot-check with humans.
• Judges cannot grade what they cannot do. For maths, code and facts the judge does not know, give it a reference or use code-based grading.

Under the hood: Calibrating a judge against human labels

Intuition

Before a judge's pass rate counts, check how often its verdict matches a person's on the same items. Raw agreement is not enough, because two raters who both say "pass" most of the time agree often by chance. Cohen's kappa asks how much better than chance the agreement is.

Mechanism

The Stanford information retrieval textbook defines the statistic: kappa = (P(A) - P(E)) / (1 - P(E)), where P(A) is "the proportion of the times the judges agreed" and P(E) is "the proportion of the times they would be expected to agree by chance" (Manning et al., Assessing relevance). Cohen's original coefficient computes P(E) from each rater's own marginal proportions; the textbook pools the two raters' marginals. Artstein and Poesio describe the first as assuming prior distributions "unique to each coder" (2008, section 2.4.3). On the textbook's own example (400 judgments: 300 both yes, 20, 10, 70 both no) both versions give 0.776. The textbook reads the value so: above 0.8 is "good agreement", between 0.67 and 0.8 "fair agreement", below 0.67 "data providing a dubious basis for an evaluation", with the caveat that the precise cutoffs depend on the purpose.

A worked example (illustrative counts). Take 200 items that a human reviewer and the judge both labelled pass or fail.

P(A) = (150 + 24) / 200 = 0.87
judge says pass: 170/200 = 0.85     human says pass: 156/200 = 0.78
P(E) = 0.85 x 0.78 + 0.15 x 0.22 = 0.663 + 0.033 = 0.696
kappa = (0.87 - 0.696) / (1 - 0.696) = 0.174 / 0.304 = 0.572

Raw agreement is 87%, yet kappa is 0.57, below the textbook's 0.67 line. The table shows why: of the 44 items humans failed, the judge caught only 24 (54.5%, with a 95% Wilson interval of about 40% to 68%), and it passes 85% of items where humans pass 78%, overstating the pass rate by 7 points. A gate built on this judge would wave bad answers through. A percentile bootstrap over the 200 items (5,000 resamples; a computation made for this page, not a sourced method) puts the kappa interval at roughly 0.42 to 0.71, a reminder that kappa from 200 items is itself an estimate.

Skewed data distorts kappa the other way. Artstein and Poesio call it the prevalence problem: "if a disproportionate amount of the data falls under one category, then the expected agreement is very high", so chance-corrected agreement "may be low even though" observed agreement is high. With 200 items split 180 / 6 / 6 / 8, observed agreement is 94% and kappa only 0.54. Always report the judge's performance on the rare class, which for gates is the fail class.

After tightening the rubric, measure again on a fresh sample, not the items used to edit it. Suppose a fresh 200 gives 146 both pass, 8 judge-pass-human-fail, 10 judge-fail-human-pass and 36 both fail: P(A) = 0.91, kappa = 0.742, the judge passes 77% against the humans' 78%, and it catches 36 of 44 failures. That is the textbook's "fair" band, usable for trend monitoring with spot checks. Whether it may gate a release alone depends on the bar the requirement sets and the cost of a missed failure.

Figure 6. The life of a judge

How to read this: a judge is never trusted for good, because any change to the judge, prompt or task, and any failed spot check, sends it back to calibration.

What this means for architects

• Put the bar in the requirement. For example: "kappa of at least 0.67 and failure recall of at least 80% on a fresh 200-item sample, re-checked monthly and after any change to judge or prompt". The numbers are yours to set; what matters is that they exist.
• Cost. Human labels are the expensive part. Label a stratified sample that over-represents failures and borderline cases, because the rare class decides the judge's usefulness; the 40% to 68% interval on 44 failures shows how few failures make failure recall hard to pin down. A sample that is not random changes the marginals, so weight each stratum back to its share of the population before computing kappa.
• Hold-out discipline and slices. Calibration items must not be the items used to tune the rubric. A judge that agrees on easy items and not on hard ones looks fine in aggregate, so report agreement per slice.
• Published figures in context. In MT-bench, GPT-4 agreed with human experts 85% of the time without ties, against 81% between humans, with 50% expected for random judges. Applying the kappa formula with 50% as the chance level gives (0.85 - 0.50) / (1 - 0.50) = 0.70. That illustrates the formula on the paper's own baseline; the paper does not report it, but it shows a headline "85% agreement" is 0.70 on the chance-corrected scale.

Under the hood: Pairwise and absolute scoring

Intuition

You can ask a judge "is this answer good?" (absolute) or "which of these two is better?" (pairwise). Absolute scores can be tracked against a fixed bar over time. Pairwise judgments are usually more reliable for choosing between two versions but say nothing about whether either is good enough.

Mechanism

Zheng et al. state the tradeoff. Pairwise comparison "may lack scalability when the number of players increases, given that the number of possible pairs grows quadratically", while single answer grading "may be unable to discern subtle differences between specific pairs" and "absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes". On MT-bench (Table 5a, first turn, non-tie votes) GPT-4 agreed with human experts 85% of the time in both modes, and the authors say single-answer grading "may sometimes perform slightly worse than pairwise comparison and give more tie votes" but "is a more scalable method".

Call counts (illustrative). For k systems and Q questions, absolute grading needs k x Q calls. Pairwise grading with a position swap needs k(k - 1)/2 pairs, two orders each: k(k - 1) x Q calls. With 6 systems and 80 questions that is 480 calls against 2,400. For one candidate against the current version (k = 2) both need 2 x Q calls, but pairwise calls carry two answers each, so about twice the answer tokens.

Reporting a pairwise result (illustrative). Run 400 items in both orders and count a win only when the candidate is preferred in both (the paper's conservative rule): 150 wins, 110 losses, 140 ties. Excluding ties, the candidate wins 150 of 260 decided items, 57.7%, with a 95% Wilson interval of about 51.6% to 63.5%, which excludes 50%. Two cautions: 140 of 400 items (35%) showed no judged difference, so the effect sits in a minority of items, and "better than the baseline" is not "good enough", so it does not replace an absolute bar.

Figure 7. Choosing a scoring mode

How to read this: the decision picks the mode, and both modes end with a person reading a sample of the cases the judge found hard.

What this means for architects

• Default (an opinion). Pairwise with a position swap to choose between model or prompt versions; absolute pass/fail on a written rubric for the release bar and monitoring. OpenAI's guidance favours pairwise or pass/fail "for more reliability", while others prefer graded scores for diagnostics, so check your judge's calibration in each mode.
• Cost. Pairwise cost grows with the number of candidates, so screen with absolute scores first when comparing many.
• History. Absolute scores are comparable over time only while judge, prompt and rubric stay fixed. Pin all three, and run the baseline beside the candidate in every pairwise comparison.
• Ties are data. A high tie rate says the change barely matters to the judge: good news for a refactor, bad news for an optimisation.

Under the hood: Retrieval metrics

Intuition

For each query the retriever returns a ranked list. Three questions matter. Did the relevant documents appear in the top k (recall at k)? How early did the first one appear (reciprocal rank)? Are the most useful documents ranked above the less useful ones (nDCG)? They need labelled queries but no model call, so they are cheap and deterministic.

Mechanism

The Stanford IR book defines recall as "the fraction of relevant documents that are retrieved", tp / (tp + fn); restricted to the top k results it is recall at k. For graded relevance it defines normalised discounted cumulative gain, "designed for situations of non-binary notions of relevance" (Manning et al., section 8.4). With R(j, m) the grade assessors gave the document at rank m for query j, and Z the factor that makes a perfect ranking score 1:

Recall@k = (relevant documents in the top k) / (all relevant documents)
NDCG(Q, k) = (1/|Q|) * sum over queries j of  Z_kj * sum for m = 1..k of (2^R(j,m) - 1) / log2(1 + m)
RR = 1 / (rank of the first relevant result), or 0 if none;  MRR = mean of RR over queries

That section of the Stanford book does not define mean reciprocal rank. The TREC-8 question answering track scored a question as "the reciprocal of the rank at which the first correct response was returned, or 0 if none of the five responses contained a correct answer", and a submission as the mean over questions (Voorhees and Tice, LREC 2000). That track listed five responses; choose your own cutoff and state it.

A worked example (illustrative). One query has four relevant documents with grades 3, 2, 2 and 1 (0 means not relevant). The retriever returns ten results; the relevant ones are at rank 2 (grade 3), rank 4 (grade 2) and rank 7 (grade 1). The other grade-2 document is never retrieved.

Recall@5  = 2/4 = 0.50          Recall@10 = 3/4 = 0.75          RR = 1/2 = 0.50
DCG@10  = (2^3-1)/log2(3) + (2^2-1)/log2(5) + (2^1-1)/log2(8)
        = 7/1.585 + 3/2.322 + 1/3 = 4.4165 + 1.2920 + 0.3333 = 6.0419
IDCG@10 = 7/log2(2) + 3/log2(3) + 3/log2(4) + 1/log2(5)
        = 7 + 1.8928 + 1.5 + 0.4307 = 10.8235
nDCG@10 = 6.0419 / 10.8235 = 0.558

For MRR take three queries whose first relevant results are at ranks 2, 1 and not in the top 10: reciprocal ranks 0.5, 1 and 0, so MRR = 1.5 / 3 = 0.5. Now put the grade-3 document at rank 1 and the grade-2 document at rank 3, leaving the grade-1 document at rank 7. Recall at 5 is still 0.50, but nDCG at 10 rises from 0.558 to 0.816. Recall at k ignores order inside the top k; nDCG rewards putting the most useful document first.

Figure 8. From labelled queries to retrieval metrics

How to read this: the retriever output never touches a language model on this path, so the numbers are repeatable and any change comes from the index, the query or the ranking.

What this means for architects

• Recall at the k you feed the model is a ceiling. If a relevant passage is not among those placed in the prompt, the generator cannot ground its answer in it. Measure recall at the k you actually use, and revisit it when you change the context budget (page 4) or add a reranker (page 5).
• Choose the metric by what is consumed. MRR when the first result matters, recall at k when a set of passages is consumed together, nDCG when grades differ and order matters.
• Labels are incomplete. The IR book notes that for TREC, relevance judgments exist "only for the documents that were among the top k returned" for some participating system. A relevant document nobody labelled counts as a miss, so add labels when you change retrievers, or the new one is penalised for finding what the old one never showed.
• Cost and slices. These metrics run in seconds on thousands of queries, so they are the fast gate for index, chunking and reranker changes, before slower judge-based checks. Slice them: per-tenant recall catches an access-filter or chunking bug that the mean hides.

Under the hood: Generation metrics

Intuition

Once the right passages are in the prompt, the questions change. Is every claim in the answer supported by those passages (faithfulness)? Does the answer address the question asked (answer relevance)? These metrics are reference-free: they need the question, the retrieved context and the answer, not a gold answer.

Mechanism

The RAGAS paper defines them (Es et al., arXiv 2309.15217). Faithfulness: the answer is faithful "if the claims that are made in the answer can be inferred from the context". An LLM extracts a set of statements S from the answer, then judges for each whether it can be inferred from the retrieved context; with V the supported statements, F = |V| / |S|. Answer relevance: an LLM generates n questions from the answer, each is embedded, and the score is the mean cosine similarity to the original question, AR = (1/n) x sum of sim(q, q_i). The paper adds that this "does not take into account factuality, but penalises cases where the answer is incomplete or where it contains redundant information". Context relevance is the fraction of the retrieved context's sentences that an LLM judges crucial to answering the question.

The library documentation, as of 2026-10-22, describes the same ideas: faithfulness is supported claims divided by total claims, and answer relevancy averages cosine similarities between the user input and questions generated from the response, 3 by default (faithfulness, answer relevancy). The documentation says the legacy interface will be deprecated in version 0.4, so pin the version you use.

A worked example (illustrative). An answer splits into 5 statements and the verifier judges 4 supported: F = 4 / 5 = 0.80. For relevance, if three questions generated from the answer have cosine similarities 0.91, 0.88 and 0.79 with the original, AR = (0.91 + 0.88 + 0.79) / 3 = 0.86. The paper's own low-faithfulness illustration names the wrong director for a film when the context names the right one.

Figure 9. Computing faithfulness for one trace

How to read this: faithfulness needs two model stages per answer, one to decompose and one to verify, and it checks the answer against the retrieved passages, not against the world.

Limits to design around.

1. Faithful is not true. The check asks whether the context supports the answer. A stale, wrong or conflicting passage yields a faithful answer that is wrong: the source-failure branch of Figure 3.
2. Answer relevance ignores facts. By the paper's definition a fluent, on-topic, wrong answer can score well, so treat it as a check against evasive or padded answers, not a quality score. The paper does not say how an abstention ("the documents do not say") scores, so test that on your data before using the metric in a system designed to abstain.
3. The metrics are model outputs. The paper used gpt-3.5-turbo-16k and text-embedding-ada-002. On its WikiEval dataset, where each instance compares two answers or two contexts, and where annotators agreed in about 95% of cases (about 90% for answer relevance), the metrics matched humans in pairwise comparisons at 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance. The authors found context relevance "the hardest quality dimension to evaluate". This shows the metrics can pick the better of two answers when one is deliberately degraded; it does not show that a threshold such as "release at 0.9 or higher" is calibrated for your data.
4. Decomposition and verification quality set the ceiling for long answers and hard claims. Calibrate against human labels as in the previous section.

What this means for architects

• Log the components per request: retrieved passage identifiers, the prompt as sent and the answer, so faithfulness can be recomputed after you improve the verifier.
• Use it as a gate and a monitor, not as truth. Faithfulness per slice catches a prompt change that makes the model drift from its sources. Pair it with retrieval metrics and periodic human checks of source correctness.
• Cost. Two judge stages per answer plus embedding calls for relevance: every item in CI, a sample in production.
• Define the response to low faithfulness. The remedy is a behaviour, not a retry: abstain or answer with citations only, route to a human queue, record the case (approval steps: page 6).

Under the hood: Confidence intervals and sample sizes for gates

Intuition

A pass rate measured on n items is an estimate. Run the same system on two different 400-item samples and the rates differ by chance. A gate that compares the point estimate with a threshold treats noise as signal in both directions: it blocks healthy changes and passes bad ones. The fix is to gate on an interval, and to know in advance how large a drop the gate can see.

Mechanism

Page 2 used the normal-approximation interval. This page uses the Wilson interval, whose limits cannot fall outside 0% to 100%. The NIST/SEMATECH e-Handbook gives it as the confidence interval for a proportion, calls an interval with an impossible lower limit "an inferior approach", and says one-sided limits use z for alpha instead of alpha/2 (NIST, section 7.2.4.1):

Wilson limits = ( p + z^2/(2n)  +/-  z * sqrt( p(1-p)/n + z^2/(4n^2) ) ) / ( 1 + z^2/n )

For 180 of 200 passing, the normal approximation on page 2 gives 85.8% to 94.2% and the Wilson interval 85.1% to 93.4% (z = 1.96).

Sample size to detect a drop. NIST gives the sample size for testing whether a proportion has moved from p0 to p1, with delta = |p1 - p0|, significance alpha and power 1 - beta (NIST, section 7.2.4.2):

N >= [ ( z_(1-alpha) * sqrt(p0(1-p0)) + z_(1-beta) * sqrt(p1(1-p1)) ) / delta ]^2     (one-sided; add 1/delta as a continuity correction)

The handbook's example (10% to 20%, one-sided 5%, 10% risk of missing) gives 102, or 112 with the correction. For a quality gate with baseline p0 = 0.92, a drop to p1 = 0.88 (delta = 0.04), alpha = 0.05 one-sided and power 0.80 (z = 1.645 and 0.8416), N >= 323.8, so 324, or 349 with the correction. An exact binomial calculation agrees: at n = 349 the test rejects 5.0% of the time when the true rate is 0.92 and 81.1% when it is 0.88. That formula treats the 92% baseline as known exactly. If baseline and candidate are independent samples of equal size, the noise adds and the standard two-proportion extension needs about 695 items per run for 80% power (an exact enumeration of that two-sample test at 695 per run gives power 0.804 and a 5.0% false-alarm rate). Running both versions on the same items and analysing per-item differences can be tighter (see the worked example).

What a gate can detect (baseline 92%, one-sided 5%, 80% power, baseline treated as known, no continuity correction):

Two different gates. A proof gate passes only when the lower bound clears the threshold, demonstrating the system meets the bar. A regression gate blocks on evidence of a drop against a baseline. The proof gate is demanding: with n = 400, a one-sided 95% Wilson lower bound and a threshold of 0.90, it needs at least 370 passes (92.5%). The chance it passes, by exact binomial arithmetic:

A system whose true rate is 0.92 fails a 0.90 proof gate 60% of the time at n = 400. That is the gate working as designed: 400 items cannot demonstrate 90% for a system two points above it.

Figure 10. A gate decision with three outcomes

How to read this: there are three outcomes, not two, and "inconclusive" is a normal result for a small set, not a reason to rerun until it turns green.

What a pass/fail gate cannot claim.

• Slices. A 30-item slice tested alone detects only drops of about 16 points from a 90% baseline at 80% power. Slices need their own item counts.
• Many slices at once. Test 10 independent slices each at a 5% false-alarm rate and the chance of at least one false alarm is 1 - 0.95^10 = 40.1%. Treat slice results as reports to review, or tighten the per-slice false-alarm rate.
• Independent items. The formulas assume independence. Paraphrases of one question, or many questions about one document, are correlated, so the effective sample is smaller than n.
• Noise beyond sampling. A stochastic model and a judge with its own error add variance the binomial formula omits. Repeat borderline runs and report judge agreement beside the pass rate.
• Severity and shift. A pass rate says nothing about how bad a failure is, and a golden set samples the past; online evaluation covers what it cannot.
• Zero failures. Page 2 shows that zero failures in 80 cases still allows a true rate near 3.7% at 95% confidence.

What this means for architects

• Write the detectable drop into the requirement. "The gate detects a drop of 4 points or more with 80% power" is checkable; "no regressions" is not. Choose n from it.
• Choose the gate deliberately. A proof gate for a feature's first release against a stated bar; a regression gate for later changes, with the proof re-run less often.
• Cost. Detecting half the drop needs roughly four times the items, because delta is squared in the denominator. Weigh judge and labelling cost against the cost of a missed regression.
• Failure behaviour. Inconclusive blocks by default, and the owner chooses between adding items, improving the system or signing a documented risk acceptance. Design slices in from the start, with extra items where traffic is thin but risk is high.

Decision framework

1. If you have no labelled set, then start with 20 to 50 items from real failures and real traffic and add every incident, because a small set of real cases finds more problems than a large invented one; revisit when slices have enough items to give intervals you can act on.
2. If a release decision rests on an LLM judge, then calibrate it on a fresh human-labelled sample and report kappa and failure recall, because raw agreement hides chance and prevalence; revisit when the judge model, rubric or task changes, or a monthly spot check disagrees.
3. If you compare two versions, then use pairwise judging with a position swap on the same items for both, because swapping removes pure order effects and shared items tighten the comparison; revisit when the judge's flip rate rises.
4. If the feature uses retrieval, then report retrieval and generation metrics separately per slice, because one end-to-end score cannot say which stage failed; revisit when the retriever, chunking, index or prompt assembly changes.
5. If a gate blocks merges, then gate on a confidence bound and state the smallest drop it can detect, because a point estimate turns noise into false alarms and missed regressions; revisit when n, the baseline or the slice mix changes.
6. If an agent acts on the world, then grade the end state first, check only policy-relevant steps of the trajectory and report pass^k over repeated trials, because paths vary legitimately while one success hides inconsistency; revisit when the agent gains a new tool or permission.

Comparison of the main options

Failure scenarios to design for

1. The judge returns malformed output. Behaviour: retry once with a format reminder, then record an evaluation error that counts against the run, not as a pass. A run whose error rate exceeds the agreed limit is invalid and cannot gate.
2. The judge model is updated or deprecated by its provider. Behaviour: judge model and prompt versions are stored on every score; a new version triggers recalibration and a re-baseline before any comparison with history.
3. The prompt is tuned until the golden set passes. Behaviour: gating uses a hold-out split nobody tunes against, refreshed from production on a schedule.
4. The aggregate is green while one slice has regressed. Behaviour: the report lists every slice with its n and interval; a slice below the minimum item count is shown as "not measurable" and blocks a change that targets it.
5. Retrieval labels go stale when documents change. Behaviour: labels are versioned with document versions, and labels for changed documents are quarantined and re-reviewed before they count.
6. The judge shares a model family with the system and inflates scores. Behaviour: release decisions use a judge from a different family or a human-confirmed sample, and the gap between judge and human pass rates is tracked.
7. The gate is inconclusive. Behaviour: the change is blocked by default; the owner adds items, improves the change or signs a risk acceptance. Re-running until green is not permitted.

Worked example

A support-answer assistant is gated on a 400-item golden set. The baseline (current production prompt) passes 368 of 400.

1. Baseline interval. 368/400 = 92.0%; Wilson 95% interval (z = 1.96) 88.9% to 94.3%.

2. Candidate interval. Prompt B passes 352/400 = 88.0%, interval 84.5% to 90.8%. The intervals overlap (88.9% to 90.8%), but overlap is not the test for a difference, so do not stop here.

3. Paired comparison on the same items. On 30 items the baseline passed and B failed; on 14 items B passed and the baseline failed. The net difference is (30 - 14)/400 = 0.04. Treat each item's difference as +1, 0 or -1: the mean is 0.04, the sample variance 0.1087, and the standard error of the mean is the square root of 0.1087/400 = 0.0165, so z = 0.04 / 0.0165 = 2.43 and the one-sided p-value is about 0.008. Treating the runs as independent gives a standard error of 0.0212, z = 1.89 and p = 0.029. The paired analysis is tighter when the same hard items sit in both runs, as the counts assume. Decision: block B, since a 4-point drop is detected at the 5% level. This is a normal-approximation check on 44 discordant items, adequate for a gate and no substitute for reading the 30 items B lost.

4. A regression the aggregate hides. A different candidate, C, passes 364/400 (91.0%), one point below baseline and far inside what 400 items can detect (about 3.6 points). All four extra failures fall in the "returns policy" category, which has 30 items: the baseline passed 27/30 and C passes 23/30 (76.7%). The Wilson intervals are 74.4% to 96.5% and 59.1% to 88.2%. With 30 items a one-sided test at 80% power detects only a drop of about 16 points from a 90% baseline, and this drop is 13.3 points, so a slice gate would probably miss it. Detecting 13.3 points from 90% takes about 41 items (49 with the continuity correction), so raise that slice to at least 50 items.

Assumptions: all counts are illustrative; items are independent; one-sided alpha = 0.05 and power 0.80 unless stated; z = 1.96 for two-sided 95% intervals; the baseline is a sample, not a known constant; the discordant counts (30 and 14) are chosen for the example; normal approximations are used where stated. Every figure was recomputed with Python.

Code sample: a CI gate with a confidence bound

This gate fails the build when the one-sided lower confidence bound of a pass rate drops below a threshold, per slice. It needs only the standard library and implements the Wilson lower limit given above.

import json
import math
import sys
from statistics import NormalDist


def wilson_lower(passes, n, confidence=0.95):
    """One-sided lower confidence bound for a pass rate (Wilson score method)."""
    z = NormalDist().inv_cdf(confidence)
    p = passes / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre - margin) / (1 + z * z / n)


def gate(results, threshold, confidence=0.95):
    """results: one bool per golden-set item. Pass only if the lower bound clears the threshold."""
    lower = wilson_lower(sum(results), len(results), confidence)
    return lower >= threshold, lower


def run(by_slice, threshold):
    """by_slice: {"all": [...], "category-a": [...]}. Returns the slices that fail."""
    failed = {}
    for name, results in by_slice.items():
        ok, lower = gate(results, threshold)
        if not ok:
            failed[name] = (len(results), round(sum(results) / len(results), 3), round(lower, 3))
    return failed


if __name__ == "__main__":
    failed = run(json.load(open(sys.argv[1])), float(sys.argv[2]))
    for name, (n, rate, lower) in failed.items():
        print(f"FAIL {name}: n={n} rate={rate} lower bound={lower}")
    sys.exit(1 if failed else 0)

Run it as python gate.py results.json 0.90, where the file maps each slice name to a list of true/false results; CI reads the exit code. Checked by running it: with 400 items, 370 passes (92.5%, lower bound 0.9004) print nothing and exit 0, while 369 (92.25%, lower bound 0.8976) print FAIL all: n=400 rate=0.922 lower bound=0.898 and exit 1, matching the table above. A slice with 27 of 30 passes prints lower bound=0.774 and fails the same threshold, because 30 items cannot demonstrate it.

Two limits are built in. A "fail" covers both "evidence of failure" and "inconclusive" (the conservative default in Figure 10), so report the rate and n beside it. And it gates a bar, not a change: for a candidate against a baseline use the paired comparison in the worked example. Count a malformed judge verdict as False, or exclude the item and enforce the error-rate limit separately; never as a pass.

Common mistakes

• Reporting a judge's raw agreement as its quality. 87% agreement hid a kappa of 0.57 and a failure recall of 55%. Report kappa, failure recall and per-slice agreement.
• Tuning and gating on the same items. Once prompts are edited until the set passes, it no longer measures generalisation. Keep a hold-out.
• Reading overlapping intervals as "no difference". The baseline and candidate intervals overlapped while the paired test detected a drop.
• Gating on the average. A one-point drop in the average hid a 13-point drop in a 30-item category. Report slices with their sample sizes.
• Treating faithfulness as correctness. A faithful answer built on a stale passage is wrong, so evaluate retrieval and sources too.
• Changing the judge and the system in one release. The movement cannot be attributed. Change one at a time, or run the old judge alongside the new.

Try it

Time: 20 to 40 minutes. Use an AI feature you work on, or a toy assistant that answers questions about a document you have.

1. Write 30 golden-set items with at least two slices and a pass/fail outcome from you, and set 10 aside as hold-out.
2. Run the feature twice on the 30 items. Compute the Wilson interval for both runs (the code sample's wilson_lower gives the lower limit; flip the sign for the upper, and pass confidence=0.975 for a two-sided 95% interval). Note how far the two runs differ.
3. Have a colleague, or yourself a day later, relabel 20 items blind. Build the 2x2 table against your first labels and compute observed agreement and kappa by hand, following the worked example.
4. Pick a baseline pass rate and a drop you care about, compute the sample size with the NIST formula above, and say in one sentence whether 30 items can detect it.

Done when you have a table of your items with slices, two interval calculations, one kappa value from a 2x2 table, a sample-size number for a drop you named, and a sentence saying what your current set can and cannot detect.

Key takeaways

• A golden set is seeded from real failures, tagged with slices, labelled against a written rubric, versioned and split into tuning and hold-out parts.
• An LLM judge is a model under test: pin its version, store its raw output, treat parse failures as errors, and calibrate it against human labels on fresh items.
• Position, verbosity and self-preference effects are documented: swap positions for pairwise judgments, control length, prefer a different judge family, and measure your own judge, because the published rates come from 2023-era models.
• Raw agreement misleads. Report Cohen's kappa and the judge's recall on the fail class, and remember that humans disagree too.
• Measure retrieval (recall at k, MRR, nDCG) separately from generation (faithfulness, answer relevance); faithful does not mean true, and answer relevance ignores facts.
• Gate on a confidence bound and write down the smallest drop the gate can detect: about 3.6 points at 400 items from a 92% baseline, about 16 points for a 30-item slice.
• For agents, grade the outcome state first, check policy-relevant steps of the trajectory, and report reliability over repeated trials.

Self-check

Further reading

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track (arXiv v4, December 2023): the three judge variants, the bias definitions and measurements, and the agreement study.
• Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation: faithfulness, answer relevance and context relevance, and the WikiEval comparison.
• Manning, Raghavan and Schuetze, Introduction to Information Retrieval, chapter 8: recall, nDCG and the kappa statistic.
• NIST/SEMATECH, e-Handbook of Statistical Methods, section 7.2.4: the Wilson interval and the sample-size formula.
• Anthropic, Demystifying evals for AI agents, 9 January 2026: tasks, trials, graders, transcripts and outcomes, pass@k and pass^k.

Last verified: 2026-10-22

Next

Part B ends here. Part C starts with LLMOps and Observability for AI Systems, which takes the metrics and gates from this page into prompt registries, tracing, canary releases and cost tagging.