What you'll learn: when a model can grade free-text answers, how to write a fixed-label rubric, and why you check the judge against human labels first. No real model was run: the page states no real-model numbers.
At a glance
Figure 1. Where the judge sits in the evaluation loop
How to read this: the judge is one more model call, and only the human labels tell you whether its labels mean anything.
- Use a model judge only where exact match fails, with two labels.
- A judge is a model under test: check it against human labels.
- Agreement needs a baseline: an always-"correct" judge can score high.
- You set the threshold and sample size; the gate refuses a judge below them.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply's copilot answers in free text, so exact match fails: "Warranty is 24 months" and "The warranty is two years" are the same answer. The team wants a model to grade answers after every change. The FDE's condition: the judge first grades 20 answers people already graded.
What you will be able to do
- Choose code, human or model grading.
- Write a fixed-label rubric and a judge that returns a label or
invalid. - Compute agreement and baseline, and block a weak judge.
Prerequisites
- Golden question set, Evaluation report, Structured output.
- Kappa and intervals: Evaluation.
Core ideas
Pick the cheapest grader that works
As of 2026-11-03, Anthropic's evaluation guide ranks code grading fastest and most reliable, human grading slow and expensive, model grading fast and flexible: "test to ensure reliability first, then scale". Check it first.
| Answer to grade (Lumen) | Grader | Why |
|---|---|---|
| "Which port does the console use?" | Code: string match on 8443 |
Exact and free |
| "How long is the warranty?" "24 months" or "two years" | Model judge, after calibration | Wording varies |
| A price or safety condition the customer acts on | Judge, plus a person reads a sample | A wrong grade costs more |
A rubric with fixed labels
The guide advises clear rubrics and a fixed output such as 'correct' or 'incorrect'. Any other reply becomes invalid: a disagreement, never a default.
Verified: capstone/fde-product-discovery-copilot/lessons/M6-L5/m6_l5_judge.py
RUBRIC = (
"TASK: judge_answer\n"
"You grade one ANSWER against one REFERENCE for a product question.\n"
"Reply with JSON: {\"label\": \"correct\" or \"incorrect\", \"reason\": one sentence}.\n"
"correct: every fact in the REFERENCE is stated, in any wording, and no fact is added that the REFERENCE lacks.\n"
"incorrect: a REFERENCE fact is missing or changed, an unsupported fact is added, or the answer declines to answer.\n"
"Use only the REFERENCE. Do not reward length or confidence."
)
class Verdict(BaseModel):
label: str
reason: str = ""
def build_messages(question: str, reference: str, answer: str) -> list[Message]:
body = json.dumps({"question": question, "reference": reference, "answer": answer})
return [Message("system", RUBRIC), Message("user", body)]
def judge(provider: LLMProvider, question: str, reference: str, answer: str) -> str:
"""One label from LABELS, or INVALID. A bad reply is a visible disagreement, never a silent default."""
reply = provider.complete(build_messages(question, reference, answer), Verdict.model_json_schema())
try:
label = parse_json_model(Verdict, reply.text).label
except ValueError: # json.JSONDecodeError and pydantic ValidationError are both ValueErrors
return INVALID
return label if label in LABELS else INVALIDJudges have known biases
| Bias | What Zheng et al. report (§3.3) | Check for Lumen |
|---|---|---|
| Position | A tendency to favour certain positions; a mitigation is to ask twice with the order swapped and accept a win only if it holds both times (§3.4). | Not applicable: one answer at a time |
| Verbosity | Favouring longer answers, even when less clear or accurate | Pad answers with a no-fact sentence; do labels change? |
| Self-enhancement | A judge may favour its own answers | Grade with a different model than the writer |
| Limited reasoning | Limited maths and reasoning ability | Check sums with code |
Anthropic's guide agrees with the self-enhancement row: use a different model from the writer. The paper tested 2023 models; measure yours.
Verified: capstone/fde-product-discovery-copilot/lessons/M6-L5/m6_l5_judge.py
PADDING = " This is a standard item from the Lumen range and is widely used on industrial sites."
def padding_flips(judge_label: Callable[[str], str], answers: Sequence[str]) -> list[int]:
"""Positions whose label changes when a sentence with no new facts is appended: a verbosity probe."""
return [i for i, a in enumerate(answers) if judge_label(a) != judge_label(a + PADDING)]Agreement is the test, and it needs a baseline
Figure 2. Calibrating a judge before use
How to read this: the loop runs on human labels fixed in advance; only a pass lets the judge grade anything you report.
Predict first. A judge always answers "correct". Humans labelled 14 of 20 answers correct. What agreement does it get?
Answer
14 of 20: 70 percent. When one label dominates, high agreement proves little.
- Agreement: the share of answers where judge and human labels match.
- On MT-bench first-turn questions, without ties, Zheng et al. report 85 percent between GPT-4 and humans and 81 percent between humans (§4.2). That task compares two chat answers: do not copy 85 as your threshold.
Verified: capstone/fde-product-discovery-copilot/lessons/M6-L5/m6_l5_judge.py