4.1 Eval Design Principles
An eval is a systematic measurement of whether your Claude integration does what it's supposed to do. Before writing a single eval case, define four things:
- Success criteria. What does a correct output look like? (Not "good" — specific and measurable.)
- Failure modes. What are the ways the system can fail, and which are acceptable?
- Grading method. How will you determine if a given output is correct?
- Budget and coverage. How many cases? What edge cases must be included?
The eval precedes the prompt. Define success criteria before writing the system prompt, not after. If you write the prompt first, you'll optimise for outputs that look good to you, not outputs that are actually correct.
Evals are treated as core infrastructure on the exam — not as testing afterthoughts. Questions about eval design appear in nearly every domain. Know the grading methods and when to apply each.
4.2 Grading Methods
Three grading approaches, each suited to different output types:
| Method | Output type | Accuracy | Cost |
|---|---|---|---|
| Exact match | Fixed-form outputs (classification labels, booleans, numeric values) | Perfect | Very low |
| Code-graded | Outputs with verifiable properties (code that must compile, JSON that must match schema) | High | Low |
| LLM-as-judge | Open-ended outputs (summaries, explanations, creative writing) | Calibrated to human judgement | Medium |
Exact match example: Sentiment classification — the output must be exactly "positive", "neutral", or "negative".
Code-graded example: JSON schema validation — use jsonschema to check the output matches the required structure:
import jsonschema
def grade_output(response_text: str) -> bool:
try:
output = json.loads(response_text)
jsonschema.validate(output, schema=EXPECTED_SCHEMA)
return True
except (json.JSONDecodeError, jsonschema.ValidationError):
return FalseLLM-as-judge example: Ask Claude to evaluate another Claude response on specific criteria:
judge_prompt = f"""
You are evaluating an AI-generated summary. Score it 1–5 on each criterion.
ORIGINAL DOCUMENT:
{original}
GENERATED SUMMARY:
{summary}
Evaluate on:
1. Accuracy: Does the summary only contain facts from the original? (1=many errors, 5=fully accurate)
2. Completeness: Does it cover the key points? (1=missing major points, 5=comprehensive)
3. Conciseness: Is it appropriately brief? (1=padded, 5=perfectly concise)
Respond as JSON: {{"accuracy": N, "completeness": N, "conciseness": N, "notes": "..."}}
"""4.3 Building and Calibrating an LLM Judge
An LLM judge is only as good as its calibration. An uncalibrated judge might score all outputs 4/5 regardless of quality (a known failure mode called "judge bias").
Calibration process:
- Build a ground truth set. Have humans score 50–100 examples on your criteria. These are your reference labels.
- Run the judge. Send the same examples through your LLM judge.
- Measure agreement. Calculate correlation between judge scores and human scores. Target >0.8 correlation.
- Iterate on the judge prompt. If correlation is low, examine the disagreements. Add examples to the judge prompt that illustrate the failing cases.
- Test for bias. Check whether the judge systematically favours certain output styles. Check for length bias (longer ≠ better), positivity bias (polite ≠ correct).
Include calibration examples directly in the judge's few-shot context — show it a "bad" example with its correct score alongside a "good" example. Without this, LLM judges tend to inflate scores.
4.4 Testing Levels
Four levels of testing for Claude integrations:
| Level | What it tests | Example |
|---|---|---|
| Unit | Individual prompt + model output in isolation | Does the classification prompt return valid labels? |
| Functional | End-to-end flow through one feature | Does the RAG pipeline return relevant chunks and a correct answer? |
| Integration | Interaction between components | Does the tool use loop pass results correctly between Claude and the database tool? |
| End-to-end | Full user journey in production-like conditions | Does the customer support agent correctly resolve a billing dispute from start to finish? |
Golden set. Maintain a set of representative inputs with known correct outputs. Run this against every model update or prompt change. If the golden set pass rate drops, block the change.
Regression testing. After fixing a bug, add the bug case to your test suite. This prevents the same failure from returning silently in future changes.
4.5 Tracing
Tracing records the full execution of an agent run — every message sent, tool call made, and result received. It is the primary debugging tool for agent failures.
What to trace: