Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Evaluating AI systems.
Context
A home-and-electronics retailer runs a shopping assistant in its app and on its website. It answers pre-purchase questions (compatibility, sizing, delivery) and after-sales questions (returns, warranty, repairs) across eight product categories, about 20,000 conversations a day. A small team owns the system prompt, which ships through a repository and release pipeline.
Problem statement
The team changed the prompt to "answer in at most three sentences, in a friendly tone" after reading ten sample chats. Two weeks later, tickets about wrong return-window answers rose, yet the average thumbs-up rate had not moved.
Answers had lost conditions: "within 30 days" survived, but the exclusion for used items was cut for length. On the team's labelled set, returns and warranty fell from 46 of 50 passing (92%) to 33 of 50 (66%), while the overall result moved from 360 of 400 (90%) to 352 of 400 (88%). The target: a category clearly below its 85% floor blocks release, whatever the overall score.
Constraints
- Fixed (policy): return and warranty conditions must be stated completely whenever an answer concerns them.
- Fixed: model calls leave only through the model gateway, to an approved provider.
- Limit: reviewers could label about 50 items per category this quarter.
- Limit: the gate must finish inside a 20-minute pipeline stage; estimate run cost with the LLM cost estimator.
Why the traditional approach fell short
Unit tests assert exact outputs, but free text has many correct wordings, so such tests are brittle or vacuous. A keyword rule can check that "30 days" appears, not that the exclusion does. A manual spot check of ten chats is tiny: at 10% of traffic, the chance that at least one of them is in the category is 1 - 0.9^10 = 65%, and one chat says nothing about a rate. An aggregate dashboard averages categories, so a 26-point fall in one (12.5% of the set) dilutes to about 3 points and is offset by gains elsewhere.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Manual spot check before release (non-AI, status quo) | Rejected | Small sample, not repeatable, no record |
| B. One overall golden set, tolerance of 3 points | Rejected as the only gate | 88% versus 90% passes; seeing a 2-point drop needs about 1,471 items |
| C. Golden set per category, interval-based gate, then a canary | Accepted | Catches a category drop before release and checks live traffic after |
| D. Render returns and warranty answers from the policy table, model only routes (simpler, non-AI) | Kept as a fallback | Testable with exact assertions, at the cost of flexible phrasing |
The default is C. If a category cannot hold its floor after two rounds of prompt work, D is the better design for it.
Solution architecture
Figure 1. Release pipeline with a per-category gate
How to read this: the Eval service is the only box that can say block, review or pass, and its inputs are the stored golden set and the canary signals.
Figure 2. One prompt change through the gate
How to read this: the alt block is the gate's three outcomes, and only the last one reaches production, behind a canary.
How AI solved it, step by step
- The author opens a change making prompt v14 shorter and friendlier; the Worker starts a gate job and asks the Eval service to run v14 against the baseline v13.
- The Eval service loads 400 items: eight categories of 50, each with a question and the facts a correct answer must contain, such as the window and the exclusion.
- Each item goes through the Model gateway with the candidate prompt, same model and settings as production.
- Each answer is graded pass or fail on its required facts by a judge whose agreement with human labels is checked separately (see the summarisation case).
- The Eval service computes a pass rate and a 95% interval per category.
- Decision: returns and warranty scores 33 of 50, upper bound 77.6%, below the 85% floor. Verdict: block, naming the category.
- The author restores the conditions in v15. Returns and warranty now scores 45 of 50 (90%), at or above the floor: pass.
- The Orchestrator serves v15 to 5% of traffic for three days. The Eval service compares the human-escalation rate per category from traces, then promotes or rolls back by configuration change.
Under the hood
The interval is the Wilson score interval as given in the NIST/SEMATECH e-Handbook: for observed proportion p, sample size n and normal quantile z, the limits are (p + z²/2n ± z·√(p(1−p)/n + z²/4n²)) / (1 + z²/n). For 33 of 50 and z = 1.96 that is 52.2% to 77.6%. The handbook notes that the common p ± z·√(p(1−p)/n) form can give an impossible limit (a negative lower limit) and calls such an approach inferior.
The sample size uses the handbook's formula for testing a proportion, N ≥ ((z₁₋α·√(p0(1−p0)) + z₁₋β·√(p1(1−p1))) / δ)², with δ = |p1 − p0| and a continuity correction of 1/δ. For a failure rate rising from 8% to 20% (one-sided α = 0.05, power 80%) it gives 42.6, so 43 items, or 51 corrected. Fifty items per category roughly suffice for a 12-point drop (43 uncorrected, 51 corrected), not a 6-point one (about 152). A 2-point overall drop, 10% to 12% failures, needs about 1,471 items (1,521 corrected), so a 400-item overall gate cannot see it. The formula treats the baseline as fixed, a simplification.
The mechanism is covered in Under the hood: Confidence intervals and sample sizes for gates.
What went wrong / failure modes
- Gate too small. Fifty items cannot show a 6-point drop. Mitigation: document the minimum detectable drop and grow high-risk categories first.
- False alarms. If each of eight independent checks has a 5% chance of a false alarm, at least one fires with probability 1 - 0.95^8 = 34%. Mitigation: a review lane for borderline results, not an automatic block.
- Tuning on the gate set. Editing the prompt until 400 items pass overfits it. Mitigation: a held-out set that authors never see, and items refreshed from production chats each quarter.
- Canary too thin. A 5% canary sees about 100 returns chats a day. Three days give 300, enough to separate 8% from 20% escalation (intervals 5.4% to 11.6% and 15.9% to 24.9%) but not 8% from 12%. Mitigation: write down the shift the canary can see and extend it for smaller ones.
Results
All results are illustrative targets, not measurements.
| Measure | Before | Target after |
|---|---|---|
| Release check | Ten chats read by one person | 400 graded items, per-category verdict |
| Returns and warranty pass rate | 33 of 50, shipped unnoticed | Blocked, then 45 of 50 |
| Time to detect a category drop | About two weeks, via tickets | One pipeline run, under 20 minutes |
Tradeoffs and when NOT to use this approach
- Low traffic. With a few hundred chats a month, read every low-rated chat instead.
- High-stakes answers. If a wrong condition has legal weight, use option D, deterministic text.
- Unreliable grading. If reviewers or the judge disagree often, the gate measures noise. Calibrate first.
- Rare events. They need more items than a gate stage can afford; monitor them online.
Design rules extracted
- Gate each category, not only the average.
- Decide what the gate may claim: a block says a category is clearly below its floor; a pass says only that it was not shown to be below.
- Size the set for the smallest drop you must catch, and write that number down.
- Keep the tuning set and the gate set apart.
- Follow the offline gate with a canary segmented by category, and put the prompt version on every trace.
Assumptions
All numbers are illustrative.
- Traffic: 20,000 chats a day, 10% in returns and warranty (2,000 a day). At least one of ten random chats in it: 1 - 0.9^10 = 0.651.
- Golden set: 8 x 50 = 400 items. Before: overall 360 of 400 = 0.90, category 46 of 50 = 0.92, others 314 of 350. After v14: overall 352 of 400 = 0.88, category 33 of 50 = 0.66, others 319 of 350 (319 + 33 = 352). Weight 50/400 = 12.5%; 0.26 x 0.125 = 3.25 points.
- Wilson 95% interval, z = 1.96, floor 85%: 33 of 50 gives 0.5215 to 0.7756; 40 of 50 (a review case) gives 0.6696 to 0.8876.
- Sample size, failure rates: 0.08 to 0.20 gives N' = 42.56 and N' + 1/0.12 = 50.9; 0.08 to 0.14 gives 151.4; 0.10 to 0.12 gives 1,470.5 and 1,520.5. Cross-check on the handbook's example (0.10 to 0.20, one-sided 5%, power 90%): 101.2 and 111.2.
- Canary: 5% of 2,000 = 100 a day, 300 in three days. Assumed escalations 24 of 300 (8%): 0.0543 to 0.1163; 60 of 300 (20%): 0.1587 to 0.2489.
- The 85% floor and the gate rule (block when the upper bound is below the floor, send to review when the observed rate is below it but the interval reaches it, pass when the observed rate is at or above it) are choices, not standards. This is a regression-style rule against a fixed floor, so it is more lenient than the proof gate in the lesson, which passes only when the lower bound clears the threshold. The case reports two-sided 95% intervals, so a block uses what is effectively a 97.5% one-sided bound; the lesson's gates use one-sided 95% bounds.
- The retailer, categories and prompt versions are invented.
Last verified: 2026-10-22