Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Evaluating AI systems.
Context
- A home-and-electronics retailer runs a shopping assistant in its app and on its website.
- It answers pre-purchase questions (compatibility, sizing, delivery) and after-sales questions (returns, warranty, repairs) across eight product categories, about 20,000 conversations a day.
- A small team owns the system prompt, shipped through a release pipeline.
Problem statement
The team changed the prompt to "answer in at most three sentences, in a friendly tone" after reading ten sample chats. Two weeks later, tickets about wrong return-window answers rose, yet the average thumbs-up rate had not moved. Answers had lost conditions: "within 30 days" survived, but the exclusion for used items was cut for length.
| Measure | Before | After the tweak |
|---|---|---|
| Returns and warranty, labelled set | 46 of 50 (92%) | 33 of 50 (66%) |
| All categories, labelled set | 360 of 400 (90%) | 352 of 400 (88%) |
Target: a category clearly below its 85% floor blocks release, whatever the overall score.
Constraints
- Fixed (policy): return and warranty conditions must be stated completely whenever an answer concerns them.
- Fixed: model calls leave only through the model gateway, to an approved provider.
- Limit: reviewers could label about 50 items per category this quarter.
- Limit: the gate must finish inside a 20-minute pipeline stage; estimate run cost with the LLM cost estimator.
Why the traditional approach fell short
| Approach | Where it breaks |
|---|---|
| Unit tests and keyword rules | Free text has many correct wordings, so exact-output tests are brittle or vacuous. A rule can see "30 days", not the exclusion |
| Manual spot check of ten chats | At 10% of traffic, the chance that at least one chat is in the category is 1 - 0.9^10 = 65% |
| Aggregate dashboard | A 26-point fall in one category (12.5% of the set) dilutes to about 3 points, offset by gains elsewhere |
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Manual spot check before release (non-AI, status quo) | Rejected | Small sample, not repeatable, no record |
| B. One overall golden set, tolerance of 3 points | Rejected as the only gate | 88% versus 90% passes; seeing a 2-point drop needs about 1,471 items |
| C. Golden set per category, interval-based gate, then a canary | Accepted | Catches a category drop before release and checks live traffic after |
| D. Render returns and warranty answers from the policy table, model only routes (simpler, non-AI) | Kept as a fallback | Testable with exact assertions, at the cost of flexible phrasing |
The default is C. If a category cannot hold its floor after two rounds of prompt work, D is the better design for it.
Solution architecture
Figure 1. Release pipeline with a per-category gate
How to read this: the Eval service is the only box that can say block, review or pass, and its inputs are the stored golden set and the canary signals.
Figure 2. One prompt change through the gate
How to read this: the alt block is the gate's three outcomes, and only the last one reaches production, behind a canary.
How AI solved it, step by step
Step through the gate, then switch to "Without the fix".
Under the hood
The interval is the Wilson score interval from the NIST/SEMATECH e-Handbook, with p the observed proportion, n the sample size and z the normal quantile. The handbook notes that the common p ± z·√(p(1−p)/n) form can give an impossible limit (a negative lower limit) and calls such an approach inferior.
Wilson limits = (p + z²/2n ± z·√(p(1−p)/n + z²/4n²)) / (1 + z²/n)
33 of 50 at z = 1.96 -> 52.2% to 77.6%
Items needed, N >= ((z(1-a)·√(p0(1-p0)) + z(1-b)·√(p1(1-p1))) / d)², d = |p1 - p0|
continuity correction: N + 1/d (one-sided a = 0.05, power 80%)
The same handbook formula sizes the gate. Failure rates are treated symmetrically, so 8% to 20% failures is the same problem as 92% to 80% passes.
| Drop to detect | Items needed | With correction |
|---|---|---|
| Category, 8% to 20% failures (12 points) | 42.6, so 43 | 51 |
| Category, 8% to 14% failures (6 points) | 151.4, so 152 | 168.1 |
| Overall, 10% to 12% failures (2 points) | about 1,471 | 1,521 |
Fifty items per category roughly suffice for a 12-point drop, not a 6-point one. The formula treats the baseline as fixed, a simplification.
Set p1 to 0.86 to see the 6-point case: the count rises to 151.4.
The mechanism is covered in Under the hood: Confidence intervals and sample sizes for gates.
What went wrong / failure modes
| Failure | Mitigation |
|---|---|
| Gate too small: 50 items cannot show a 6-point drop | Document the smallest detectable drop; grow high-risk categories first |
| False alarms: with eight independent checks at 5% each, at least one fires with probability 1 - 0.95^8 = 34% | A review lane for borderline results, not an automatic block |
| Tuning on the gate set: editing the prompt until 400 items pass overfits it | A held-out set that authors never see; refresh items from production chats each quarter |
| Canary too thin: 5% sees about 100 returns chats a day, 300 in three days. That separates 8% from 20% escalation (5.4% to 11.6% against 15.9% to 24.9%), not 8% from 12% | Write down the shift the canary can see and extend it for smaller ones |
Results
All results are illustrative targets, not measurements.
| Measure | Before | Target after |
|---|---|---|
| Release check | Ten chats read by one person | 400 graded items, a verdict per category |
| Returns and warranty pass rate | 33 of 50, shipped unnoticed | Blocked, then 45 of 50 |
| Time to detect a category drop | About two weeks, via tickets | One pipeline run, under 20 minutes |
Tradeoffs and when NOT to use this approach
- Low traffic. With a few hundred chats a month, read every low-rated one.
- High-stakes answers. If a wrong condition has legal weight, use option D, deterministic text.
- Unreliable grading. If reviewers or the judge disagree often, the gate measures noise. Calibrate first.
- Rare events. They need more items than a gate stage allows; monitor them online.
Design rules extracted
- Gate each category, not only the average.
- Decide what the gate may claim: a block means clearly below the floor; a pass means only not shown to be below.
- Size the set for the smallest drop you must catch, and write that number down.
- Keep the tuning set and the gate set apart.
- Follow the offline gate with a canary segmented by category, and put the prompt version on every trace.
Assumptions
All numbers are illustrative.
- Traffic: 20,000 chats a day, 10% in returns and warranty (2,000 a day). At least one of ten random chats in it: 1 - 0.9^10 = 0.651.
- Golden set: 8 x 50 = 400 items. Before: overall 360 of 400, category 46 of 50, others 314 of 350. After v14: overall 352 of 400, category 33 of 50, others 319 of 350. Weight 50/400 = 12.5%; 0.26 x 0.125 = 3.25 points.
- Wilson 95% interval, z = 1.96, floor 85%: 33 of 50 gives 0.5215 to 0.7756.
- Sample size, failure rates (N' and N' + 1/delta): 0.08 to 0.20 gives 42.56 and 50.9; 0.08 to 0.14 gives 151.4 and 168.07; 0.10 to 0.12 gives 1,470.5 and 1,520.5.
- Canary: 5% of 2,000 = 100 a day, 300 in three days. Assumed escalations 24 of 300 (8%): 0.0543 to 0.1163; 60 of 300 (20%): 0.1587 to 0.2489.
- The 85% floor and the gate rule (block when the upper bound is below the floor, review when the observed rate is below it but the interval reaches it, pass otherwise) are choices, not standards. The rule is more lenient than the proof gate in the lesson, which passes only when the lower bound clears the threshold. Two-sided 95% intervals make a block effectively a 97.5% one-sided bound; the lesson's gates use one-sided 95% bounds.
- The retailer, categories and prompt versions are invented.
Last verified: 2026-10-22