Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to LLMOps and observability for AI systems.
Context
- A B2B software vendor runs an assistant that drafts replies to customer support tickets. Agents edit and send them.
- Volume: 40,000 drafts a day. About 10% (4,000) come from tenants whose tickets are long email threads, the long-thread segment.
- Quality signal: the accepted-draft rate, the share of drafts an agent sends with light edits or none. Baseline: 0.90.
Problem statement
A shorter, cheaper prompt (v14) passed the offline gate and went to all traffic on a Tuesday. In the long-thread segment the accepted-draft rate fell from 0.90 to 0.82. Support agents complained on Friday.
| Measure | Value |
|---|---|
| Drafts exposed before anyone noticed | 12,000 (4,000 a day for 3 days) |
| Extra rewrites | 960 (12,000 x 0.08) |
| Change in the all-tenant rate | 0.8 points (0.90 to 0.892) |
| Target | Catch a segment drop of this size in 1 day, with at most about 200 segment drafts exposed, and roll back without a deploy |
Constraints
- Fixed: the accepted-draft rate arrives only after an agent acts, so it lags the request.
- Fixed: prompts change several times a week; a release process that takes days will be skipped.
- Limit: the golden set has 400 items, only 20 of them long-thread tickets.
- Limit: one canary at a time, so results are not confounded.
Why the traditional approach fell short
An offline gate on an average hides a small segment: 20 items cannot show a drop that matters. Deploy and watch relies on complaints, which arrive days late. Prompts shipped like code need a redeploy to undo, so rollback is slow.
An all-traffic A/B test on the overall average has the same blind spot: the segment is diluted by the other 90%.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Offline gate only, then ship to everyone (status quo) | Rejected | Passed v14. Cannot see a segment it barely contains |
| B. Pilot tenants plus an agent lead reading 30 drafts by hand (non-AI, no new tooling) | Kept as rung 1 for risky changes | Cheap and human, but slow, and 30 drafts cannot detect a drop of 8 points |
| C. Prompt registry, shadow run, 5% canary with a per-segment gate, alias rollback | Accepted | Limits exposure, measures the real outcome, undoes in seconds |
Solution architecture
Figure 1. Where the release decision sits
How to read this: the router is the only place that knows which version serves whom, so the gate can change it without a deploy.
Figure 2. One draft during the canary, with the gate
How to read this: the alt block is the decision that the offline gate could not make for the long-thread segment.
How AI solved it, step by step
Step through the release, then switch to "Without the fix" to compare.
Under the hood
Google's SRE Workbook, chapter 16, "Canarying Releases", defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation". The part that gets the change is the canary and the rest is the control. Applied to a prompt, the version is the change and the release router is the splitter.
The size of the canary is set by the effect you must see. The NIST formula n = (z_alpha x sqrt(p0 q0) + z_beta x sqrt(p1 q1))^2 / delta^2 with p0 = 0.90 and p1 = 0.82 (one-sided alpha 0.05, power 0.80) gives 104.2 items, or 116.7 with the continuity correction. About 200 segment drafts a day clears that in one day. The overall average needs far more: detecting 0.90 to 0.892 takes about 8,899 items, 4.5 days of the 2,000 canary drafts.
Set passes to 180: the lower bound rises to 0.8596 and the gate passes. The interval maths is in Under the hood: Confidence intervals and sample sizes for gates. The release mechanics are in Under the hood: Shadow, canary and A/B releases.
What went wrong / failure modes
| Failure | Mitigation |
|---|---|
| Canary too small for a segment | Size each segment's gate with the sample-size formula, and wait for its outcome count, not for a clock |
| Outcome lags the request | Gate on accepted drafts after the agent acts; keep the canary at least a day to cover the daily cycle |
| Segment label missing or wrong | Assign the segment at request time from tenant metadata and log it on the trace |
| Version drift: retrieval settings change outside the prompt | The registry version pins retrieval settings and model too |
| Two canaries overlap | One canary at a time, as the SRE chapter advises |
| Shadow run adds model cost | Cap shadow at 5% for one day; it adds about 5% to that day's calls |
Results
All numbers are illustrative targets, not measurements.
| Measure | Without the fix | With the fix |
|---|---|---|
| Segment drafts exposed to v14 | 12,000 | 200 |
| Extra rewrites | 960 | about 16 |
| Time to detect | 3 days, by complaint | 1 day, by gate |
| Time to undo | Redeploy | Alias change |
Tradeoffs and when NOT to use this approach
- Low traffic. A segment with 10 drafts a day never fills a gate. Use pilot tenants and manual review.
- Delayed outcomes. If the outcome arrives weeks later, the canary cannot gate. Use a judge or labels as a proxy and say so.
- Cost. A router, a registry and per-segment dashboards are worth it only when prompts change often.
- Risky changes (model swap, new tools) deserve shadow and a longer canary.
Design rules extracted
- Make a prompt a pinned, immutable version; a release moves an alias, never edits a version.
- Gate per segment on the real outcome, sized with the sample-size formula, not on the overall average.
- Stratify the canary so small segments are in it.
- Keep rollback to one configuration change.
- Stamp the version on every trace so a bad answer points to its release.
Assumptions
All numbers are illustrative.
- Volume 40,000 drafts a day; the long-thread segment is 10% = 4,000 a day; baseline accepted-draft rate 0.90 in every segment. Accepted means an agent sends the draft with light edits or none.
- v14 lowers the segment rate to 0.82 and leaves other segments at 0.90. Overall: 0.90 - 0.10 x 0.08 = 0.892, a 0.8-point drop.
- Without the fix: 4,000 x 3 days = 12,000 drafts; 12,000 x 0.08 = 960 extra rewrites. Rewrites mean drafts that would have been accepted under v13.
- Offline set: 400 items, 370 pass = 0.925; Wilson (z 1.645) lower bound 0.9004; gate 0.90. Long-thread items: 20 (5% of 400). A slice of 20 would need about 105 items to detect 0.90 to 0.82, so the slice is grown to 120 (above the 116.7 with the correction) after the incident.
- Canary 5%: 0.05 x 4,000 = 200 segment drafts a day; 164 accepted = 0.82; Wilson lower bound 0.7711; gate = baseline minus 5 points = 0.85. A healthy 180 of 200 gives lower bound 0.8596 and passes. Extra rewrites: 36 observed minus 20 expected at baseline = 16.
- Sample size (NIST/SEMATECH e-Handbook, one-sided alpha 0.05 z 1.645, power 0.80 z 0.842): 0.90 to 0.82 gives 104.2 (116.7 corrected); 0.90 to 0.892 gives 8,899 (9,024 corrected). Canary overall: 0.05 x 40,000 = 2,000 a day; 8,899 / 2,000 = 4.45 days (4.51 with the correction).
- Shadow: 5% of requests for one day adds 5% to that day's calls.
Last verified: 2026-10-22