Skip to content

    AI Architect Case Files

    Case File: Silent Regression After a Prompt Tweak

    A retailer's shopping assistant got worse in one product category after a prompt tweak while the overall score held. A per-category gate caught it.

    By Published 8 min read

    • ai-architect
    • case-file
    • industry-retail-ecommerce
    • pattern-eval-gates
    • failure-quality-regression
    • kind-composite
    Share

    Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.

    This case belongs to Evaluating AI systems.

    Context

    • A home-and-electronics retailer runs a shopping assistant in its app and on its website.
    • It answers pre-purchase questions (compatibility, sizing, delivery) and after-sales questions (returns, warranty, repairs) across eight product categories, about 20,000 conversations a day.
    • A small team owns the system prompt, shipped through a release pipeline.

    Problem statement

    The team changed the prompt to "answer in at most three sentences, in a friendly tone" after reading ten sample chats. Two weeks later, tickets about wrong return-window answers rose, yet the average thumbs-up rate had not moved. Answers had lost conditions: "within 30 days" survived, but the exclusion for used items was cut for length.

    Measure Before After the tweak
    Returns and warranty, labelled set 46 of 50 (92%) 33 of 50 (66%)
    All categories, labelled set 360 of 400 (90%) 352 of 400 (88%)

    Target: a category clearly below its 85% floor blocks release, whatever the overall score.

    Constraints

    • Fixed (policy): return and warranty conditions must be stated completely whenever an answer concerns them.
    • Fixed: model calls leave only through the model gateway, to an approved provider.
    • Limit: reviewers could label about 50 items per category this quarter.
    • Limit: the gate must finish inside a 20-minute pipeline stage; estimate run cost with the LLM cost estimator.

    Why the traditional approach fell short

    Approach Where it breaks
    Unit tests and keyword rules Free text has many correct wordings, so exact-output tests are brittle or vacuous. A rule can see "30 days", not the exclusion
    Manual spot check of ten chats At 10% of traffic, the chance that at least one chat is in the category is 1 - 0.9^10 = 65%
    Aggregate dashboard A 26-point fall in one category (12.5% of the set) dilutes to about 3 points, offset by gains elsewhere

    Options considered

    Option Verdict Reason
    A. Manual spot check before release (non-AI, status quo) Rejected Small sample, not repeatable, no record
    B. One overall golden set, tolerance of 3 points Rejected as the only gate 88% versus 90% passes; seeing a 2-point drop needs about 1,471 items
    C. Golden set per category, interval-based gate, then a canary Accepted Catches a category drop before release and checks live traffic after
    D. Render returns and warranty answers from the policy table, model only routes (simpler, non-AI) Kept as a fallback Testable with exact assertions, at the cost of flexible phrasing

    The default is C. If a category cannot hold its floor after two rounds of prompt work, D is the better design for it.

    Solution architecture

    Figure 1. Release pipeline with a per-category gate

    Release pipeline with a per-category eval gateWorker to Eval service (run gate); Eval service to Document store; Eval service to Model gateway; Model gateway to LLM provider; Eval service to Worker (block); Eval service to Approval service (review); Eval service to Orchestrator (pass); Orchestrator to Trace store; Trace store to Eval service (signals)RELEASE PIPELINEWorker: CI jobWorkerCI jobEval service: per-category gateEval serviceper-category gateDocument store: golden setDocument storegolden setModel gatewayModel gatewayApproval serviceApprovalserviceOrchestrator: canary 5%Orchestratorcanary 5%Trace store: prompt versionTrace storeprompt versionLLM provider: approved listLLM providerapproved listrun gateblockreviewpasssignals

    How to read this: the Eval service is the only box that can say block, review or pass, and its inputs are the stored golden set and the canary signals.

    Figure 2. One prompt change through the gate

    sequenceDiagram accTitle: One prompt change through the gate accDescr: The CI worker asks the Eval service to run a candidate prompt. The service loads the golden set, answers and grades each item through the Model gateway, computes per-category intervals and returns a verdict. A block fails the build. A review goes to a person. A pass starts a canary whose traces are checked before promotion or rollback. participant WK as Worker participant EV as Eval service participant DS as Document store participant MG as Model gateway participant LP as LLM provider participant AP as Approval service participant OR as Orchestrator participant TS as Trace store WK->>EV: Run gate for prompt v14 against baseline v13 EV->>DS: Load golden set, 8 categories of 50 loop each item EV->>MG: Question with candidate prompt MG->>LP: Request LP-->>MG: Answer MG-->>EV: Answer, graded against required facts end EV->>EV: Wilson interval per category, compare with floor alt upper bound below floor EV-->>WK: Block, category named else observed below floor but interval reaches it EV->>AP: Review request with failing items AP-->>WK: Human decision else observed at or above floor EV-->>OR: Start canary at 5 percent OR->>TS: Traces carry the prompt version TS-->>EV: Escalation rate per category after 3 days EV-->>OR: Promote or roll back end

    How to read this: the alt block is the gate's three outcomes, and only the last one reaches production, behind a canary.

    How AI solved it, step by step

    Step through the gate, then switch to "Without the fix".

    Gating prompt v14 (interactive when JavaScript is on)

    As designed

    1. Start the gate. The author's change makes prompt v14 shorter and friendlier. The Worker asks the Eval service to run v14 against baseline v13.
    2. Load the golden set. The Eval service loads 400 items: eight categories of 50, each with a question and the facts a correct answer must contain, such as the window and the exclusion.
    3. Answer each item. Each item goes through the Model gateway with the candidate prompt, same model and settings as production.
    4. Grade each answer. Each answer is graded pass or fail on its required facts by a judge whose agreement with human labels is checked separately (see the summarisation case).
    5. Interval per category. The Eval service computes a pass rate and a 95% interval per category.
    6. Decision: block. Returns and warranty scores 33 of 50, upper bound 77.6%, below the 85% floor. Verdict: block, naming the category.
    7. Fix and pass. The author restores the conditions in v15. Returns and warranty now scores 45 of 50 (90%), at or above the floor: pass.
    8. Canary. The Orchestrator serves v15 to 5% of traffic for three days. The Eval service compares the human-escalation rate per category from traces, then promotes or rolls back by configuration change.

    Without the fix

    1. Spot check and ship. The author reads ten sample chats. v14 goes to all traffic with no per-category record.
    2. Dashboard stays flat. Overall quality moves from 90% to 88% on the labelled set, and the thumbs-up rate does not move.
    3. Tickets arrive. About two weeks later, tickets about wrong return-window answers rise. Returns and warranty had fallen from 92% to 66%.

    Under the hood

    The interval is the Wilson score interval from the NIST/SEMATECH e-Handbook, with p the observed proportion, n the sample size and z the normal quantile. The handbook notes that the common p ± z·√(p(1−p)/n) form can give an impossible limit (a negative lower limit) and calls such an approach inferior.

    Wilson limits = (p + z²/2n ± z·√(p(1−p)/n + z²/4n²)) / (1 + z²/n)
    33 of 50 at z = 1.96  ->  52.2% to 77.6%
    
    Items needed, N >= ((z(1-a)·√(p0(1-p0)) + z(1-b)·√(p1(1-p1))) / d)², d = |p1 - p0|
    continuity correction: N + 1/d   (one-sided a = 0.05, power 80%)

    The same handbook formula sizes the gate. Failure rates are treated symmetrically, so 8% to 20% failures is the same problem as 92% to 80% passes.

    Drop to detect Items needed With correction
    Category, 8% to 20% failures (12 points) 42.6, so 43 51
    Category, 8% to 14% failures (6 points) 151.4, so 152 168.1
    Overall, 10% to 12% failures (2 points) about 1,471 1,521

    Fifty items per category roughly suffice for a 12-point drop, not a 6-point one. The formula treats the baseline as fixed, a simplification.

    Set p1 to 0.86 to see the 6-point case: the count rises to 151.4.

    Items needed to catch the category drop (interactive when JavaScript is on)
    Items to detect 92.0% falling to 80.0%42.6
    With continuity correction (n + 1/delta)50.9

    The mechanism is covered in Under the hood: Confidence intervals and sample sizes for gates.

    What went wrong / failure modes

    Failure Mitigation
    Gate too small: 50 items cannot show a 6-point drop Document the smallest detectable drop; grow high-risk categories first
    False alarms: with eight independent checks at 5% each, at least one fires with probability 1 - 0.95^8 = 34% A review lane for borderline results, not an automatic block
    Tuning on the gate set: editing the prompt until 400 items pass overfits it A held-out set that authors never see; refresh items from production chats each quarter
    Canary too thin: 5% sees about 100 returns chats a day, 300 in three days. That separates 8% from 20% escalation (5.4% to 11.6% against 15.9% to 24.9%), not 8% from 12% Write down the shift the canary can see and extend it for smaller ones

    Results

    All results are illustrative targets, not measurements.

    Measure Before Target after
    Release check Ten chats read by one person 400 graded items, a verdict per category
    Returns and warranty pass rate 33 of 50, shipped unnoticed Blocked, then 45 of 50
    Time to detect a category drop About two weeks, via tickets One pipeline run, under 20 minutes

    Tradeoffs and when NOT to use this approach

    • Low traffic. With a few hundred chats a month, read every low-rated one.
    • High-stakes answers. If a wrong condition has legal weight, use option D, deterministic text.
    • Unreliable grading. If reviewers or the judge disagree often, the gate measures noise. Calibrate first.
    • Rare events. They need more items than a gate stage allows; monitor them online.

    Design rules extracted

    1. Gate each category, not only the average.
    2. Decide what the gate may claim: a block means clearly below the floor; a pass means only not shown to be below.
    3. Size the set for the smallest drop you must catch, and write that number down.
    4. Keep the tuning set and the gate set apart.
    5. Follow the offline gate with a canary segmented by category, and put the prompt version on every trace.

    Assumptions

    All numbers are illustrative.

    • Traffic: 20,000 chats a day, 10% in returns and warranty (2,000 a day). At least one of ten random chats in it: 1 - 0.9^10 = 0.651.
    • Golden set: 8 x 50 = 400 items. Before: overall 360 of 400, category 46 of 50, others 314 of 350. After v14: overall 352 of 400, category 33 of 50, others 319 of 350. Weight 50/400 = 12.5%; 0.26 x 0.125 = 3.25 points.
    • Wilson 95% interval, z = 1.96, floor 85%: 33 of 50 gives 0.5215 to 0.7756.
    • Sample size, failure rates (N' and N' + 1/delta): 0.08 to 0.20 gives 42.56 and 50.9; 0.08 to 0.14 gives 151.4 and 168.07; 0.10 to 0.12 gives 1,470.5 and 1,520.5.
    • Canary: 5% of 2,000 = 100 a day, 300 in three days. Assumed escalations 24 of 300 (8%): 0.0543 to 0.1163; 60 of 300 (20%): 0.1587 to 0.2489.
    • The 85% floor and the gate rule (block when the upper bound is below the floor, review when the observed rate is below it but the interval reaches it, pass otherwise) are choices, not standards. The rule is more lenient than the proof gate in the lesson, which passes only when the lower bound clears the threshold. Two-sided 95% intervals make a block effectively a 97.5% one-sided bound; the lesson's gates use one-sided 95% bounds.
    • The retailer, categories and prompt versions are invented.

    Last verified: 2026-10-22

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments