What you'll learn: what the customer's team must be able to watch, fix, change and pay for the day after go-live, and how to prove you handed it over.
At a glance
Figure 1. Life after go-live
flowchart LR
accTitle: Operating the assistant after go-live
accDescr: Signals feed alerts and a monthly review. Alerts open a runbook row. Changes to the model, the prompt or a policy document go through a rerun of the golden set. Everything the customer needs is collected in a handover pack.
A["Signals"] --> B["Alert or monthly review"]
B --> C["Runbook row"]
B --> D["Change request"]
D --> E["Golden set rerun"]
E --> F["Release or roll back"]
C --> G["Handover pack"]
F --> G
How to read this: every path ends in the handover pack, because the customer's team runs it after you leave.
Watch six signals, each with a paged owner.
A runbook row is symptom, check, action and who decides.
No model, prompt or policy-document change ships without a golden-set rerun.
Your artifact: handover checklist and runbook skeleton.
Scenario
Composite scenario, not a real company. The Lumen Order and Policy Assistant passed its pilot review. The Head of Customer Operations asks: "Who is paged when it gives a wrong answer, and what does it cost next month?" After the support period in the proposal, the platform team and a customer engineer take over. All numbers are illustrative.
What you will be able to do
Choose signals, thresholds and an owner to page for each.
Write a runbook row and a change process that reruns the golden set.
Plan a model change from a provider's deprecations page.
Fill a handover checklist with an owner and evidence per row.
Figure 2. Signals for the Lumen assistant (thresholds illustrative)
Signal
Alert when
Paged first
Quality sample: 50 answers a week, read by the policy owner's delegate
One high-severity error (a superseded or invented rule)
Policy owner
Refusal and fallback rate
Above 15 percent (assumed pilot level 8 percent)
Customer engineer
Latency, 95th percentile
Above the target in the proposal
Customer engineer
Cost per request
Weekly average above 1.5 x plan
Customer engineer
Tool errors on order lookup
Error rate above 2 percent over 15 minutes
Platform team
Approval queue age (phase 2 only)
Oldest proposal waiting over 1 working hour
Team lead on duty
How to read this: each alert names a person, because an alert nobody owns is a log line. Page on symptoms agents feel; causes go in the runbook.
For metric names, the OpenTelemetry GenAI conventions define gen_ai.client.inference.duration (histogram, seconds) and token counters such as gen_ai.client.inference.usage.input_tokens (unit {token}). Status is Development as of 2026-11-04, so names can change: record the version you emit. Prompt and answer text attributes are opt-in and marked likely to hold personal data: keep them off.
A runbook row
RUNBOOK ROW
Symptom: <what an agent or an alert shows>
Check: <the first two things to look at, with where>
Action: <the safe step; what can be switched off>
Who decides: <role that approves anything beyond the safe step>
Filled row. Symptom: the weekly sample finds a 14-day return answer. Check: the policy id and version the log cites; whether P-RET-01 v2 is still indexed. Action: remove v2, rerun the 9 version traps; meanwhile switch policy answers off (order lookup keeps working). Who decides: the policy owner.
Changing the model, the prompt or a policy document
Figure 3. The change flow
flowchart TD
accTitle: Change flow for model, prompt and policy document updates
accDescr: A change is written down and applied to a staging configuration while the old release stays live. The golden set is rerun. If the pass rules are met with no regression, the change rolls forward to a few agents first and signals are watched. If not, or if signals turn bad, the previous release is restored.
A["Trigger: provider notice, prompt edit or policy update"] --> B["Write down what, why and who approves"]
B --> C["Apply to staging, old release stays live"]
C --> D["Rerun the golden set"]
D --> E{"Pass rules met, no regression?"}
E -- yes --> F["Roll forward to a few agents first"]
F --> G{"Signals normal for a week?"}
G -- yes --> H["All agents"]
G -- no --> I["Roll back to the previous release"]
E -- no --> I
How to read this: the old release stays pinned, so rolling back is a config change.
The golden set is the 40 acceptance questions: 30 answerable, 10 unsupported. Reuse the pass rules: at least 27 of 30 correct with citations, 0 high-severity errors, 9 of 9 version traps, 10 of 10 declined. Compare with the last release's run, not only the threshold. A new policy version adds a question. See Golden set and Evaluation report.
Active, Legacy, Deprecated (works; replacement and retirement date assigned), Retired (requests fail)
Notice
At least 60 days before retirement for publicly released models, by email and in the docs
Dates
Active models also carry a tentative "not sooner than" retirement date
Example row
claude-sonnet-4-5-20250929 on the Claude API: deprecated 2026-09-30, retirement 2026-11-30, replacement claude-sonnet-5-5
Partner platforms
Amazon Bedrock and Google (Google managed) set their own schedules
Audit
A usage export shows requests by API key and model
The example row gives 61 days of notice, and 26 days remain from 2026-11-04. The page warns that deprecated models are likely to be less reliable. So pin the model id, put the retirement date in the runbook, and start Figure 3 when the notice arrives.
Decision exercise. A notice says the pinned model retires in 26 days. Options: (A) change the id in production today and watch; (B) run Figure 3 on staging now and decide from the golden-set result; (C) wait until the last week. Choose one.
Model answer
Choose B. A skips the rerun, so a regression reaches agents first. C leaves no room to fix a failed rerun or to roll back before requests start failing.
Monthly cost review and the handover pack
Compare actual with plan monthly; find the cause of any gap. Plan: 6,720 assisted contacts at 3,200 tokens in and 350 out, placeholder prices 1.00 in and 5.00 out per million tokens, in EUR (Estimation and TCO).
Figure 4. Handover pack
Item
Owner after handover
Evidence the customer can open
Architecture diagrams and ADRs
Platform team
Dated files in the customer's repository
Evaluation report and golden set
Policy owner
Report with counts, test set and rerun script
Runbook
Customer engineer
At least one dry run, dated and signed
Access list
IT security lead
Who can read what, reviewed on a date
Cost sheet and review date
Head of Customer Operations
Plan against actual, last three months
Open risks
Head of Customer Operations
List with an owner and a date per risk
How to read this: a row missing owner or evidence is not handed over.
Worked example
Month 2 at Lumen. Assumptions: illustrative; 7,500 assisted contacts; a policy update added passages, so input grew from 3,200 to 4,100 tokens per request; output stays 350.
Plan per request: 3,200 x 1.00 + 350 x 5.00 = 4,950 per million, so 0.00495. Plan per month: 6,720 x 0.00495 = 33.264.
Actual: 7,500 x 4,100 = 30.75 million tokens in, 7,500 x 350 = 2.625 million out. Cost: 30.75 + 13.125 = 43.875, which is 31.9 percent above plan.
Cost per request is 0.00585, 1.18 times plan. The weekly alert fires at 1.5 times (0.007425), so it stayed silent.
Volume is 11.6 percent above plan and cost per request 18.2 percent above; together 1.116 x 1.182 = 1.319. The monthly review found it. Under Figure 3: cap the passages kept and rerun the golden set, or raise the budget.
At the customer: "Who is paged when it is wrong?" Answer with a role from Figure 2, a time, and the runbook row they will open.
Practice task
Copy Figure 4 and give every row for the Lumen project a named role and one piece of evidence.
Explain it back. In three sentences, tell the Head of Customer Operations why a model update is a release, not a settings change. A good answer: the golden set is rerun, the old release stays ready, and agents see it only after it passes.
Operating calls (interactive when JavaScript is on)
An agent reports that the assistant quoted the 14-day return rule. What do you do first?
Raise the refusal alert level
Check which policy version the log cites, then follow the runbook row
Wait for the monthly cost review
Answer
Check which policy version the log cites, then follow the runbook row. The runbook row says to check the cited policy id and version, then remove v2 and rerun the version traps. The refusal alert is about a different signal.
Weekly cost per request is 0.0066 against a plan of 0.00495. Does the weekly alert fire?
Yes, it is above plan
No, it fires above 1.5 x plan, which is 0.007425
Yes, any rise fires it
Answer
No, it fires above 1.5 x plan, which is 0.007425. The alert is set at 1.5 times plan. A smaller rise is left to the monthly review, which finds the cause.
The golden-set rerun for a new model scores 26 of 30 correct. What happens?
Roll forward, it is close
Roll forward to a few agents only
Keep the old release, then fix or drop the change
Answer
Keep the old release, then fix or drop the change. The pass rule is at least 27 of 30, and the flow keeps the old release when the rules are not met. Showing it to a few agents first is a step for changes that already passed.
Exercise
Choose one failure: the order system is slow, the model provider is down, or an old policy version is back in the index. Write its first three runbook rows. Then list what you hand over and what you keep supporting under the proposal.
Each row has all four fields, and each check names a place to look.
The keep-supporting list names a task, such as the golden-set rerun for a model change, and an end date.
Done when the customer engineer could follow your rows without calling you.
Hint 1: start from the signal
Take the alert in Figure 2 that fires first and write its symptom as an agent would.
Hint 2: split decide from act
The safe action is what the on-call engineer can do alone. Policy, scope or cost changes go to a role.
Common mistakes
No owner on an alert. Page a role, not a channel.
A runbook nobody has run. Dry-run it before handover.
An unpinned model id. You cannot say which model produced an answer.
Logging prompt and answer text. It can hold personal data; log ids.
Knowledge check
1. Why page on symptoms and not causes?
A symptom is what agents feel. Causes are many and belong in the runbook.
2. Why does a policy-document update need a golden-set rerun?
A new version can break the version traps. The rerun shows it before agents do.
3. The provider gives 61 days of notice. Why start the change flow at once?
A failed rerun needs time for a fix or another model.
4. In the worked example the weekly cost alert stayed silent. Is the alert wrong?
No. It guards against jumps, and 0.00585 is below 0.007425. The monthly review catches slow growth.
5. What makes a handover row complete?
An owner on the customer's side and evidence the customer can open. Without both, the supplier still holds the risk.
You can now
Add a handover checklist and runbook skeleton to your design pack.
Set signals, thresholds and who is paged for each.
Write a runbook row with a safe action and a deciding role.
Run a model, prompt or policy change through a golden-set rerun.
Lesson 8 of 15 · AI Solution Architect: Estimate, Propose and Defend
Give the customer's team what it needs on day one: signals to watch, a runbook, a model-change process, a monthly cost review and a handover pack.
In the rest of this lesson
Worked example
Practice task
Exercise
Common mistakes
Knowledge check
You can now
Enrolling is free. It unlocks every lesson in AI Solution Architect: Estimate, Propose and Defend, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.