What you'll learn: how to judge a pilot against the outcomes the customer agreed, plan a staged rollout with a gate and a rollback trigger at each step, measure adoption without confusing usage with value, and write a proposal that asks for named decisions.
At a glance
Figure 1. The rollout ladder
How to read this: solid arrows need a written gate to pass and dotted arrows are the way back when a trigger fires, so every rung can be undone.
- Judge the pilot against the outcome sheet agreed before it started.
- Roll out in rungs, each with a gate up and a trigger back.
- Count adoption as agents who keep returning, not total requests.
- Your artifact: the rollout plan and solution proposal.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, has built its read-only Order and Policy Assistant, which reads the order management system (OMS). Its 24 agents work in four teams of about 6 (an assumption), each with a team lead. The Head of Customer Operations wants it open to everyone, plus a phase 2 where the assistant proposes address changes and discount exceptions for a human to approve. Say what the pilot proved and recommend. All numbers are illustrative.
What you will be able to do
- Compare pilot results with baselines and targets and give a verdict with limits.
- Draft gates and rollback triggers for each rung of a ladder.
- Choose adoption measures that show lasting use.
- Write a proposal that ends in decisions the customer can approve.
Prerequisites
Read Demonstrating value and handling change: the outcome sheet, criteria and change log feed the proposal. For release stages see LLMOps and observability; the estimate comes from AI project estimation.
Core ideas
Evaluate the pilot against the sheet
Figure 2. Pilot evaluation decision
How to read this: safety criteria are checked first and cannot be traded against a speed gain.
Safety criteria here are AC-2 to AC-6: version traps, refusal, held orders, order-change rules and no writes.
Adoption is not request count
| Measure | What it tells you | What it hides |
|---|---|---|
| Weekly active agents divided by invited | Whether agents come back | Why they leave |
| Wrong-answer reports | Where trust breaks | Errors nobody reports |
| Handling time against baseline | The outcome itself | Selection of easy contacts |
The ladder and its gates
| Rung | Who | Gate to the next rung | Rollback trigger | Decided by |
|---|---|---|---|---|
| Offline | Test set, no agents | AC-1 to AC-6 pass | n/a | Head of Customer Operations |
| One team | 6 agents, 1 team lead | Criteria rerun on the pilot log; at least 5 of 6 agents active 3 weeks running | A quoted superseded policy, or any OMS write call | Same |
| Two teams | 12 agents, one per region | Wrong-answer reports within the agreed ceiling for 4 weeks | Same, or the ceiling exceeded | Same, with the IT security lead |
| All four teams | 24 agents | Support load within the agreed limit | Same | Same |
The thresholds are examples; the customer's owner chooses the real ones. A rollback sends that group back to the old tools the same day.
ROLLOUT PLAN
Rung: <name>
Population: <who, how many, how chosen>
Entry gate: <criteria, test sets, sign-off>
Minimum time: <weeks at this rung>
Rollback trigger: <measured condition, who may pull it, what happens>
Support: <training, channel, office hours>
Decision: <who approves moving up>Explain it back. Tell the Head of Customer Operations in three sentences why you start with one team of 6. A good explanation names the cost of a wrong answer, says criteria are rerun on real contacts before widening, and weighs waiting.
The solution proposal
| Section | The test it must pass |
|---|---|
| 1. Summary: problem, recommendation, the ask | A busy reader can decide from this alone |
| 2. Problem and baseline | Every number has a sample size |
| 3. What the pilot showed | Misses sit in the same table as passes |
| 4. Scope: in, later, out | Every item has a reason |
| 5. Acceptance criteria for the next phase | Each can fail |
| 6. Rollout plan | Each gate has an owner |
| 7. Risks, assumptions, open questions | Each has an owner and a date |
| 8. Security and data | Links the checklist; unknowns listed |
| 9. Effort, timeline, cost | Placeholder prices; assumptions stated |
| 10. Decisions requested | Each has a name and a date |
Write "placeholder price" and fill it from the provider's published list on the day you send it.
Worked example
Assumptions: hypothetical pilot results, to practise the verdict. Baseline median 9 minutes (24 timings), target 6 or less.