Turn a request like make agents faster and consistent into a problem statement and an outcome sheet with indicators, baselines and targets you can test.
What you'll learn: how to turn "agents should answer faster and consistently" into an outcome sheet: ask what each vague word means, use the measured baseline before promising a target, and write each outcome with an indicator, a threshold, a population and a place to measure it.
At a glance
Figure 1. From a vague request to requirements
flowchart LR
accTitle: From a vague request to requirements
accDescr: A vague request is clarified into a problem statement. The problem statement gives outcomes. Each outcome gets an indicator and a baseline measurement. The baseline and the owner decide a target. Outcomes with targets become requirements, and unknowns become open assumptions that go back to the customer.
A["Vague request"] --> B["Clarify: who, what, when, how bad"]
B --> C["Problem statement"]
C --> D["Outcomes"]
D --> E["Indicator and baseline"]
E --> F["Target set with the owner"]
F --> G["Requirements"]
E -. unknown .-> H["Open assumption"]
H -.-> B
How to read this: a target without a baseline is a guess, so a missing baseline sends you back to measure instead of forward to commit.
"Faster", "consistent" and "correct" are not requirements until each has an indicator, a threshold and a way to measure.
Write outcomes in the customer's world (minutes, wrong answers), then derive system requirements.
Your artifact: the outcome sheet.
Scenario
Composite scenario, not a real company. After discovery, Lumen Industrial Supply, a fictional distributor, still has one sentence from the Head of Customer Operations: "Agents should answer faster and more consistently." Your discovery notes hold 24 agents, 9,600 contacts a month, a baseline median of 9 minutes from 24 timings (6 agents, 4 contacts each) and pain points such as an old return policy being quoted. Nobody has agreed what success means. All numbers are illustrative.
What you will be able to do
Rewrite a vague phrase into an outcome with an indicator, a threshold and a measurement point.
Report a baseline without overclaiming and explain a small-sample caveat.
Write a four-sentence problem statement and an outcome sheet.
Explain to a sponsor why a number they proposed needs a baseline.
Median minutes from contact opened to answer sent; baseline 9 (n = 24)
"Consistent"
Consistent between whom?
Two agents asking the same question get the same cited policy version
"Correct"
What is a wrong answer, what does it cost?
Zero high-severity wrong policy answers on the test set
"Safe"
Which outputs must never happen?
No write action and no personal data in logs
"Easy to use"
Which task, how many steps?
An agent completes an order lookup in the pilot without help
Clarify by decision tree
Figure 2. Clarifying one vague word
flowchart TD
accTitle: Clarifying decision tree
accDescr: For each vague word, ask whether it can be counted. If not, ask what event would show it. Then ask whether a baseline exists. If not, measure it. Then ask who owns the target. If nobody, escalate to the sponsor. If yes, write the outcome.
A["A vague word"] --> B{"Can an event be counted?"}
B -- no --> C["Ask: what would you see if it were better?"]
C --> B
B -- yes --> D{"Baseline exists?"}
D -- no --> E["Measure a small sample"]
E --> F
D -- yes --> F{"Someone owns the target?"}
F -- no --> G["Ask the sponsor to name an owner"]
G --> F
F -- yes --> H["Write the outcome and its target"]
How to read this: the two loops turn a feeling into an event and a number into someone's decision.
The ladder from outcome to test
Level
Example
Owner
Business outcome
Agents answer order and policy contacts faster, without more wrong answers
Head of Customer Operations
Measurable outcome
Median handling time falls from 9 to 6 minutes or less
Same
System requirement
Every policy answer cites a policy id and version
You and the policy owner
Test
A policy test set run in the evaluation harness
You
You own the last two rows. The first two belong to the customer: do not invent a number for a sponsor who will not own it.
The outcome card
Each outcome has the same parts. Google's SRE workbook recommends an indicator as good events divided by total events, and separates the specification (what matters to users) from the implementation (how you measure it).
OUTCOME CARD
Outcome: <one sentence, in the customer's words>
Indicator: <what is counted, as a ratio or a time>
Population: <which contacts, which agents>
Baseline: <value, sample size, date, who measured>
Target: <value, owned by whom>
Measured: <where, how, over which window>
Unknown: <assumptions still open>
PROBLEM STATEMENT (4 sentences)
1. Who has the problem, in which task.
2. What it costs today, with the baseline and its sample size.
3. What change the customer wants, as outcomes.
4. What is out of scope for the first release.
Explain it back. Tell the sponsor in three sentences why "answer faster" cannot go into the proposal as written. A good explanation names the missing start and end events, says a target needs a baseline, and offers a next step (time more real contacts this week).
Worked example
Assumptions: illustrative; the policy test set is 40 questions the policy owner and agents write together: 30 answerable and 10 the library cannot answer. Nine of the 30 are version traps, one per document that exists in more than one version.
Time. Baseline median 9 minutes (24 timings, 6 agents). Target: 6 or less, set by the Head of Customer Operations. The fall is 9 - 6 = 3 minutes, 3 / 9 = 33 percent. Across 6,720 assisted contacts a month (70 percent of 9,600) that frees 3 x 6,720 = 20,160 minutes, or 336 hours of time (time freed, not cash saved; see the estimate).
Quality on a small set. A rule "at least 27 of 30 answerable questions correct and cited" is 90 percent. The 95 percent Wilson interval for 27 of 30 runs from about 74 to 97 percent, so a pass fits a true rate well under 90. "Zero high-severity errors in 30" does not show a rate of zero: it is compatible with a rate up to about 11 percent (the upper end of the same interval for 0 of 30).
Outcome
Indicator
Baseline
Target
Measured
Handling time
Median minutes, contact opened to answer sent
9 (n = 24)
6 or less
Pilot log
Cited answers
Policy answers citing policy id and version
Not measured
100 percent of answers shown
Test set and reviewer sample
Wrong policy answers
High-severity errors (a superseded or invented rule)
Not measured
0 on 30 answerable questions
Test set
Refusal
Unsupported questions declined
Not measured
10 of 10
Test set
Consistency
Same question, same cited version
Not measured
Same version on 9 of 9 version traps
Test set
"Not measured" is honest: measure those baselines first in the next phase.
At the customer: the sponsor asks, "Can you promise no wrong answers?" Say: "Not before we define what counts as wrong and test on questions your policy owner wrote. With 30 questions zero errors still leaves a range, so I will report the count and the interval."
Practice task
Take the sentence "Make agents faster, and never quote an old policy." Using the card and the template above, write:
A four-sentence problem statement for Lumen.
Two outcome cards: handling time, and policy versions.
Three things you still do not know, each with the person who can answer.
Check your work: every number is a baseline with a sample size or a target with an owner. Together the cards are your outcome sheet.
Decide first, then reveal.
Turn the phrase into an outcome (interactive when JavaScript is on)
A team lead wants the assistant "easy to use". What goes on the outcome sheet?
A usability score out of 10
Ask which task and how many steps: an agent completes an order lookup in the pilot without help
Nothing: ease cannot be measured
Answer
Ask which task and how many steps: an agent completes an order lookup in the pilot without help. Ask what would visibly change and count that event. A score has no event behind it.
The assistant makes 0 high-severity errors on the 30 answerable questions. The sponsor wants "error rate is zero" in the proposal.
Report the count and the interval: 0 of 30 fits a rate up to about 11 percent
Write it: zero errors means a zero rate
Add easier questions so zero is more likely
Answer
Report the count and the interval: 0 of 30 fits a rate up to about 11 percent. A small test set cannot show a rate of zero. Easier questions change the test, not the system.
Pilot handling times: most contacts are short, a few take an hour. Which number goes on the sheet?
The mean, as it uses every contact
The longest contact
The median, with the sample size
Answer
The median, with the sample size. A few long contacts pull the mean up, and the median matches the indicator. The longest says nothing typical.
Exercise
Rewrite these five phrases as outcome cards and flag which lack a baseline: "answers should be quick", "policy answers must be correct", "it must be easy for agents", "it must handle our policies", "it must not hallucinate".
Acceptance criteria:
Each card has an indicator that counts events, a population and a place to measure.
Each card gives a baseline or says how it will be measured.
No card names a model, a vendor or a tool.
Done when a colleague who has never met the customer could run the measurement from your card alone.
Hint 1: ask for the event
For each phrase, write the one thing that would visibly change: a timestamp, a cited policy id, a refusal. That event is the numerator of your indicator.
Hint 2: "hallucinate" is not an indicator
Define it for Lumen: an answer states a rule that no current policy passage supports. The indicator is the share of shown answers with no supporting passage, found by a named reviewer.
Common mistakes
Promising a target before the baseline. A median of 9 from 24 timings is a start, not a proof.
Using the mean for skewed times. Report the median and the sample size.
Taking a pass at face value. A small test set has wide intervals.
Setting the target yourself. The customer owns "good enough".
Mixing outcomes and features. "A search box" is a feature; "median 6 minutes" is an outcome.
Knowledge check
1. Why use the measured baseline before agreeing a target?
Without it you cannot tell whether a target is easy, hard or impossible, or show later that anything changed. Even 24 timings turn "faster" into a distance the customer can judge and own.
2. Why is the 9-minute baseline only an estimate?
It rests on 24 timings from 6 of 24 agents, and four timings from one agent are not independent. It is enough to size the problem and set a first target, not to promise a result, so plan more timings.
3. Why does "at least 90 percent of 30 questions" really mean "at least 27"?
90 percent of 30 is 27 exactly, so the first count that satisfies the rule is 27. Writing the count into the requirement removes any argument about rounding at sign-off, and it shows the reader how few questions the rule rests on.
4. Who should set the target for handling time, and why?
The Head of Customer Operations. They know what a faster contact is worth and what a wrong answer costs. You supply the baseline, options and likely consequences, but a target you chose leaves the customer owning nothing.
5. Why is "zero personal data in logs" easier to state than "correct"?
You can name the event, a personal-data field appearing in a log, and count it, and the rule comes from the security lead. "Correct" needs a definition of wrong, a test set and a threshold, all agreed first.
You can now
Add the outcome sheet to your design pack.
Decode a vague phrase into an event, an indicator and a threshold.
Report the baseline with its median and sample size.
Explain why a threshold on a small test set carries uncertainty.
Separate outcomes the customer owns from requirements you own.
Lesson 2 of 8 · AI Solution Architect: Discovery, Scope and Security
Turn a request like make agents faster and consistent into a problem statement and an outcome sheet with indicators, baselines and targets you can test.
In the rest of this lesson
Practice task
Exercise
Common mistakes
Knowledge check
You can now
Further reading
Enrolling is free. It unlocks every lesson in AI Solution Architect: Discovery, Scope and Security, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.