Cut an AI pilot to the smallest scope that tests the riskiest assumption, then write acceptance criteria with a threshold and a measurement a test can fail.
What you'll learn: how to cut a wish list to a minimum viable product (MVP) that tests the riskiest assumption inside a fixed effort, and how to write acceptance criteria that name a test set, a count and an owner.
At a glance
Figure 1. Cutting scope and fixing acceptance
flowchart LR
accTitle: From wish list to signed acceptance
accDescr: A wish list is scored on value, risk, data readiness and effort. Each item goes in, later or out. The items that are in get acceptance criteria. The criteria are agreed with the sponsor, tested, and the result is signed off or goes to change control.
A["Wish list"] --> B["Score: value, risk, data, effort"]
B --> C{"In, later or out"}
C -- in --> D["Acceptance criteria"]
C -- later --> E["Written parking list"]
C -- out --> F["Written exclusions"]
D --> G["Agree with sponsor"]
G --> H["Test and sign off"]
How to read this: "later" and "out" are written down, not silently dropped, so the exclusions are part of what the sponsor agreed.
An MVP is the smallest build that tests your riskiest assumption.
Score every wish, then say in, later or out, in writing.
An acceptance criterion names the test set, the measure and a pass rule written as a count.
Your artifact: MVP scope and acceptance criteria.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, wants its Order and Policy Assistant, which reads the order management system (OMS), for 24 customer-service agents. The wish list: order lookup, cited policy answers, combined questions, address changes, discount exceptions, return authorisations. The riskiest assumption: answers can always come from the current policy version, because 9 of 62 documents exist in several versions and an old one is still searchable. All numbers are illustrative.
What you will be able to do
Score candidate features and decide in, later or out with a reason.
Check an MVP against a capacity in person-days.
Write an acceptance criterion with a test set, a measure, a count and an owner.
An MVP answers one question the customer cannot answer today. Here: can every answer come from the current policy? Only what tests that question or keeps it safe goes in.
Figure 2. Scope matrix (illustrative effort in person-days)
Item
Risk
Effort
Decision
Order lookup from the OMS, agent's identity
Medium
10
In
Current-version policy search
High
14
In
Cited answers, with refusal
High
12
In
Combined answers: order status plus policy
Medium
6
In
Answer log without personal data
Medium
5
In
Evaluation harness and test set
Medium
9
In
Identity, gateway, security evidence
Medium
8
In
Pilot support and handover
Low
4
In
Change address, with human approval
High
15
Later
Discount exception proposal
High
12
Later
Create a return authorisation
High
10
Later
Vector search
Medium
8
Later, if evaluation shows a need
Write to the ticketing system
Medium
not sized
Out
How to read this: "in" items carry the riskiest assumption or keep it safe. "Later" items need a human approval step and an idempotency key (a unique request id, so a retry cannot apply twice).
SCOPE STATEMENT
In scope: <features, each with its acceptance criteria ID>
Out of scope: <features, each with the reason>
Later: <features, each with the condition that brings it back>
Assumptions: <taken as true, each with an owner to confirm>
Dependencies: <what the customer provides, by when>
Capacity: <person-days available minus the estimate>
Acceptance criteria that can fail
"Answers are accurate" cannot fail. "At least 27 of the 30 answerable test questions are correct and cite a policy id and version" can.
ACCEPTANCE CRITERION
ID: AC-<n>
Outcome: <link to the outcome card>
Test set: <named set and version, frozen before the run>
Measure: <indicator, how measured, where>
Pass rule: <threshold written as a count>
Evidence: <report or file the sponsor can open>
Signed by: <name and role>
If missed: fix | re-scope | waive with written sign-off
Figure 3. From criterion to sign-off
flowchart TD
accTitle: Acceptance flow
accDescr: A criterion is agreed and its test set is frozen. The test runs and produces a result. A pass is signed off. A miss goes to a decision: fix and rerun, re-scope with change control, or waive with a written sign-off.
A["Criterion agreed, test set frozen"] --> B["Run the test"]
B --> C{"Pass rule met?"}
C -- yes --> D["Sign off with the report"]
C -- no --> E{"Decision"}
E -- fix --> B
E -- re-scope --> F["Change control"]
E -- waive --> G["Written waiver, owner signs"]
How to read this: freezing the test set before the run stops anyone adjusting the questions after seeing the result.
Decision exercise. A team lead asks to add "change the delivery address" to the pilot. Options: (A) add it and drop the evaluation harness; (B) add it behind human approval and cut nothing; (C) keep it in "later" and show a labelled mock-up. Choose one and justify it with the capacity numbers in the worked example.
Model answer
Choose C. Option A removes the only way to prove answers are grounded, and 68 - 9 + 15 = 74 is still over capacity. Option B adds 15 days to a full plan, and a write also needs the P-ORD-04 warehouse rule. C keeps the riskiest assumption testable.
Worked example
Assumptions: illustrative; 2 engineers, 40 working days each, 75 percent availability, plus an architect at 0.2 x 40 days.
Engineers: 2 x 40 x 0.75 = 60 person-days. Architect: 0.2 x 40 = 8. Capacity: 68.
The "in" rows sum to 10 + 14 + 12 + 6 + 5 + 9 + 8 + 4 = 68: the plan is full.
Adding change-address: 68 + 15 = 83, which is 15 days (22 percent) over.
The policy test set has 40 questions (illustrative; the policy owner and agents write them): 30 answerable, including the 9 version traps, and 10 unsupported. AC-4 to AC-6 use test orders and the gateway log. Each acceptance criterion (AC) can fail:
ID
Test set
Pass rule
AC-1
30 answerable policy questions
At least 27 correct, each citing policy id and version; 0 high-severity errors (a superseded or invented rule)
AC-2
9 version traps (one per multi-version document)
All 9 from the current version: a return 20 days after delivery is accepted under P-RET-01 v3 (30 days), never refused under v2 (14 days)
AC-3
10 unsupported questions
Declines all 10
AC-4
Test orders on credit held
Reports the hold, says only Finance can release it (P-HLD-05), never offers to
AC-5
One confirmed and one picked order, address question
Confirmed: allowed. Picked: needs warehouse approval. Both cite P-ORD-04
AC-6
Gateway log, all test runs
0 write calls to the OMS
If AC-1 scored 26 of 30, Figure 3 applies. It is never "close enough".
At the customer: "Can we just see how it feels and decide later?" Say: "Yes, weekly. Sign-off still needs a pass rule agreed today."
Practice task
Add two requests from the Head of Customer Operations to the matrix: "update the ticket automatically" and "answer in the agent's language".
Score the new items on risk and effort and mark each in, later or out with one reason.
Check your "in" list against 68 person-days.
Write the scope statement and AC-1 to AC-3 from the templates.
Check yourself: ticket updates change a system the MVP leaves untouched, so out. Another language needs its own test set and the plan has no free days, so "later".
Decide first, then reveal.
Scope and acceptance calls (interactive when JavaScript is on)
A developer wants vector search in the MVP. The plan is full at 68 of 68 person-days.
Keep it in "later" until the evaluation shows a need
Add it and drop pilot support, only 4 days
Add it: the 75 percent availability leaves slack
Answer
Keep it in "later" until the evaluation shows a need. Vector search is sized at 8 days, so dropping 4 still leaves the plan over capacity. The 75 percent already holds back a quarter of the time.
In a test on a credit-held order, the assistant offers to release the hold. Which criterion fails?
AC-6, it wrote to the OMS
AC-4: it must say only Finance can release it, never offer
AC-1, fewer than 27 of 30 correct
Answer
AC-4: it must say only Finance can release it, never offer. AC-4 requires saying only Finance can release a hold (P-HLD-05). AC-6 counts write calls; an offer is not one.
Midway through testing, someone wants to swap three hard questions in the AC-1 set for easier ones.
Allow it: the policy owner wrote the questions
Allow it and rerun only the changed questions
Refuse: the set is frozen before the run, so a change goes to change control
Answer
Refuse: the set is frozen before the run, so a change goes to change control. A frozen set stops anyone adjusting questions after seeing the result. Rerunning only the changed ones would hide that.
Exercise
Write AC-7 and AC-8: one for "every answer is traceable in logs" and one for "the assistant never uses a superseded version for an order-change rule".
Acceptance criteria for your criteria:
Each names a test set, a count and an evidence file, and could fail on a real run.
Neither uses "good" or "correct" undefined.
Done when a colleague could write the test from both without asking you a question.
Hint 1: define "traceable"
Decide what one log entry must contain (request id, agent role, policy ids and versions, answer) and test a sample of stated size.
Hint 2: name the planted case
The test set is the questions where two versions differ. Count them and say how many must use the current version.
Common mistakes
Cutting scope silently. Unwritten exclusions come back as "you said you would".
Criteria without a frozen test set. Name and version the set first.
Calling a demo an acceptance test. A demo shows one path; a criterion tests a population.
Knowledge check
1. Why does the MVP include the evaluation harness, which agents never see?
Without a harness there is no measured result, so the pilot could neither pass nor fail. A feature nobody sees can still prove the outcome.
2. What is wrong with the criterion "answers are accurate"?
It has no test set, no definition and no pass rule, so it cannot fail. A testable version names the set, the measure and a count such as 27 of 30.
3. Why write down the "later" and "out" lists?
An unwritten boundary is easy to dispute. Written reasons and return conditions let the sponsor change a decision deliberately.
4. Why is the plan full at 68 of 68 person-days not a problem?
The 75 percent availability already holds back a quarter of the engineers' time, so the buffer is inside the 60 days. Nothing can be added without a swap.
5. AC-1 scores 26 of 30 against a rule of 27. What do you do?
Do not round it up. Fix and rerun if the failures share a cause, go to change control if the threshold is questioned, or take a written waiver.
You can now
Add MVP scope and acceptance criteria to your design pack.
Score a wish list and cut it with a written reason per item.
Check an MVP against a capacity in person-days.
Write a criterion with a test set and a count-based pass rule.
Lesson 3 of 8 · AI Solution Architect: Discovery, Scope and Security
Cut an AI pilot to the smallest scope that tests the riskiest assumption, then write acceptance criteria with a threshold and a measurement a test can fail.
In the rest of this lesson
Worked example
Practice task
Exercise
Common mistakes
Knowledge check
You can now
Enrolling is free. It unlocks every lesson in AI Solution Architect: Discovery, Scope and Security, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.