Audit every source an AI assistant needs before MVP scope is signed: score owner, access, freshness, quality and legal basis, then mark each in, fix or out.
What you'll learn: how to audit every source an assistant will read before MVP scope is signed: score eight facts per source, apply a green, amber and red rule, and write the findings into the proposal as assumptions with owners.
At a glance
Figure 1. The readiness audit, from source list to proposal
flowchart LR
accTitle: The data readiness audit
accDescr: List every source the assistant needs. Score each source on eight columns. Apply the green, amber and red rule. Each source ends as in scope, needs work or out of scope. The findings go into the proposal as assumptions with an owner.
A["List every source"] --> B["Score eight columns each"]
B --> C["Apply the rule"]
C --> D["In scope"]
C --> E["Needs work"]
C --> F["Out of scope"]
D --> G["Proposal: assumptions with owners"]
E --> G
F --> G
How to read this: no source skips the scoring, and every result, including "out", is written into the proposal.
A demo extract proves little: audit the sources, not the sample.
Red on owner, access, sensitive fields or legal basis means out for the MVP.
Red on the data itself means fix first, or phase two.
The assistant reads data and never changes it. Your artifact: the readiness scorecard.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, is a week from signing MVP scope for its Order and Policy Assistant. The sponsor sent a demo extract of 10 policy documents, all tidy. Nobody has listed every source the assistant will read or who owns it. The policy library holds 62 documents, 9 of them in more than one version. All numbers are illustrative.
"Datasheets for Datasets" (Gebru et al., arXiv 1803.09010, version 8, revised 1 December 2021) proposes that every dataset come with a datasheet documenting its motivation, composition, collection process and recommended uses. Its questions cover seven stages, such as "Are there any errors, sources of noise, or redundancies in the dataset?" and "Who will be supporting/hosting/maintaining the dataset?"
The paper documents machine learning datasets. This scorecard is our own adaptation. Completeness, quality and sensitive fields echo its composition questions (it asks about missing information and confidential or sensitive data); owner echoes maintenance. Access route, freshness, format and legal basis are our additions.
Figure 2. The readiness scorecard (copy this)
Column
Ask
Green
Amber
Red
Owner
Who confirms the meaning and fixes it?
A named person
A team only
Nobody
Access route
How is it read, as whom?
Approved API or export, agent's identity
Manual export
None, or a shared login
Freshness
How fast does a change arrive?
Fast enough for the outcome
Slower, with a workaround
Unknown, or can go stale
Completeness
Are the needed fields present?
Yes, sample checked
Gaps with a fallback
Gaps, no fallback
Format
Table, text PDF, scan, spreadsheet?
Table or clean text
Needs conversion
Scans or loose files
Quality
Duplicates, conflicting versions?
None found
Minor, listed
Conflicts the assistant would quote
Sensitive fields
Personal, credit, secrets?
None, or masked at source
Present, rule agreed
Present, no rule
Legal basis
May the customer use it for this?
Legal team confirmed
Asked, open
Not asked, or refused
How to read this: score each source against the ask; the customer's legal or compliance team, not you, decides the legal basis.
Turn scores into a verdict
If the source has
Verdict
Red on owner, access route, sensitive fields or legal basis
Out of scope for the MVP, until the customer fixes it
Red on freshness, completeness, format or quality, and the outcome needs the source
Needs work: the owner fixes it before the assistant answers from it
Red on those, and the outcome does not need the source
Out of scope: phase two
Amber only
In scope, with an assumption that has an owner and a date
All green
In scope
Figure 3. From "source found" to a verdict
flowchart TD
accTitle: Source to verdict
accDescr: A source is found. If it has no named owner, no approved access route, or no legal basis with sensitive fields handled, it is out of scope. Otherwise, if freshness, completeness, format and quality have no red, it is in scope and any ambers become assumptions. If one is red and the outcome needs the source, it needs work. If one is red and the outcome does not need it, it is out of scope.
A["Source found"] --> B{"Named owner?"}
B -- no --> OUT["Out of scope: written exclusion with the reason"]
B -- yes --> C{"Approved access route, agent's identity?"}
C -- no --> OUT
C -- yes --> D{"Legal basis given and sensitive fields handled?"}
D -- no --> OUT
D -- yes --> E{"Red on freshness, completeness, format or quality?"}
E -- no --> IN["In scope: ambers become assumptions"]
E -- yes --> F{"Does the outcome need this source?"}
F -- yes --> NEED["Needs work: owner fixes before answers"]
F -- no --> OUT
How to read this: the first three questions are ownership and permission, which the project cannot fix; the last two are the data.
Decision exercise. The library holds 9 documents in several versions. Options: (A) index all versions and prefer the newest date; (B) the policy owner confirms the current version of each of the 9 first, and only those enter the index; (C) drop policy answers from the MVP. Choose one and justify it.
Model answer
Choose B. Policy contacts are 3,360 of 9,600 a month, so C drops 35 percent of the work and leaves the riskiest assumption untested. A guesses which version is current; only the policy owner can say.
Worked example
Assumptions: illustrative scores. G green, A amber, R red, in the column order of Figure 2.
Figure 4. The Lumen sources scored
Source
Own
Acc
Fre
Com
For
Qua
Sen
Leg
Verdict
Orders and shipments (OMS)
G
G
G
A
G
A
A
A
In, with assumptions
Credit hold flag and reason (OMS)
G
G
G
G
G
G
A
A
In, with assumptions
Policy library, 62 documents
G
A
A
A
A
R
G
G
Needs work
Approval limits (P-DSC-03)
G
A
G
G
A
G
G
G
In for quoting; routing is phase two
Ticket history, free text
A
A
G
A
R
A
R
R
Out of scope
How to read this: one red in the policy library's quality column holds it, even though most of its cells are green or amber.
Policy library. An old version is still searchable. P-RET-01 v2 says 14 days, v3 says 30, so a return 20 days after delivery is refused under v2 and accepted under v3.
Credit hold. The assistant reports the flag and reason. Only Finance can release a hold (P-HLD-05).
Ticket history. Free text holds names and addresses; nobody has asked if the assistant may read it.
The consequence for scope: policy answers become conditional. The policy owner lists the current version of each of the 9 documents first, and the assistant indexes only that list. At 5 minutes a document (an assumption), the owner's check of the 9 takes 9 x 5 = 45 minutes; checking all 62 would take 62 x 5 = 310 minutes, about 5.2 hours.
Figure 5. Readiness findings written as proposal assumptions
#
Assumption
Owner
If it is late
A1
Current version of each of the 9 documents is confirmed, before scope sign-off
Policy owner (Commercial)
Policy answers leave the pilot
A2
Read access to the OMS through the gateway, agent's identity
Platform team
No order lookup
A3
The assistant may read order data, including customer names
Customer's legal or compliance team
Order answers wait; scope is re-cut
How to read this: each row names who can fix the gap and what the pilot loses if they do not.
WARNING
Never fix the customer's data quietly in your own copy. The source stays wrong for everyone else. Send findings to the owner, in writing.
At the customer: "Can't you just clean it as you load it?" Say: "We can mask and filter what we index. Which version is current stays with its owner."
Practice task
Score two more Lumen sources on the Figure 2 scorecard: (1) a spreadsheet one team lead keeps of past discount exceptions, shared by email; (2) carrier tracking data that arrives in batches.
Score all eight columns for each.
Apply the verdict table.
Add an assumptions row for any source that is in scope with an amber.
Check yourself: the spreadsheet has one person as owner, no approved route, and records past practice, not P-DSC-03: out of scope. Batched tracking is amber on freshness: in scope, with an assumption about how late a status may be.
Readiness calls (interactive when JavaScript is on)
A policy exists as v2 and v3. Someone suggests the assistant simply prefers the newest effective date. Is that enough?
Yes, the newest date is the current version
No: the policy owner confirms the current version, and the assistant only indexes the confirmed list
No: delete the old version from the intranet yourself
Answer
No: the policy owner confirms the current version, and the assistant only indexes the confirmed list. Only the policy owner can say which version is current. Deleting from the intranet changes the customer's data, which the assistant and the project never do.
Ticket history holds names and addresses in free text, and nobody has asked the legal team about it. What is the verdict?
In scope: mask the names later
Needs work: the owner fixes the format first
Out of scope for the MVP, with the reason written down
Answer
Out of scope for the MVP, with the reason written down. Red on legal basis and on sensitive fields means out of scope until the customer fixes it. Masking later skips the question the legal team must answer.
Order data is live through the gateway, but some promised dates are empty. Completeness is amber. What is the verdict?
In scope, with an assumption that has an owner and a date
Needs work: wait until every date is filled
Out of scope: incomplete data is unsafe
Answer
In scope, with an assumption that has an owner and a date. Amber only means in scope with an assumption. Waiting for perfect data would hold the pilot for a gap that has a fallback.
Exercise
Spot the flaw. A draft scope says: "Assumption: the policy library is clean and consistent. Evidence: the 10 documents the customer sent for the demo each had one version." What is wrong, and how do you rewrite it?
Even a random draw of 10 from the 62 misses all 9 multi-version documents with probability C(53,10) / C(62,10), about 18 percent (computed). A hand-picked extract is worse. Ask how it was chosen.
Done when your rewrite names the 62 and the 9, an owner, a due date and what happens if it is late.
Hint 1: what does the evidence cover
Only 10 of 62 documents. Who chose them?
Hint 2: turn the claim into an assumption
Replace "is clean" with "is confirmed by the owner by a date".
Common mistakes
Trusting the demo extract. It shows the best documents, not the library.
Deciding the legal basis yourself. The customer's legal or compliance team decides.
Leaving findings in a meeting note. They belong in the proposal.
Knowledge check
1. Why does a red on "owner" put a source out of scope for the MVP, instead of "needs work"?
Nobody can confirm what the data means or fix it, and only the customer can name that person.
2. A source is amber on freshness and green everywhere else. What goes into the proposal?
It is in scope, with an assumption about how late a change may arrive, an owner and a date.
3. Why is the policy library "needs work" and not "out of scope"?
Policy questions are 35 percent of contacts, so the outcome needs the library. The fix is small and sits with one owner, who lists the 9 current versions.
4. What may the assistant do about a wrong value it finds in a source?
It reports the finding to the owner and, if the owner agrees, leaves the value out of its index. It never edits the source.
5. What does the Datasheets for Datasets paper give you here?
A fixed question list across seven stages, to ask of every source. It covers machine learning datasets; the scorecard is our adaptation.
You can now
Add the data readiness scorecard to your design pack.
List every source an assistant needs and score it on eight columns.
Turn scores into a verdict with the rule.
Write findings as assumptions with an owner and a consequence.
Lesson 4 of 8 · AI Solution Architect: Discovery, Scope and Security
Audit every source an AI assistant needs before MVP scope is signed: score owner, access, freshness, quality and legal basis, then mark each in, fix or out.
In the rest of this lesson
Worked example
Practice task
Exercise
Common mistakes
Knowledge check
You can now
Enrolling is free. It unlocks every lesson in AI Solution Architect: Discovery, Scope and Security, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.