# AI design-review checklist

From the AI Architect course on buildingai.in: https://buildingai.in/learning/ai-architect-templates

Every question comes from a decision framework, a failure table, a checklist or the design-review question table in the lesson named at the end of its group. It adds nothing the lessons do not teach. Open the lesson for the reasoning behind a question.

## How to use it

1. Run it at the design review: after the diagrams and the draft decision records are on the table, before the decision is recorded. The review itself is described in https://buildingai.in/learning/ai-architect-mindset#design-reviews
2. Answer every question with Yes, No or N/A. N/A needs a reason, for example "no tools in this design".
3. Every No needs an owner, a date and, if the design still goes ahead, a condition. The page to read is at the end of the group.
4. Record the answers in the ADR for the decision under review (the "Design-review record" section of the ADR template), then record the outcome: approved, approved with conditions, or not approved.
5. Run it again when the design, the model, the prompt or a constraint changes.

## 1. Requirements and design reviews

- RQ1. Does every requirement have a number and a measurement point?
- RQ2. Can the owner of each requirement state what result would make them say no?
- RQ3. Is each constraint marked fixed or negotiable, with a named owner?
- RQ4. Were options that break a fixed constraint excluded before the options were compared?
- RQ5. Was an owned capability trialled first, before buying or building?
- RQ6. Is the risk tolerance named by someone with authority and written down where the review can check it?
- RQ7. Is every decision that is costly to reverse, or that constrains later choices, written as an ADR before the build?
- RQ8. Does each ADR have an owner and a review trigger?
- RQ9. Where one wrong output is high risk and hard to detect, is there a human approval step that is tested as a requirement?
- RQ10. Where a quality bar is claimed, is an interval that straddles the bar reported as not demonstrated and not as a pass?
- RQ11. Does any change to the model or prompt re-run the release set, with a drop below the bar blocking promotion?
- RQ12. Is the worst plausible output known, and who is harmed if it reaches a user?
- RQ13. Is the behaviour defined for a provider that is slow, down or returns something malformed?
- RQ14. Is the cost per request known at expected volume and at ten times that?
- RQ15. Is it known which data goes to which component, and where it is logged?
- RQ16. Is it written down what the system can do without a human approving it, and the maximum damage of one wrong action?
- RQ17. Is there a way to detect that quality dropped after a model or prompt change?
- RQ18. Is the rollback defined, and is its duration known?
- RQ19. Is the simplest non-AI option named, with the reason it is not enough?
- RQ20. Is the decision that is hardest to reverse written down?

Read: page 2, https://buildingai.in/learning/ai-architect-mindset

## 2. Diagrams

- DG1. Does each diagram have a figure title and one sentence saying what question it answers?
- DG2. Are shapes, line styles and zone markers explained beside each diagram?
- DG3. Do shape, line style or text carry whatever colour shows?
- DG4. Is each diagram about 15 nodes at most, or split into an overview and zoom-ins?
- DG5. Does every component keep the same name in every diagram and in the text?
- DG6. Does every arrow have one direction and a label for what travels?
- DG7. Does at least one diagram show a failure path (timeout, empty result or denial)?
- DG8. Are trust boundaries visible, with the control named at each crossing?
- DG9. Is the source of each diagram text next to the code, reviewed with it?
- DG10. Where there are 3 or more participants, or any async step, is there a sequence diagram?
- DG11. Where a component can loop or end more than one way, is there a state diagram in which every state has an exit?
- DG12. Has the review walked every outbound arrow from the application zone, asking which boundary it crosses and who holds the credential?
- DG13. Does each user-facing sequence have companions for empty-result, timeout and denial?

Read: page 3, https://buildingai.in/learning/ai-architect-diagramming

## 3. Models and routing

- MR1. Where the request type is known before the call, are requests routed by rules first?
- MR2. Where a cascade is used, can the cheap answers be scored and is a slower tail acceptable?
- MR3. Is the cascade threshold set on labelled data, with costly intents always sent to the large model?
- MR4. Is the prompt laid out for prefix caching, with stable content first?
- MR5. Where an answer depends on user, tenant, time or changing data, is semantic caching avoided?
- MR6. Do cache keys include the tenant and the version, with expiry on a version change and hits logged?
- MR7. Does an alert watch the cache-read share, with one warm-up request per release?
- MR8. Is each routing decision logged with its route, reason and confidence, and is a weekly sample reviewed per intent?
- MR9. Are model versions pinned and tokens counted with the target tokenizer, with trimming by policy or a defined error?
- MR10. Where a user is waiting on a long answer, does the response stream?
- MR11. If a stream fails after the first token, is the partial text kept and marked incomplete, with no tool call from it and an explicit retry?
- MR12. Where residency, pinning or sustained load is a hard requirement, was self-hosting evaluated, with an API behind a gateway otherwise?

Read: page 4, https://buildingai.in/learning/ai-architect-models-routing

## 4. Retrieval

- RT1. Where exact search meets the latency budget, is it used with a filter?
- RT2. Where queries contain identifiers or defined terms, is there a lexical index fused with RRF?
- RT3. Where recall at 50 is good and at 5 is poor, is a reranker added?
- RT4. Is the access filter applied from verified identity inside the search and not afterwards?
- RT5. Is a change of model, chunker or index parameters built as a new index beside the old one and then switched?
- RT6. Do sources keep version, date and status, with the superseding document preferred?
- RT7. Does the Retriever refuse a query when the index and the embedding model do not match?
- RT8. When a filtered query returns too few results, does the system abstain below the minimum?
- RT9. Does a version count as indexed only when both the lexical and the vector writes succeed?
- RT10. If the reranker times out, is the fused order used and the trace marked?
- RT11. Below the evidence threshold, is the model not called?
- RT12. When a permission is revoked, is the filter metadata updated first, with permission staleness measured separately?

Read: page 5, https://buildingai.in/learning/ai-architect-data-retrieval

## 5. Agents and tools

- AT1. Where the steps are known in advance, is the design a workflow in code and not an agent loop?
- AT2. Does each irreversible tool have an idempotency key, a policy decision and approval above a limit?
- AT3. Are calls, tokens and time capped from measured runs?
- AT4. Are large tool results paginated, filtered, truncated or returned as a handle?
- AT5. Are servers you do not own allow-listed, pinned, prefixed and treated as untrusted?
- AT6. Where facts must outlive a conversation, is memory scoped, with expiring and policy-checked writes?
- AT7. Does a used-up budget end the run in BudgetStop, with partial findings and one audit event?
- AT8. Does an identical repeated call get a tool error?
- AT9. Does every write carry an idempotency key, with the target returning the first result and one layer retrying?
- AT10. Do tool results stay in tool-result blocks, with the Policy engine deciding from identity and validated arguments and approval on outbound tools?
- AT11. When the context fills mid-run, is the transcript archived and compacted or cleared, or the run stopped at a resumable checkpoint?
- AT12. Does an unanswered approval request expire and return a tool error to the model?

Read: page 6, https://buildingai.in/learning/ai-architect-agents-tools-mcp

## 6. Multi-agent and event-driven designs

- EB1. Where a task cannot be split into independent parallel subtasks, or its parts share context, is it one agent or a fixed workflow?
- EB2. Where work can wait seconds and arrives in bursts, is there an Event bus between the Orchestrator and the Workers?
- EB3. Are offsets committed after processing, with an idempotent effect keyed by event id?
- EB4. Where a state change and its event must agree, is it one local transaction through an outbox?
- EB5. For an action that spans services and has irreversible steps, is there a saga with idempotent compensations?
- EB6. Is there a retry limit with backoff, a dead-letter queue with error context, a per-key quarantine and an alarm on depth?
- EB7. Is the batch size times the worst-case time per event well inside the poll interval?
- EB8. Does a failed saga compensation get idempotent retries and then a human queue with the full state?
- EB9. For work you cannot lose, does a consumer that returns after its offset expired use none or earliest, with the replay deduplicated?
- EB10. Are there effort rules by task type, a cap on subagents and a token budget per job?
- EB11. After a Worker crash, is the offset committed after the batch, with dedupe skipping applied events?

Read: page 7, https://buildingai.in/learning/ai-architect-multi-agent-eventdriven

## 7. Evaluation

- EV1. Is there a labelled golden set, started with 20 to 50 real items and extended with every incident?
- EV2. Where a judge decides the release, has it been calibrated, with kappa and failure recall reported?
- EV3. When two versions are compared, is the judging pairwise, with a position swap, on the same items?
- EV4. Where the feature uses retrieval, are retrieval and generation metrics reported per slice?
- EV5. Does the gate use a confidence bound and state the detectable drop?
- EV6. For an agent that acts on the world, are the end state, the policy steps and pass^k measured?
- EV7. Does a malformed judge output retry once and then count as an evaluation error, with a run above the error limit invalid?
- EV8. Are judge versions stored on every score, with recalibration and a new baseline when the judge changes?
- EV9. Does every slice report n and an interval, showing not measurable below the minimum count?
- EV10. Does the judge come from another model family than the system, or is a human-confirmed sample used?
- EV11. Is an inconclusive gate blocked by default?

Read: page 8, https://buildingai.in/learning/ai-architect-evaluation

## 8. Release and operations

- RO1. Does every change to an answer-shaping part get a release ID stamped on every span?
- RO2. For a change with side-effecting tools, is shadow run with stubbed tools and the canary behind approval?
- RO3. Where the canary yields fewer judged items per day than needed, is it used for hard signals, with the share or the duration widened for quality?
- RO4. Do traces hold identifiers and redacted text with short retention?
- RO5. Where teams share a model account, are gateway tags required and reconciled with the provider each month?
- RO6. Does the message header carry traceparent across a queue, with an alert on root spans that have no downstream spans?
- RO7. If the trace store or the collector is down, are requests unaffected, with spans dropped, counted and alerted?
- RO8. Does an inconclusive canary hold at its stage, with items added or a review, and never a re-run until green?
- RO9. Does the release record store the retirement date of its rollback target, with an alert ahead of it?
- RO10. Are content attributes off by default, redaction run at the collector and tests run with seeded fake personal data?
- RO11. When untagged spend rises, are direct keys revoked and the gap alert sent to the platform owner?

Read: page 9, https://buildingai.in/learning/ai-architect-llmops-observability

## 9. Reliability

- RL1. Is there one retry owner per failure class, with SDK retries set to 0?
- RL2. Is a token bucket sized to the limit in the Model gateway?
- RL3. Do retry loops use full or decorrelated jitter and an attempt limit?
- RL4. Does each feature have its own bulkhead pool, sized with Little's law?
- RL5. Is work that can wait queued with a deadline stamp and bounded consumers?
- RL6. Are hedged requests limited to idempotent reads, sent after the p95 delay and under a cost cap?
- RL7. When the spend cap is reached, are retries skipped, a degraded path taken and the budget owner alerted?
- RL8. When a provider region is down, does the breaker open on 5xx and timeouts, do calls go to the second deployment and does a status banner show?
- RL9. Are jobs past their deadline stamp skipped and reported?
- RL10. While autoscaling, does an aggregate bucket cap total calls, with a gradual scale-out ramp?
- RL11. On a 429, does the bucket pause to retry-after, with one owner retrying with jitter?

Read: page 10, https://buildingai.in/learning/ai-architect-reliability-scaling

## 10. Cost

- CO1. Is per-task cost measured at the Model gateway before any lever is chosen?
- CO2. Where output tokens dominate, were the format and max_tokens shortened first?
- CO3. Where a stable prefix is reused above break-even, is the prompt laid out for prefix caching?
- CO4. Where nobody waits and the deadline exceeds the window, is the work batched with a cutoff and a fallback?
- CO5. Is easy traffic routed by rule, with the rest cascaded?
- CO6. Where sustained load keeps utilisation above r, was self-hosting evaluated with operating cost?
- CO7. Does a team budget degrade at a soft limit, reject at a hard limit with a defined error and queue non-urgent work?
- CO8. Does a per-task cap on calls and tokens stop a runaway agent loop, returning a partial result and logging the cap?
- CO9. Does an alert fire on the cache read share, with a warm-up request after a prompt release?
- CO10. Does an alert fire at 80% of the cascade's break-even escalation rate, with those intents moved to the large model by rule?
- CO11. Does a batch that misses its window cancel at the cutoff, with the stragglers run in real time within rate limits?
- CO12. Does the cost model read ratios from config with a review date, and recount tokens on version changes?
- CO13. Were scenarios reported and not one forecast number, with a monthly refresh and a kill switch for optional features?

Read: page 11, https://buildingai.in/learning/ai-architect-cost-economics

## 11. Security

- SE1. Is every source of untrusted text drawn as an entry point with an owner and a control?
- SE2. Where the model's output can trigger a tool, does authority come from the session and a scoped credential and never from a model-written value?
- SE3. For an irreversible or external action, is approval required, showing the exact content and restricting destinations?
- SE4. Is any classifier kept as one layer next to structural controls?
- SE5. When injected text asks for a send, does the Policy engine deny it, does the step have no send tool and is a security event logged?
- SE6. If the classifier is down or refuses, are tool calls blocked while answer-only paths continue with remote images off?
- SE7. Do retention and read limits cap the exposure of a value the redactor misses, with a weekly seeded test?
- SE8. Does the approved description of a third-party MCP tool stay pinned until it is reviewed?
- SE9. Do approvals cover only external side effects, with a sample reviewed?
- SE10. If the Policy engine is down, are calls refused and the user told?

Read: page 12, https://buildingai.in/learning/ai-architect-security-threat-modeling

## 12. Governance

- GV1. Is every AI system registered with its intended purpose before launch?
- GV2. Where the intended use is in an Annex III area, is it treated as high-risk, with counsel asked before relying on Article 6(3)?
- GV3. Where decision records hold personal data, are ids and hashes kept in the audit store and the content elsewhere?
- GV4. Where a person can override the output, are override rate, review time and sampled agreement measured?
- GV5. Where the system talks to the public, is the disclosure made at first interaction, with the version shown logged?
- GV6. Does a change of model, prompt, retrieval or purpose run change control: re-evaluate, update the card, re-classify?
- GV7. Is a production system that is missing from the registry treated as unclassified, registered by a deadline or switched off?
- GV8. If the audit store is unavailable, are the decisions that need a record not released?
- GV9. Can an impact query list the answers that cited a superseded source document, for re-review?
- GV10. Where the classification is genuinely ambiguous, is the stricter handling used and counsel asked?
- GV11. Do sampled agreement and review time raise an alert when reviewers approve almost everything, quickly?

Read: page 13, https://buildingai.in/learning/ai-architect-governance-responsible-ai

Nothing in this checklist is legal advice.

## 13. Cloud

- CL1. Where a workload holds a long-lived key, has it moved to workload identity with short-lived credentials?
- CL2. Where the model endpoint is public, is there a private endpoint with the public path closed?
- CL3. Where a rule names a region or geography, is the narrowest routing level the rule allows chosen and verified in the audit log?
- CL4. Where a feature stores prompts or answers, are they redacted before storage, with retention set?
- CL5. Where stored artifacts need your control, is there a customer-managed key, with revocation rehearsed?
- CL6. If private DNS is not reachable from another network, does the call fail closed or get denied by the network rule?
- CL7. When a role was just assigned, does the caller retry briefly and then alert?
- CL8. If the key is disabled or unavailable, do requests that need protected data fail with a clear error and an alert?
- CL9. Does an audit check find inference routed outside the allowed region and raise an incident?
- CL10. Where content logging captures personal data, do redaction, retention and access limits cap the exposure?
- CL11. When a call uses an endpoint the log does not cover, does a pipeline test fail?
- CL12. Is DNS planned for on-premises and other networks, with name resolution tested from each?
- CL13. Does the private endpoint carry a policy or network rule naming the principals and actions allowed?
- CL14. Are stored keys removed from images, environment files and CI variables?
- CL15. Do administration and inference use separate identities?
- CL16. Does each service have its own least-privilege role, so the audit log names who called?
- CL17. Before the design says it is encrypted with your key, has each provider's list of what the key covers been read?
- CL18. Are prompts in transit protected by TLS and the private path, and not assumed to be covered by the key?
- CL19. Is the content-logging policy decided in the design review and recorded?
- CL20. Is there one adapter per provider behind the Model gateway, so a migration changes an adapter and a configuration and not the Orchestrator?
- CL21. Is anything not compared across providers, such as tool schemas or caching, treated as untested until the evaluation set says otherwise?
- CL22. Does a CI check fail the build on a configuration with a stored key, a public path, a region outside the allowed set or unredacted content logging?
- CL23. Does each of the six parts (identity, private path, keys, logs, region and capacity) have a named setting and a source link?

Read: page 14, https://buildingai.in/learning/ai-architect-cloud-platforms. It points to page 10 for retries, https://buildingai.in/learning/ai-architect-reliability-scaling, to page 12 for redaction, https://buildingai.in/learning/ai-architect-security-threat-modeling, and to page 13 for audit-trail rules, https://buildingai.in/learning/ai-architect-governance-responsible-ai

## Recording the answers

Paste this block into the "Design-review record" of the ADR and fill it in.

- Checklist run on: <YYYY-MM-DD>
- Answered No or N/A: <question ID: reason, owner, date>
- Outcome: <Approved | Approved with conditions | Not approved>