AI Architect

Capstone: Two Full AI System Designs

Apply the whole AI Architect course to two complete designs: a regulated knowledge-and-actions copilot and a high-volume event-driven agent platform.

By Avaneesh Kumar YadavPublished 23 Oct 202651 min read

ai-architect
architecture
capstone
design-review

This lesson is part of AI Architect (Advanced): Operate, Secure and Govern. Enroll for free, then sign in, to read it in full.

Enroll to unlock Already enrolled?

What you'll learn: how to turn the whole course into two complete, reviewable designs: requirements with numbers, a full set of views, decision records, evaluation and release gates, a cost envelope, a failure table and a threat model. You finish with a design-review checklist and a readiness quiz.

At a glance

Figure 1. The course in one figure: two designs and the pages each draws on

flowchart LR accTitle: The course in one figure accDescr: Two shared groups of pages feed both designs. Pages 2 to 4 give requirements, views and the Model gateway. Pages 8, 9 and 12 give evaluation, release and security. Design A also draws on retrieval, tools and approval, governance and the private cloud path. Design B also draws on events and sagas, retry ownership with degraded modes, and the cost envelope. S1["Pages 2 to 4: requirements, views, Model gateway"] S2["Pages 8, 9, 12: evaluation, release, security"] A["Design A: regulated copilot"] B["Design B: event-driven agent platform"] A1["Page 5: retrieval and entitlements"] A2["Page 6: tools and approval gates"] A3["Page 13: governance and audit"] A4["Page 14: private cloud path"] B1["Page 7: events, idempotency, saga"] B2["Page 10: retry owner, degraded modes"] B3["Page 11: cost envelope"] S1 --> A S1 --> B S2 --> A S2 --> B A --> A1 A --> A2 A --> A3 A --> A4 B --> B1 B --> B2 B --> B3

How to read this: the two boxes on the left are the method both designs share, the middle boxes are the designs, and the boxes on the right are the pages only that design leans on.

Both designs use one method: testable requirements, a view set, decision records, gates, a cost envelope, a failure table and a threat model.
Design A is limited by authority: who may read which document, which action may run, and what evidence is kept.
Design B is limited by volume: more than 94 percent of events never reach a model.
Every number is an illustrative assumption listed under Assumptions and recomputed with Python. Both scenarios are composite.

How to read a full design

A design is not one diagram. It is a set of views, each answering one question for one reviewer (page 3), tied together by numbers (page 2). Design A uses all fifteen views below, in this order. Design B uses the same set, with governance (view 15) kept to decision B7.

#	View	The question it answers	Drawn as	Page
1	Requirements	What must be true, and how is it measured?	Table with a number and a check	2
2	Context	Who uses it and what does it depend on?	Architecture diagram	3
3	Containers	What are the parts and how do they talk?	Architecture diagram	3
4	Components	Where does each rule live inside the Orchestrator or worker?	Architecture diagram	3
5	Runtime	In what order, and who owns the retry, on the normal and the failure path?	Sequence diagrams	3, 10
6	Trust boundaries	Where does trust change, and what controls the crossing?	Architecture diagram with zones	3, 12
7	Deployment	Where does it run, on what network path?	Architecture diagram	3, 14
8	Data model	What must outlive a request?	ER diagram	3, 13
9	Decisions	Which choices are costly to reverse, and why were they made?	ADR table	2
10	Evaluation	How does the team know it works, and with what uncertainty?	Gate table and widget	8
11	Release	How does a change reach users and roll back?	Release ladder	9
12	Cost	What does a successful task cost, and what moves it?	Cost-unit table and widget	11
13	Failure	What does the system do when each part fails?	Failure table	10
14	Threats	Where does text the team did not write reach something that acts?	Threat table	12
15	Governance	Which tier, which evidence, who oversees?	Registry and audit design	13

How to read this: each row is one view, the third column is the reviewer's question, and the last column is the page that teaches how to draw and judge it.

Each design opens with its requirements, because every later view exists to satisfy or test one of them.

Check yourself: a reviewer asks "who retries when the ticket call is slow, and how many times?" Which of the fifteen views answers it?

The runtime view with the failure path drawn (view 5). The container view has no order and no retry owner, which is why page 3 asks for one sequence per scenario.

Design A: A regulated-industry knowledge and actions copilot

Composite scenario, not a real company. An insurer's claims, underwriting and service staff ask a copilot about internal policy and procedure. It answers from approved documents, with citations, and can propose two actions in the case system. A person approves the risky one. Every number is an illustrative assumption listed under Assumptions.

Tool	Effect class	Retry class	Rule
`policy_search` (the Retriever)	Read-only	Safe to repeat	Entitlement filter inside the search
`task_create`	Reversible write	Repeat only with an idempotency key	The Policy engine decides; a 5 percent sample is reviewed afterwards
`letter_send`	External, irreversible	Never repeat blindly	Approval always, bound to the exact call. One retry only with the call's key, which the Case system must honour

How to read this: the effect-class table from page 6, filled in for this system. The rule column is the control the rest of the design builds.

Requirements

Kind	Requirement (illustrative values)	Check
R1 Quality	At least 90% of answers on a 400-item release set are rated acceptable (rubric v3, two reviewers). The proof gate needs 370 of 400. Re-measured on every prompt, model or index change	Eval service in CI
R2 Latency	p95 end to end at most 8 s for requests under 6,000 input tokens, measured at the API gateway over a rolling 7 days	Load test, then monitor
R3 Cost	Mean all-in cost per successful answer at most 0.016 CU. Soft limit 3,200 CU a month, hard cap 4,000 CU	Cost log and gateway budget
R4 Access and actions	In a 300-case suite of attempts to reach another department's documents or to trigger a letter without approval, all 300 are blocked (lower bound 99.1% on that suite). Re-run on every index change	CI, then quarterly red-team
R5 Audit	Every released answer and executed action has an audit entry appended before release. A monthly reconciliation of audit entries against gateway counts finds a gap of 0	Reconciliation job
R6 Residency	Prompts, passages, traces and audit entries stay in one region (a fixed constraint from the contract)	Egress tests and a configuration review each release
R7 Freshness	A changed or withdrawn policy is searchable within 30 minutes for 99% of changes. A permission revocation applies within 5 minutes	Age of the oldest unprocessed change

How to read this: each row names a number, a place it is measured and a check, so a reviewer can say what result would make them reject it (page 2).

Classification and governance reasoning

Figure 2. How the copilot is classified, and what triggers a new classification

flowchart TD accTitle: Classification reasoning for Design A accDescr: The registry entry states the intended purpose. The first check is whether that use is in an Annex III area, using the points that page 13 quotes and the full Annex read by counsel. If yes or unsure, treat it as high-risk until counsel documents an assessment. If no, check whether the system interacts directly with people, which adds the Article 50 disclosure. Any change of purpose returns to the first step. P["Registry entry: intended purpose in one sentence"] --> Q1{"Is the use in an Annex III area?"} Q1 -- "yes or unsure" --> H["Treat as high-risk until counsel documents an assessment"] Q1 -- "not on the points page 13 quotes" --> C["Counsel reads the full Annex and dates the note"] C --> Q2{"Interacts directly with people?"} Q2 -- yes --> T["Article 50(1): disclose at first interaction and log the version shown"] Q2 -- no --> N["Registry entry, logs, change control"] T --> K{"Does the purpose change?"} N --> K K -- yes --> P K -- no --> R["Keep the record and set a review trigger"]

How to read this: classification attaches to the intended purpose, so the loop back to the first box fires on any change of purpose, not on a model change alone.

WARNING

Not legal advice. This reasoning follows the text as page 13 quotes it, checked on 2026-10-22. The law and its dates have been amended and may change again: ask counsel and read the official pages.

Annex III. Page 13 quotes point 4(a), recruitment and selection, and point 5(b), creditworthiness. The copilot's purpose is neither. Page 13 quotes only those two points, so counsel must still read the full Annex III. This page does not say what it contains.
Recorded result. "Not in an Annex III area on the points quoted on page 13; counsel to confirm and date." A registry field, not a conclusion.
Purpose is the trigger. If the copilot were later used to rank or score individual customers or staff, the system returns to Classified before release (page 13, Figure 5).
Article 50(1) and 50(5). The copilot interacts directly with staff, so a visible line at the first interaction, with the version logged, removes the question of what is obvious (page 13).
Controls chosen anyway. Fail-closed audit and approval-bound actions are design controls chosen because of the actions. Whether any law requires them for this use is a question for counsel.

Context and containers

Figure 3. Context view of Design A

How to read this: person icons are people, the shaded box groups systems the team does not run, and each arrow is one direction with a label for what travels.

The first diagram already exposes one decision: question text and passages leave the system for the LLM provider. The decision table fixes where that provider runs.

Figure 4. Container view of Design A

How to read this: solid arrows are calls the user waits for, the dashed arrow is asynchronous, and the large box is the network the team operates in one region.

The Audit store and the Trace store are separate on purpose: traces are sampled and redacted, the audit log is complete per decision (page 13).
The Retriever, not the Orchestrator, owns the entitlement filter, so no other component can forget it.
Left out to stay near the node limit: the ingestion path (as in page 5, Figure 1) and the Eval service.

The Orchestrator

Figure 5. Component view of the Orchestrator

How to read this: the large box is one container, the boxes inside it are components with one responsibility each, and everything outside is another container from Figure 4.

Component	One responsibility	The rule that lives here	Page
Input checks	Reject oversize or unauthenticated requests	Before any model spend	3
Context builder	Choose what the model sees	Tools, system prompt and reference text first, then passages, then the question, so the prefix cache can hit	4
Workflow runner	Own the step budget	At most 3 model calls: the answer or tool request, the final text after a tool result, and one repair attempt for malformed output. A workflow written in code, not an open loop	6
Tool dispatcher	The only code that calls the Tool server	Validate, hash the exact call, ask the Policy engine, request approval, add the idempotency key	6
Output checks	Decide what reaches the user	Every citation was retrieved; links and images removed; the AI disclosure line present	12
Audit emitter	Append, wait for the acknowledgement	Fail closed: no entry, no release	13

How to read this: each row is a box in Figure 5, with the rule that explains why the box exists and where the course teaches it.

Why not a free loop? The steps are known before the request arrives, so the decision tree of page 6 stops at a workflow written in code. The one action that cannot be undone gets a human decision step inside it.

Runtime: the normal path and the failure path

Figure 6. A normal question, answered with citations and recorded before release

sequenceDiagram accTitle: Normal question in Design A accDescr: The Client app sends a question through the API gateway to the Orchestrator. The Orchestrator appends a received entry to the Audit store, asks the Policy engine for the caller's scopes, searches with the Retriever, calls the model through the Model gateway, checks the draft, appends the outcome and only then returns the answer with a disclosure line. autonumber participant C as Client app participant G as API gateway participant O as Orchestrator participant P as Policy engine participant R as Retriever participant M as Model gateway participant L as LLM provider participant A as Audit store C->>G: question over HTTPS with session token G->>O: question plus verified identity O->>A: append received, system and version, purpose A-->>O: entry hash O->>P: which scopes may this user read P-->>O: department claims plus shared O->>R: search with scopes R-->>O: top 6 passages with document ids and versions Note over O: Context builder puts the stable prefix first, then passages O->>M: prompt, 6.9 s left, timeout 5 s M->>L: model request over the private path L-->>M: draft answer with citations M-->>O: draft answer plus token counts Note over O: Output check - every citation was retrieved, links and images removed O->>A: append evidence ids, output hash, outcome answered A-->>O: entry hash O-->>G: answer with citations and disclosure line G-->>C: answer

How to read this: time runs downward, the notes mark decisions the Orchestrator makes, and the answer leaves only after the second audit acknowledgement.

Figure 7. Failure paths: the model does not answer in time, and the audit store is down

sequenceDiagram accTitle: Failure paths in Design A accDescr: The Orchestrator calls the Model gateway with a 5 second timeout. If the model answers in time the normal path continues. If it does not, the Model gateway reports a timeout, the Orchestrator computes that 1.2 seconds remain after the reserve, which is too short for a retry, and returns the retrieved passages with citations and no generated text. The outcome is appended to the Audit store first. If the Audit store does not acknowledge, no answer is released. autonumber participant C as Client app participant O as Orchestrator participant M as Model gateway participant L as LLM provider participant A as Audit store O->>M: generate, 6.9 s left, timeout 5 s M->>L: model request alt Model answers in time L-->>M: draft answer M-->>O: draft answer, normal path continues else No reply within 5 s M-->>O: timeout, the breaker counts one failure Note over O: 1.9 s left minus 0.7 s reserved leaves 1.2 s, too short for a retry O->>A: append outcome degraded passages only alt Audit store acknowledges A-->>O: entry hash O-->>C: passages with citations, no generated text, a banner explains else Audit store unavailable O-->>C: no answer released, use the manual process end end

How to read this: the outer alt is the model failure, the inner alt is the audit failure, and both end in an honest message rather than a hang or a silent answer.

A retry is allowed only if the time left holds a typical attempt, so the design skips it and takes a rung of the fallback ladder of page 10. Version 1 has no second in-region deployment, so the rung is passages only.

Check yourself: in Figure 7 the model timed out after 5 s. How many seconds could a retry run, and why is it not attempted?

1.2 s (8 minus 1.1, minus 5, minus the 0.7 s reserve). A typical model attempt takes longer than that, so a retry cannot finish inside the deadline and the Orchestrator falls back to passages only.

Now the action path. Step through a letter that needs approval, first as designed and then without the controls.

One letter_send proposal: as designed and without the controls

As designed

The model proposes. The reply asks for letter_send. Nothing has run: it is a request to check.
Validate. The schema is checked first. The recipient and address come from the case record, never from a value the model wrote.
Policy decides. The letter is external and irreversible, so the Policy engine answers: approval required.
Bind the approval. The request is bound to the hash of the exact letter and recipient and expires after 60 minutes. The approver sees both.
Execute once. After approval the Tool server is called with an idempotency key derived from the call hash. One retry, same key, one owner.
Append, then release. The approval, key and result are appended to the Audit store before the outcome is shown. No entry means no release.

Without the controls

The model proposes. The reply asks for letter_send, and the system treats the reply as a command.
The model names the recipient. The recipient is a tool argument written by the model, so a hostile paragraph in a case note can choose it.
No enforced approval. A prompt rule says to ask first, but no code checks that an approval exists or which call it covered.
Timeout and retry. The call times out. The SDK retries 3 times under a dispatcher that also retries 3 times, each with a fresh key: up to 9 letters.
No record. The audit write is asynchronous and a crash loses it. Nobody can say who approved what.

Figure 8. Life of one approval request

stateDiagram-v2 accTitle: Approval request life cycle accDescr: A proposal is validated and either rejected or sent to the Approval service, where it is pending. A pending request is approved, declined, expired after 60 minutes or cancelled by the kill switch. An approved request is executed with an idempotency key and ends confirmed or unconfirmed after the single retry. Every path ends in a terminal outcome with an audit entry. [*] --> Proposed: model requests letter_send Proposed --> Rejected: validation or policy denies Proposed --> Pending: approval required, hash bound Pending --> Approved: approver approves the exact call Pending --> Declined: approver declines Pending --> Expired: 60 minutes pass Pending --> Cancelled: kill switch Approved --> Executing: call with idempotency key Executing --> Confirmed: result returned Executing --> Unconfirmed: timeout twice Rejected --> [*] Declined --> [*] Expired --> [*] Cancelled --> [*] Confirmed --> [*] Unconfirmed --> [*]

How to read this: each box is a state of one request, each arrow carries its trigger, and the six terminal outcomes each write one audit entry.

Trust boundaries and deployment

Figure 9. Trust-boundary view of Design A

How to read this: arrows whose label starts with T1 to T6 cross a trust boundary and name the control at that crossing, and plain labels stay inside a zone.

Crossing	What to ask	Control
T1 Client app to API gateway	Spoofing and tampering from a device	TLS and session token
T2 API gateway to Orchestrator	Is identity a verified claim or a header?	Verified identity, rate and size limits
T3 Retriever to the data	Could one department's text rank for another?	Filter inside the search, from the verified identity
T4 Model gateway and provider	What leaves, and what comes back?	Prompt text only; the reply is untrusted data
T5 Tool server to the case system	Does the credential allow more than it needs?	Credential scoped to the user, the tool and the call
T6 Approval to a person	Does the approver see what will run?	The exact call and recipient, bound by hash

How to read this: each row is a T arrow from Figure 9, with the reviewer's question and the control (page 3 and page 12).

Figure 10. Deployment view of Design A in one region

How to read this: shaded boxes are network zones, a stacked box shows the instance count, and the only paths out of the application subnet are the two labelled egress rules.

Egress rules attach to exactly two containers: the Model gateway and the Tool server (page 3). Everything else has no route out.
The Model gateway reaches the model endpoint with workload identity, not a stored key. A customer-managed key protects stored artifacts, not a prompt in transit (page 14). The Tool server's credential for the Case system is scoped per call (Figure 9, T5).
What "same region" covers, and how routing is verified, is whatever the provider documents for your service and date (page 14). On the provider pages that page 14 read, content logging is opt-in, so the audit store is the record.

Check the latency budget with the widget. Set the model stage to 7 s and see how far over the deadline the total goes.

Latency budget for one answer in Design A (p95 target 8 s)

Gateway and identity	0.1 s
Input checks	0.1 s
Retrieval	0.8 s
Entitlement lookup	0.1 s
Model call timeout	5 s
Output checks	0.4 s
Two audit appends	0.2 s
Render	0.1 s
Total	6.8 s of 8 s
Slack	1.2 s

The stages total 6.8 s with 1.2 s of slack. At 7 s for the model the total is 8.8 s, 0.8 s over, and even at 6 s no retry could fit, so the timeout stays at 5 s (page 10).

Data model

Figure 11. Audit and decision data of Design A

erDiagram accTitle: Audit and decision data model of Design A accDescr: A system version serves many requests. Each request has retrieved passages, action proposals and audit entries. An action proposal may have one approval. Each audit entry stores a hash of the entry before it. SYSTEM_VERSION ||--o{ REQUEST : serves REQUEST ||--o{ RETRIEVED_PASSAGE : "used as evidence" REQUEST ||--o{ ACTION_PROPOSAL : proposes ACTION_PROPOSAL ||--o| APPROVAL : "decided by" REQUEST ||--o{ AUDIT_ENTRY : records SYSTEM_VERSION { string version_id PK string model_id string prompt_version string parameters string toolset string retrieval_config string index_version } REQUEST { string request_id PK string version_id FK string user_ref string department datetime received_at string outcome string disclosure_version } RETRIEVED_PASSAGE { string request_id FK string doc_id string doc_version int rank } ACTION_PROPOSAL { string call_hash PK string request_id FK string tool_name string idempotency_key string status } APPROVAL { string approval_id PK string call_hash FK string approver_ref string decision datetime expires_at } AUDIT_ENTRY { string entry_id PK string request_id FK string event_type string payload_hash string prev_hash string entry_hash }

How to read this: each box is a table, each line says one or many, and the hash fields on AUDIT_ENTRY make an edit to old history visible.

Design choice	Why
Passages are stored as `doc_id` plus `doc_version`, not text	"Which answers cited the superseded policy?" becomes a query (page 13, lineage)
The request carries the version id, which pins prompt, model, parameters, tools, retrieval and index	A bad answer names its release, and a rollback is a pointer move (page 9)
Audit entries hold ids and hashes, not conversation text	Retention and deletion duties are met in different stores (page 13)

How to read this: each row is a decision visible in Figure 11, with the page that teaches it.

Decisions

ADR	Decision and options	Chosen	Consequence and review trigger	Page
A1	Build, buy or configure. Options: configure the existing search, buy a vendor assistant, build on a model API	Build on a managed model API behind the Model gateway, once the trial of the first two has been recorded against the same release set	A trial that passes R4 and R6 first wins: buy it. Revisit when a product enforces approval bound to the exact call	2
A2	Where the model runs. Options: public endpoint, private endpoint in the same region, self-hosted	Private endpoint, a single-region call with no cross-region routing where the provider documents one, workload identity instead of a stored key, reached only through the Model gateway	Residency (R6) is a fixed constraint, so the public endpoint is excluded first. Choose the narrowest routing level the rule allows and prove it from the provider's audit log. Revisit if volume makes self-hosting break even	4, 14
A3	Retrieval. Options: exact scan with a filter, approximate index with a post-filter, approximate index with a filter inside the scan	Hybrid retrieval (lexical plus vector fused with RRF), exact scan with the entitlement filter, no reranker in version 1	240,000 chunks fit an exact scan, so recall is perfect. Revisit when p95 retrieval passes 0.8 s, then use a pre-filtered approximate index. Add a reranker when recall at 50 is good and recall at 5 is poor	5
A4	Autonomy. Options: free agent loop, workflow with a human decision, answers only	Workflow, 3 model calls at most, letters approved by a person, tasks by policy with a 5% sample	Fewer failure modes and a lower cost than a loop. Revisit when users need multi-step actions the workflow cannot express	6
A5	Audit. Options: provider logs only, asynchronous audit events, append before release	Append before release, fail closed, ids and hashes in a write-once store	Answers now depend on the audit store: it is replicated in region and the manual process is the fallback. Revisit if reconciliation shows the answer path is low risk	13
A6	Governance. Options: classify the model, classify the use	Register the intended purpose, record the classification with a counsel note, add the Article 50 line	Any change of purpose returns to Classified. Revisit when the Annex text or its dates change	13

How to read this: each row is one decision record in table form, and the last two columns say what it costs and when to revisit it (page 2 and its decision log).

Evaluation and release

Gate	What it measures	Number	Page
Proof gate	Quality on the release set, with a one-sided 95% Wilson lower bound	370 of 400 clears 0.90 (lower bound 0.9004)	8
Slices	Pass rate and retrieval recall at k per department, with n and an interval; reported, not gated alone	Claims 90, Underwriting 70, Customer service 60, Legal and compliance 50, Finance 40, Shared 40, plus 50 abstention items	8
Judge	Calibration against human labels on a fresh sample	Kappa at least 0.67 and failure recall at least 80% on 200 items, re-checked monthly and after any change	8
Access and actions	The 300-case suite from R4	300 of 300 blocked, lower bound 99.1%	12
Action quality	End state graded, policy steps checked, pass^k	With a 0.97 success per trial, pass^3 = 0.9127	8

How to read this: each gate names its metric and its number, and the last column is the page that teaches the method.

Try the proof gate. The widget starts at 366 of 400 (91.5 percent). Set passes to 370 and watch the lower bound cross 90 percent.

Proof gate for Design A: 400 items, bar 90 percent, one-sided 95 percent

Observed pass rate	91.5%
Interval (z = 1.645)	88.9% to 93.5%
Gate: lower bound at least 90.0%	fail

A pass rate of 91.5 percent looks fine and is not demonstrated: the lower bound is 88.9 percent. A system whose true rate is 94 percent passes this gate 91.1 percent of the time, and one at 92 percent only 40.0 percent (page 8). Slices are weaker: a 90-item slice detects only a drop of about 7.1 points, a 40-item slice about 11.2.

Check yourself: why is the Finance slice, with 40 items, reported but not used as a gate on its own?

At a 94% baseline, 40 items detect only a drop of about 11 points, so a gate on that slice would miss all but a collapse. It is reported with n and an interval, and a change that targets Finance waits until the slice has more items (page 8). Otherwise the whole-set gate and the isolation suite carry the decision.

Release ladder. After the proof gate the first release climbs the ladder of page 9, without the A/B rung (no user-outcome experiment is planned). Later changes use the regression rule, one canary at a time.

Stage	Who sees the output	What it decides
Offline gate	Nobody	The proof gate and the 300-case suite pass
Shadow	Nobody	Latency, cost and format errors on real questions, with `task_create` and `letter_send` stubbed
Canary	5% of questions: 480 a day	35% judged: 168 a day. A drop from 94% to 90% needs 284 items (258.5 plus the correction of 25), so about 1.7 days
Full	Everyone	Monitoring, sampled judging and the monthly audit reconciliation

How to read this: each rung buys a different kind of evidence, and the canary row shows how traffic share sets the days to a verdict (page 9).

The canary rule: promote with 264 or more passes of 284 (the lower bound clears 0.90), block with 259 or fewer (page 9's regression rule, a one-sided test against the 94% baseline), hold from 260 to 263. An exact binomial check gives a 3.7% false-block rate at a true 94% and a 77.7% chance of catching a true 90%. A 4-point drop exposes 19.2 extra bad answers a day on the canary, against 384 after a full rollout.

Cost envelope

Cost is in placeholder cost units (CU); the widget never carries a price. Start from one answer: 7,200 input tokens, of which the stable prefix of 3,000 is cached, and 400 output tokens.

Cost per request (units)	0.006100
Cost per 1,000 requests (units)	6.1

One model call costs 0.0061 CU, or 6.1 CU per 1,000. The per-call number is not the budget. Page 11 says to budget cost per successful task, so every term goes in.

Term	CU per question	Share
Model: 1.08 calls (1 plus 6% second call plus 2% repair)	0.006588	50.8%
Retrieval	0.002000	15.4%
Tool calls	0.000030	0.2%
Retries: 3% of model and retrieval	0.000258	2.0%
People: 0.01725 reviews per question at 0.15 CU	0.002588	20.0%
Infrastructure	0.001500	11.6%
All-in per question	0.012963	100%
Per successful answer at 94%	0.013791	2.26 times one model call

How to read this: the first three terms are what a provider bills, the rest are the team's, and the last row divides by the share of answers accepted.

The base case is within the 0.016 CU requirement with 13.8 percent headroom. The model is the largest term, so the first levers are a stable prefix, a tight output format and intent rules (page 11). A cascade is not used: escalated answers wait for two models, and the 8 s deadline has no room for that (page 11).

Scenario	Users x questions a day	Questions a month	Calls per question	All-in CU a month
Low	600 x 6	79,200	1.05	1,012
Base	1,200 x 8	211,200	1.08	2,738
High	2,400 x 12	633,600	1.25	8,890

How to read this: each row is one coherent story. The high case is 8.8 times the low case, and it passes the 4,000 CU hard cap, so the design defines what happens then.

At the soft limit of 3,200 CU the Model gateway degrades: a shorter maximum output, no repair attempt. At the hard cap it rejects with a defined message and the manual process. People cost is real: 144 letter approvals and a 5% sample of 432 tasks add up to 2.76 hours a day, 0.46 of one approver (page 13).

Failure table

Failure	What the system does	Page
Retrieval returns nothing, or times out at its 0.8 s budget	No model call; the answer says nothing was found and offers a task; audit outcome `no_evidence`	3
The model times out	Passages only, labelled degraded; the breaker counts a failure	10
The audit store is unavailable	Nothing is released and the user is told to use the manual process	13
The approver does not respond	The request expires after 60 minutes and the model gets a tool error	6
The tool call times out	One retry by the dispatcher with the same key; then `unconfirmed` and an honest message	10
A policy is superseded after answers were issued	The impact query lists answers that cited the old version, for re-review	13
A permission is revoked	The fast lane updates filter metadata first; revocation lag is measured separately (R7)	5
The budget is exhausted	Degrade at the soft limit, reject at the hard cap with a defined message	11

How to read this: each row is a failure with its defined behaviour, so a reviewer can look for a row that is missing.

Threat model

Entry point	Example, illustrative and non-functional	Controls	Page
E1 Direct injection	A user asks the copilot to ignore its rules and send a letter	The reply is a request; the Policy engine decides from identity and arguments; approval for letters	12
E2 Retrieved or attached text	A case attachment contains text addressed to the assistant	Content marked as data; letters always need approval; recipient from the case record	12
E3 Cross-department access	A query phrased to pull another department's text	Filter inside the search from verified identity; the 300-case suite	5
E4 Output channels	Links or images in an answer fetch data on render	Links and images stripped; no remote fetch in the Client app	12
E5 Logs and traces	Prompts hold personal data	Redact before storage; ids and hashes in the audit store	12

How to read this: each row is an entry point where text the team did not write reaches something that acts, with the structural controls that hold even if the model is steered.

Design B: A high-volume event-driven agent platform

Composite scenario, not a real company. An online marketplace publishes order-exception events: late shipments, failed payments, address mismatches, listing flags. Peak is 3,000 events per second and the average 1,200. Triage agents classify each exception and trigger a fixed set of actions: reship, hold a payout adjustment, notify, open a carrier claim. Most events are routine, so the platform exists to keep the model off the hot path.

Each event class has a priority: P1 classes are time-critical and have a safe default action, P2 and P3 classes can wait. About 3 percent of events lead to an action, run as a three-step saga.

Requirements

Kind	Requirement (illustrative values)	Check
R1 Throughput and lag	Sustains 3,000 events per second at peak. Per-partition lag alerts when the oldest unprocessed event is older than 10 s	Lag per partition, not the total
R2 Latency	p99 from event publish to decision event at most 30 s for model-lane events	Trace store, by lane
R3 Quality	At least 93% correct on a 3,000-event golden set; the proof gate needs 2,813 passes (one-sided 95% lower bound at least 0.93)	Eval service in CI
R4 Exactly-once effects	Zero duplicate side effects in a replay test of 10,000 redelivered events (a true rate up to about 0.03% at 95% confidence)	Replay test in CI
R5 Safety	The model never writes an action argument. In 300 adversarial events none leads to an action outside its class's allowed set (lower bound 99.1% on that suite)	Red-team suite
R6 Cost	At most 0.30 CU per 1,000 events all-in at the planned funnel. Soft limit 960,000 CU a month, hard cap 1,200,000 CU. Alert when rules coverage falls below 90%	Gateway budget and coverage metric
R7 Recovery	After a 300 s provider outage the backlog drains within 20 minutes	Game day

How to read this: each row has a number and a place it is measured, and R1, R6 and R7 are the volume requirements that Design A does not have.

Context and the funnel

Figure 12. Context view of Design B

How to read this: flowing dashes are events, person icons are people, and the shaded box groups systems the team does not run.

The model is the most expensive and slowest step, so the design puts four cheaper steps in front of it. Each one has a number.

Figure 13. The event funnel at peak, per second

flowchart LR accTitle: Event funnel at peak load in Design B accDescr: 3,000 events per second arrive. The inbox drops 10 percent as duplicates, leaving 2,700. Rules resolve 92 percent of those, 2,484 per second, without a model. 216 per second remain. The exact cache answers 25 percent, 54 per second. 162 per second go to the small model with a scorer, and 25 percent of those, 40.5 per second, escalate to the large model. E["Ingest: 3,000 per second"] --> D["Inbox drops duplicates: 2,700 continue"] D --> R["Rules resolve 92 percent: 2,484 per second, no model"] D --> C["Remaining: 216 per second"] C --> H["Exact cache answers 25 percent: 54 per second"] C --> S["Small model and scorer: 162 per second"] S --> L["Escalate 25 percent to the large model: 40.5 per second"]

How to read this: each box is a stage and the number is the rate that reaches it at peak, so the model lane carries 5.4 percent of the ingest rate.

Stage	Rule behind it	Page
Inbox	Dedupe on the event id before any model spend; delivery is at-least-once	7
Rules	Route by rule where the event type is known: free, deterministic, testable	4
Exact cache	Key includes event signature, seller, model, prompt version and policy version. No semantic cache: an answer depends on the order	4
Cascade	A small model and a calibrated scorer, with escalation to the large model	11

How to read this: the order is cheapest and most reversible first, the same order as the lever ladder.

Check yourself: if rules resolved only 85% of the events that pass the inbox, how many events per second would reach the small model?

303.75 per second: 2,700 x 0.15 = 405 remain, 25% hit the cache, so 405 x 0.75 = 303.75. The planned lane carries 162, so the funnel is the capacity plan.

Containers

Figure 14. Container view of Design B

How to read this: flowing dashes are events, the dashed red arrow is the dead-letter path, and the two Event bus boxes are two topics on one bus, drawn twice so no arrow has to loop back.

The Orchestrator keeps the saga state and writes each event through an outbox row in the same transaction as the state change (page 7).
Only the Triage worker calls the Model gateway and only the Orchestrator calls the Tool server, so each has one place to enforce its rules.
Left out to stay near the node limit: the Inbox store, the Trace store and the Eval service.

Figure 15. Component view of the Triage worker

How to read this: the large box is one container, the model is called only from the Cascade, and the Action selector is the one component that turns a class into an action.

Component	One responsibility	The rule that lives here	Page
Inbox check	Skip events already processed	Event id as the key, kept 14 days, longer than the topic's retention	7
Action selector	Map a class to an action	The model returns a class, never an action or an argument; the scorer adds a score. Arguments come from validated event fields. Unknown class: dead-letter	12
Decision emitter	Publish the decision, then commit the offset	Commit after processing, so a crash repeats work instead of losing it	7

How to read this: each row is a box in Figure 15 that carries a rule from the course.

Runtime: the normal path and two failure paths

Figure 16. One late-shipment event, from the bus to a saga

sequenceDiagram accTitle: Normal event path in Design B accDescr: Producers publish an order event keyed by order id. The Triage worker records the event id in the Inbox store, finds no rule and no cache entry, classifies it through the Model gateway with the small model, maps the class to an action with the Action selector, publishes a decision event and commits the offset. The Orchestrator saves a saga and an outbox row in one transaction and runs the three saga steps, each with an idempotency key. autonumber participant P as Producers participant E as Event bus participant W as Triage worker participant I as Inbox store participant G as Model gateway participant L as LLM provider participant O as Orchestrator participant T as Tool server P-)E: order.delayed, key order-42, event id e1 E-)W: deliver e1 W->>I: insert e1 if absent I-->>W: new Note over W: No rule matches and the exact cache misses, so the cascade runs W->>G: classify, deadline stamp 30 s G->>L: small model, timeout 3 s L-->>G: class late_shipment Note over G: Scorer gives 0.93, above the threshold, so no escalation G-->>W: class and score Note over W: Action selector maps late_shipment to reship, arguments from event fields W-)E: decision event, key order-42, id e1-d W->>E: commit offset after the emit E-)O: deliver decision O->>O: save saga and outbox row in one transaction O->>T: steps 1 to 3 in order, each with its own key T-->>O: all confirmed O-)E: outbox relay publishes saga.completed

How to read this: time runs downward, open-headed arrows are asynchronous events, and the offset is committed only after the decision event has been sent.

Figure 17. Failure path 1: step 3 of the saga fails and the compensations run

sequenceDiagram accTitle: Saga failure with compensations in Design B accDescr: The Orchestrator reserves replacement stock at the Inventory tool, holds a payout adjustment at the Ledger tool and tries to create the shipment at the Carrier tool, which fails after the Tool server's retries. The Orchestrator decides to compensate in reverse order, releasing the hold and then the reservation, each with its own key, and publishes saga.aborted through the outbox. A compensation that keeps failing is retried and then escalated to a person. autonumber participant O as Orchestrator participant INV as Inventory tool participant LED as Ledger tool participant CAR as Carrier tool participant E as Event bus O->>INV: T1 reserve replacement stock, key s7-t1 INV-->>O: reservation r1 O->>LED: T2 hold payout adjustment, key s7-t2 LED-->>O: hold h1 O->>CAR: T3 create shipment, key s7-t3 CAR-->>O: error after retries Note over O: Decision - T3 failed, run compensations in reverse order O->>LED: C2 release hold h1, key s7-c2 LED-->>O: released O->>INV: C1 release reservation r1, key s7-c1 INV-->>O: released O-)E: outbox relay publishes saga.aborted Note over O: A compensation that keeps failing is retried until it succeeds, then escalated to a person

How to read this: steps 1 to 6 go forward, the note is the decision, and steps 7 to 10 undo in reverse order. The shipment is last because it cannot be recalled.

Figure 18. Failure path 2: the provider refuses calls at peak

sequenceDiagram accTitle: Degraded mode when the provider is limited in Design B accDescr: The Triage worker asks the Model gateway to classify an event. If the breaker is closed and a token is available, the call goes to the LLM provider and the class returns. If the breaker is open or the bucket is empty, the gateway makes no call and returns an overload status with a wait time. For a class with a safe default the worker applies the default and flags the decision degraded. For a class that can wait the worker parks the event on the delay topic with its deadline stamp and commits the offset. autonumber participant E as Event bus participant W as Triage worker participant G as Model gateway participant L as LLM provider participant D as Delay topic W->>G: classify, deadline stamp 30 s Note over G: One bucket and one breaker per model tier, and the only retry owner alt Breaker closed and a token is available G->>L: model call L-->>G: class G-->>W: class and score else Breaker open or the bucket is empty G-->>W: overloaded, retry after 40 s, no call made alt Class has a safe default Note over W: Apply the default from the class table, flag the decision degraded W-)E: decision event with degraded flag else Class can wait W-)D: park with the deadline stamp Note over W: Commit the offset, the partition keeps moving end end

How to read this: the outer alt is the gateway's decision, the inner alt is the class table's decision, and neither branch lets a worker retry in a loop.

Step through the same limit event with the controls and without them.

A provider limit at peak: with the controls and without them

As designed

The provider returns 429. A burst pushes the large tier past its limit, and calls come back with a retry-after value.
The gateway pauses. The bucket stops sending until the wait has passed. The gateway is the only retry owner and the SDK's own retries are set to 0.
Workers get an overload status. No call is made. Each Triage worker looks at the class of the event it holds.
Defaults and parking. Classes with a safe default take it, flagged degraded. Classes that can wait are parked on the delay topic with their deadline stamp, and the offset is committed.
The breaker probes. After the open period a few trial calls pass. If they succeed the breaker closes.
The delay topic drains. Parked events return at the headroom rate, 42.8 events per second above the arrivals.

Without the controls

429 at the large tier. The same burst, the same limit.
Two layers retry. An SDK with 3 attempts sits under a worker loop with 3 attempts: up to 9 provider calls per event.
Lag grows. Workers sit in retry loops, partitions stop moving and the lag per partition rises.
A worker misses its poll deadline. A consumer that does not poll in time is treated as failed, and a rebalance redelivers its batch.
Duplicates and cost. Redelivered events run again: up to 25 per worker, 800 events and 2,000 worker-seconds if all 32 restart, plus the extra model calls.

Figure 19. Life of one saga

stateDiagram-v2 accTitle: Saga life cycle in Design B accDescr: A saga starts when a decision event is saved with its outbox row. Above the value cap it waits for approval and either continues or expires. Below the cap it runs. Running steps repeat until the last is confirmed and the saga completes, or a step fails after retries and the saga compensates. Compensation ends aborted, or needs a person if a compensation keeps failing. [*] --> Started: decision event, saga record and outbox row in one transaction Started --> AwaitingApproval: order value above the cap Started --> Running: below the cap AwaitingApproval --> Running: approved AwaitingApproval --> Expired: no decision in 30 minutes Running --> Running: step confirmed, next step Running --> Completed: last step confirmed Running --> Compensating: a step fails after retries Compensating --> Aborted: all compensations done Compensating --> NeedsPerson: a compensation keeps failing Completed --> [*] Aborted --> [*] Expired --> [*] NeedsPerson --> [*]

How to read this: each box is a state of one saga, each arrow is a trigger, and every path ends in a terminal outcome that publishes one event through the outbox.

Capacity arithmetic

Question	Calculation	Result
Model-lane arrivals at peak	3,000 x 0.9 x 0.08 x 0.75	162 per second
Mean step time of the lane	1.5 s small attempt with scorer, plus 25% x 4.0 s large	2.5 s
Calls in flight (Little's law)	162 x 2.5	405
Partitions, one event at a time each	512 partitions, 32 instances x 16	Capacity 512 / 2.5 = 204.8 per second, 79.1% utilised
Fast lane	3,000 x 0.02 s = 60 in flight, 96 partitions	Capacity 4,800 per second, 62.5% utilised
Backlog after a 300 s provider outage	162 x 300	48,600 events, 94.9 per partition
Drain time	48,600 / (204.8 - 162)	1,135.5 s, about 18.9 minutes

How to read this: the plan is partitions divided by step time. The 42.8 per second of headroom turns a 5-minute outage into a 19-minute recovery (page 7).

Provider limits (illustrative)	Peak load	Share of the limit	With a 10% retry budget
Small tier: 15,000 requests a minute	9,720 a minute	64.8%	71.3%
Small tier tokens: 18,000,000 a minute at 800 per request	7,776,000 a minute	43.2%, so requests bind first	n/a
Large tier: 4,000 requests a minute	2,430 a minute	60.8%	66.8%
Gateway bucket refill at 80% of the limit	200 per second small, 53.3 per second large	Peak 162 and 40.5 fit	n/a

How to read this: the binding limit is the smaller of the request limit and the token limit divided by tokens per request, and the bucket is sized below it so retries have headroom (page 10).

Check yourself: in Design B the large tier's peak load is 2,430 requests a minute against a limit of 4,000. If the escalation rate doubled to 50%, would the limit still hold?

No. Large-tier calls would double to 4,860 a minute, 121.5% of the limit, so the gateway would refuse part of them and the degraded path would run. The escalation rate has to be watched against its break-even and against the quota.

Latency budget

Stage	Budget (s)
Bus lag at p99 (alert threshold)	10.0
Rules and inbox	0.2
Cache and reference lookups	1.0
Small model with scorer, timeout	3.0
Large model, timeout (25% of events)	6.0
Validate and select action	0.3
Outbox relay and publish	1.0
Total against a 30 s p99 target	21.5, with 8.5 s of slack

How to read this: the largest stage is the queue, not the model, and the worst case still leaves room for one retry.

If the large call times out after 6 s, 9.8 s of the deadline remain and 8.5 s after the 1.3 s reserve, so one retry with a 4 s typical attempt fits. The Model gateway is the only layer that retries (page 10).

Trust boundaries and deployment

Figure 20. Trust-boundary view of Design B

How to read this: arrows labelled T1 to T5 cross a trust boundary and name the control, and the decision event drawn directly from the worker to the Orchestrator travels on the decision topic of the Event bus.

Figure 21. Deployment view of Design B

How to read this: the stacked boxes show instance counts, the 608 partitions are 512 for the model lane plus 96 for the fast lane, and the only paths out are the two egress rules. Figures 14 to 16 show the Triage worker as one pipeline: here the Rules workers run its inbox and rules boxes and publish the rest to the model lane, and the Triage workers run the cache lookup and cascade.

Crossing	What to ask	Control
T1 Producers to the bus	Can a producer flood the platform or send a huge payload?	Schema check, size cap, per-producer rate limit
T2 Model gateway and provider	What leaves, and what comes back?	A short prompt; the reply is a class, parsed against a schema
T3 Tool server to business systems	Does the credential reach more than one order?	A credential issued per request, scoped to one order
T4 Policy to the review team	Does the reviewer see what will run?	The exact case, bound by hash, expiring after 30 minutes
T5 Orchestrator to notifications	Can model text reach a buyer?	Fixed templates filled from validated fields

How to read this: each row is a T arrow from Figure 20 with the control that holds even if the model is steered (page 12).

Data model

Figure 22. Event, decision and saga data of Design B

erDiagram accTitle: Event, decision and saga data model of Design B accDescr: An event inbox row may produce one decision. A decision may start one saga and has usage records. A saga has one or more steps, may emit outbox rows and may need one approval. EVENT_INBOX ||--o| DECISION : "decided as" DECISION ||--o| SAGA : starts DECISION ||--o{ USAGE_RECORD : costs SAGA ||--|{ SAGA_STEP : has SAGA ||--o{ OUTBOX : emits SAGA ||--o| APPROVAL : "may need" EVENT_INBOX { string event_id PK int partition int event_offset datetime received_at string outcome } DECISION { string decision_id PK string event_id FK string class float score string route string prompt_version string model_id boolean degraded } USAGE_RECORD { string usage_id PK string decision_id FK string tier int fresh_input_tokens int cached_input_tokens int output_tokens } SAGA { string saga_id PK string decision_id FK string order_id string state } SAGA_STEP { string saga_id FK int step_no string action string idempotency_key string status int attempts } OUTBOX { string row_id PK string saga_id FK string topic string payload_hash datetime sent_at } APPROVAL { string approval_id PK string saga_id FK string call_hash string decision datetime expires_at }

How to read this: each box is a table and each line says one or many. The outbox row id is also the event id, which makes it the dedupe key and the idempotency key.

Design choice	Why
The event id is the primary key of the inbox	The dedupe check is one insert, and a replay finds the key (page 7)
`route` and `degraded` are stored on every decision	Rules coverage, cache hits and escalation rate become queries, which the cost alarm needs (page 11)
Step status and attempts are stored per saga step	A stuck compensation is visible, and the person it escalates to sees the full state (page 7)

How to read this: each row is a decision visible in Figure 22, with the page that teaches it.

Decisions

ADR	Decision and options	Chosen	Consequence and review trigger	Page
B1	Model path. Options: one large model for every event, rules then cache then cascade, classifier router	Rules first, an exact cache, then a small-first cascade with a calibrated scorer	The model term falls from 5.9 CU to 0.118 CU per 1,000 events. The design now depends on rules coverage: alert below 90%, because the lane saturates	4, 11
B2	Ordering and parallelism. Options: key by seller, by order, none	Key by order id, 512 model-lane and 96 fast-lane partitions, offset committed after processing	Per-order order is kept. The two lanes are separate pools, a bulkhead, so a slow model cannot hold the rules lane. Extra workers beyond the partition count sit idle. Revisit if skew overloads one partition	7
B3	Duplicates. Options: trust the producer, idempotent producer only, inbox plus idempotency keys	Inbox on the event id, kept 14 days, plus a key on every tool call. 7-day topic retention, 3-day longest pause	Retention of the keys must exceed any replay. 1.31 billion keys, about 131 GB at 100 bytes each	7
B4	Multi-step actions. Options: choreography, orchestration, no saga	Orchestrated saga, compensations idempotent and in reverse order, irreversible step last, outbox for every event	About 5.9 percent of sagas abort, so compensation is a normal path. Revisit if a step cannot be compensated	7
B5	Retries and degraded modes. Options: workers retry, SDK retries, one owner	The Model gateway owns retries, SDK retries 0, a bucket and breaker per tier, safe defaults for P1 classes, a delay topic for the rest	A degraded decision is flagged and audited. Product owns the default table and reviews it quarterly	10
B6	Model authority. Options: the model chooses the action, the model chooses a class, rules only	The model returns a class. Code maps the class to an action; arguments come from validated event fields	Actions are bounded by the class table. Revisit when a new event type needs open-ended handling	12
B7	Governance. Options: no registry, registry per platform, registry per use	One registry entry for the platform's intended purpose, with the rules, class table and prompts as releases	A change to the class table or the purpose returns to Classified. Revisit if actions start to evaluate individual people, such as seller reputation	13

How to read this: each row is one decision record in table form, and the third and fourth columns say what was chosen and what it costs (page 2).

Evaluation and release

Gate	What it measures	Number	Page
Proof gate	Classification quality, one-sided 95% Wilson lower bound	2,813 of 3,000 clears 0.93. A true 95% passes 99.9% of the time, a true 94% 72.0%	8
Classes	10 event classes of 300 events each, reported with n	A 300-item class detects a drop of about 3.4 points (150 items would give 5.0). Ten classes tested at 5% each give a 40.1% chance of a false alarm	8
Scorer	Threshold set on labelled data for a target false-accept rate	Escalation rate f tracked weekly against the break-even f* of 88%, with an alert at 80% of it	11
Replay	Kill all workers, redeliver 10,000 events	0 duplicate effects	7
Saga failure injection	Fail each step in turn	End state graded: no hold or reservation left behind	8
Adversarial events	300 events with hostile free text	300 of 300 map to an allowed action, lower bound 99.1%	12

How to read this: each gate names what it measures and the number it must reach, so a reviewer can check that none is a feeling.

Try the canary sample size for a drop from 95 to 91 percent.

Items to detect 95.0% falling to 91.0%	224.5
With continuity correction (n + 1/delta)	249.5

That is 224.5 items, or 249.5 with the continuity correction: 250. Release follows page 9:

Offline gate. The golden set, the replay test and the adversarial events pass.
Shadow. A second consumer group reads a copy of the stream, with tools stubbed. Its decisions are traced and discarded.
Canary. 1% of order ids, hashed, which is 55,987 model-lane events a day. Judging 0.5 percent of them gives 280 items a day, so 250 items take 0.9 days.
Rule. Block with 231 or fewer passes of 250: an exact check gives a 4.7% false-block rate at a true 95% and an 81.0% chance of catching a true 91%.
Exposure. A 4-point drop on the canary keys means about 2,239 wrong classifications a day, against about 224,000 on the whole lane.

Check yourself: why does the canary pick keys by a hash of the order id and not a random 1% of events?

Order is the ordering key. Hashing keeps every event of one order on the same release, so a saga never mixes two versions of the decision logic. A random sample of events would split one order across releases.

Cost envelope

At 3,000 events per second the largest cost term is what reaches the model. Compare the cascade with sending every model-lane event to the large tier. Set the escalation rate to 0.9 and watch the cascade reach the always-large cost.

Design B cascade: small tier costs 0.1 of the large tier, 25 percent escalate

Cascade cost per request	0.35
Always-large cost per request	1
Break-even escalation rate	90.0%

The widget omits the scorer, so it shows 0.35 and a 90% break-even. With a scorer at 0.02 of the large call the cascade costs 0.37 and the break-even is 88%, as in page 11. In CU the small call is 0.00059, the large call 0.0059 and the scorer 0.000118, so the cascade costs 0.002183 CU per model-lane event against 0.0059 for always-large, a 63 percent saving.

Term per 1,000 events	CU	Share
Model: 54 model-lane events at 0.002183	0.1179	47.1%
Tool calls: 27 sagas x 3 steps x 0.0005	0.0405	16.2%
Retries: 5% of model and tool work	0.0079	3.2%
People: 0.027 reviews x 0.15	0.0041	1.6%
Infrastructure	0.0800	32.0%
All-in per 1,000 events	0.2504	100%
Per 1,000 successful events at 97%	0.2581

How to read this: the funnel turns 1,000 ingested events into 54 model-lane events, and the model is still under half the cost.

The naive design, every event to the large model with no dedupe, rules or cache, costs 5.9 CU per 1,000 for the model alone: 50 times the model term here.

Scenario	Events a month	Rules coverage	All-in CU a month
Low (half the traffic)	1.56 billion	92%	389,346
Base (1,200 a second average)	3.11 billion	92%	778,692
High (double)	6.22 billion	92%	1,557,384
High, coverage falls to 85%	6.22 billion	85%	2,231,122

How to read this: each row is one coherent story. The cap is 1,200,000 CU, so the high rows pass it and the design must degrade, not just alert.

Rules coverage is the cost driver and the capacity driver. At 85 percent the model term per 1,000 events becomes 0.2210 CU, 1.875 times the plan, and the model lane receives 303.75 events a second against a capacity of 204.8. At the soft limit of 960,000 CU the Model gateway moves P2 and P3 classes to the delay topic. At the hard cap it stops model calls: P1 classes take their safe default and P2 and P3 stay parked.

People are a cost term too: 0.1% of sagas need an approval above the cap, 2,799 a day, which is 46.7 hours of review and about 8 reviewers (page 13).

Failure table

Failure	What the system does	Page
Provider returns 429 or is down	Bucket pauses, one retry owner with jitter, P1 classes take their default, the rest are parked	10
A Triage worker crashes, or a deploy restarts all 32	Offsets commit after the batch, so at most 25 records per worker replay (32 x 25 = 800 for all), and the inbox skips those already done: up to 2,000 worker-seconds saved	7
A poison event	Retry transient failures, park permanent ones at once in the dead-letter topic, quarantine the order id, alarm on depth	7
A saga step fails	Compensate in reverse with idempotent steps. About 4.7 compensations a second at peak is the normal path	7
Rules coverage drops	Alert below 90%, move P2 and P3 to the delay topic, fix the rules first, add partitions last	11
A slow batch triggers a rebalance	Keep batch size x worst-case step well inside the poll interval: 25 x 9 s = 225 s against the 300 s default	7
The budget is exhausted	Soft limit parks P2 and P3, hard cap stops model calls so P1 takes its default, both are audited	11

How to read this: every row is a failure with a defined behaviour, and the numbers come from the capacity arithmetic above.

Threat model

Entry point	Example, illustrative and non-functional	Controls	Page
E1 Free text in events	A seller note contains text addressed to the triage model	The model returns a class only. Code maps class to action and takes arguments from validated fields. A steered model can still choose a wrong class, so actions stay limited to the class table	12
E2 Event flood	A producer sends millions of events	Per-producer rate limit, token cap per event, budget caps. This is the Unbounded Consumption risk	12
E3 Replay	An operator replays a topic	Idempotent consumers and keys that outlive the replay	7
E4 Cross-seller access	A tool call names another seller's order	The credential is issued per request and scoped to one order	12
E5 Outbound text	Model text reaches a buyer	Fixed templates filled from validated fields	12
E6 Traces and logs	Free text lands in the Trace store	Store ids, class, route and token counts, not event text; redact before storage	12

How to read this: each row is an entry point where text the team did not write reaches something that acts, with a structural control rather than a prompt rule.

Comparing the two designs

Topic	What is the same	What differs, and why
Method	Requirements with numbers, ADR table, view set, gates, cost envelope, failure and threat tables	Design A's binding constraint is authority; Design B's is volume
Unit of work	One request, one trace	A conversation turn with a person waiting (A); an event nobody waits for (B)
Role of the model	Output is a request to check, never a command	Text with citations (A); a class only (B)
Retrieval	Access comes from verified identity, in code	A central Retriever with an entitlement filter (A); no document retrieval, only reference lookups in the state stores (B)
Autonomy	Irreversible steps need a stronger control	Human approval bound to the call (A); a saga with a value cap and compensations (B)
Ordering and duplicates	Idempotency keys on every write	Not an issue for answers (A); the key, inbox and outbox carry the design (B)
Retry ownership	One owner, SDK retries off	The Orchestrator's dispatcher for the tool call (A); the Model gateway for model calls and the Tool server for saga steps (B)
Failure stance	A defined, honest behaviour for every failure	Fail closed on the audit store (A); degrade by class and keep the line moving (B)
Cost lever	Cost per successful task, decomposed first	Stable prefix and output limits (A); the funnel and rules coverage (B)
Evaluation	A gate on a confidence bound, slices with n	400 items and a judge calibration (A); 3,000 events, a replay test and end-state grading (B)
Governance	A registry entry with intended purpose; change returns to Classified	Article 50 line and counsel note (A); class table and rules as releases (B)

How to read this: the middle column is the method that transfers to any AI system, and the right column is where a binding constraint changes the design.

Design-review checklist

The questions below are the short form. The full checklist, with one question per line and a downloadable Markdown file, is on AI Architect Templates. Run it at the design review described on page 2 and record every answer in the ADR.

Page	Reviewer's question	If no, read	Shown in
2	Does every requirement have a number and a measurement point, and does each constraint say fixed or negotiable and who owns it?	Requirements	A and B requirements
3	Is there a failure-path sequence, a state diagram wherever something loops, and every trust crossing named with its control?	Diagramming	Figures 7, 8, 9, 17, 19, 20
4	Are routing and caching decided, with a cache key that includes everything the answer depends on?	Models and routing	B1, Figure 13, Figure 5
5	Is the access filter applied inside the search from the verified identity, with a test that tries to cross?	Retrieval	A3, R4
6	Does every tool have an effect class, a retry class and a permission, and is approval bound to the exact call?	Agents and tools	Design A tool table
7	Is the offset committed after processing, with an idempotent effect keyed by event id and a key lifetime longer than any replay?	Events	B2, B3, B4
8	Does the gate use a confidence bound, state the smallest detectable drop, and report slices with n?	Evaluation	Both evaluation plans
9	Does a release pin prompt, model, parameters, tools and retrieval, with a rollback that is a pointer move and a canary sized in advance?	LLMOps	Figure 11, B7
10	Is there one retry owner, a deadline split across the chain, a fallback with its own quota, and a bulkhead per feature?	Reliability	Figures 7, 18, B2, B5
11	Is cost per successful task decomposed, with a soft limit, a hard cap and a per-task cap?	Cost	Both cost envelopes
12	Does authority ever live in a value the model writes, and is there an egress allow-list and redaction before storage?	Security	Both threat models
13	Is the intended purpose registered and the classification dated, is the audit entry appended before release, and is oversight capacity computed?	Governance	A5, A6, B7
14	Are the private path, identity, key ownership, log contents and region boundary written down from the provider's own documentation, with a date?	Cloud platforms	Figures 10, 21

How to read this: answer each row yes, no or not applicable. A no sends the reviewer to the page that teaches the fix, and the last column shows where this capstone answers it.

Try the checklist on four review comments.

Which page do you open first?

A reviewer asks why Design A filters by department inside the search and not after the top results.

Page 5, access control in retrieval
Page 8, evaluation gates
Page 13, governance

Answer

Page 5, access control in retrieval. Pre-filter versus post-filter is the access-control mechanism of page 5: a post-filter returns too few results and holds forbidden text in memory.

A proposal releases answers even when the audit append fails, to protect availability.

Page 11, cost
Page 13, audit logging
Page 4, model routing

Answer

Page 13, audit logging. Page 13 says to append before release and fail closed, because a decision without a record cannot be reconstructed.

The cascade costs more than sending everything to the large model.

Page 11, cascade break-even
Page 3, diagramming
Page 9, release

Answer

Page 11, cascade break-even. Page 11 gives the break-even escalation rate: above it, every request pays for the small attempt, the scorer and a large call.

Workers retry 3 times under an SDK that retries 3 times.

Page 10, one retry owner
Page 2, ADRs
Page 12, redaction

Answer

Page 10, one retry owner. Page 10: attempts multiply across layers, 3 x 3 = 9, so one layer owns retries and the SDK's are set to 0.

Assumptions

Every number in the two designs is an illustrative assumption or follows from one. Nothing here is a measurement, a real company or a price. All arithmetic was recomputed with Python 3.

Show every assumption and calculation behind the numbers

Design A

Traffic: 3,000 staff, 40% daily active gives 1,200; 8 questions each gives 9,600 a day; 22 working days gives 211,200 a month. The peak hour carries 18%: 1,728 an hour, 0.48 a second.
Actions: 6% of questions propose one, so 576 a day: 25% letters (144) and 75% tasks (432). A letter approval takes 60 s: 2.4 h. A 5% sample of tasks is 21.6 reviews: 0.36 h. Total 2.76 h a day, 0.46 of a 6-hour approver day. Reviews per question: (144 + 21.6) / 9,600 = 0.01725.
Corpus: 6,000 documents x 40 chunks = 240,000 chunks of 1,024 float32 values, 983 MB. Partitions: Claims 79,200, Underwriting 60,000, Customer service 36,000, Legal and compliance 28,800, Finance 21,600, Shared 14,400. Finance plus Shared is 36,000 (15%), so a post-filtered top 40 holds 40 x 0.15 = 6 allowed chunks. Claims plus Shared is 93,600 chunks: 95.8 million multiply-adds per exact scan, 245.8 million for the whole corpus.
Latency: 0.1, 0.1, 0.8, 0.1, 5.0, 0.4, 0.2, 0.1 sum to 6.8 s of 8 s. Before the model 1.1 s, so 6.9 s left; after a 5 s timeout 1.9 s; reserve 0.4 + 0.2 + 0.1 = 0.7 s, leaving 1.2 s. The 5 s stands in for a measured p99.9.
Tokens: 3,000 cached prefix, 3,000 passages (6 x 500), 1,000 history, 200 question: 7,200 input, 400 output. Placeholder rates 1, 4 and 0.1 CU per million tokens: (4,200 + 3,000 x 0.1 + 400 x 4) / 10^6 = 0.0061 CU per call. Cached share 3,000 / 7,200 = 0.4167.
Terms per question: calls 1 + 0.06 + 0.02 = 1.08 gives 0.006588; retrieval 0.002; tools 0.0005 x 0.06 = 0.00003; retries 0.03 x (0.006588 + 0.002) = 0.000258; people 0.01725 x 0.15 = 0.002588; infrastructure 0.0015. Sum 0.012963, and 0.013791 at a 94% success rate (2.26 times the 0.0061 of one call).
Scenarios: low 600 x 6 x 22 = 79,200 questions at 1.05 calls; base 211,200 at 1.08; high 2,400 x 12 x 22 = 633,600 at 1.25. Per question 0.012775, 0.012963 and 0.014031 give 1,012, 2,738 and 8,890 CU. People and infrastructure per question are held constant. Soft limit 80% of the 4,000 CU cap.
Gates: 370 of 400 has a one-sided 95% Wilson lower bound of 0.9004 (z = 1.645); 366 gives 0.8892 to 0.9352. At a true 94% the gate passes 91.1% of the time, at 92% it passes 40.0% (exact binomial). A drop from 94% to 90% (one-sided 5%, power 80%) needs 258.5 items, 283.5 with the correction 1 / 0.04 = 25, so 284. Detectable drop at a 94% baseline, without the continuity correction as on page 8: 3.2 points at 400 items, 7.1 at 90, 11.2 at 40. 300 of 300 blocked has a lower bound of 0.9911. With 0.97 per trial, pass^3 = 0.9127.
Canary: 5% of 9,600 is 480 a day; 35% judged is 168; 284 / 168 = 1.69 days. Exact binomial at n = 284 and a true 94%: block with 259 or fewer (size 3.7%, power 77.7% at a true 90%); promote with 264 or more. A 4-point drop exposes 480 x 0.04 = 19.2 a day, and 9,600 x 0.04 = 384 on full rollout.
Slices 90, 70, 60, 50, 40, 40 plus 50 abstention items total 400.

Design B

Traffic: peak 3,000 events a second, average 1,200 (103,680,000 a day, 3.1104 billion in 30 days).
Funnel at peak: duplicates 10% leaves 2,700; rules resolve 92%: 2,484; 216 remain; exact cache hits 25%: 54; small model 162; escalation 25%: 40.5 to the large model. The lane is 162 / 3,000 = 5.4% of ingest. At 85% coverage 405 remain and 303.75 reach the small model; at 90%, 202.5, which is 98.9% of capacity.
Lanes: small attempt with scorer 1.5 s, large 4.0 s at 25%: mean step 2.5 s. In flight 162 x 2.5 = 405. 512 partitions give 204.8 a second (79.1%), headroom 42.8. Fast lane: 60 in flight, 96 partitions, 4,800 a second (62.5%). Instances 32 x 16 = 512 and 12 x 8 = 96, so 608 partitions. A 300 s outage: 48,600 events, 94.9 per partition, drained in 48,600 / 42.8 = 1,135.5 s.
Limits: small tier 15,000 requests and 18,000,000 tokens a minute at 800 tokens per request (22,500 requests); peak 9,720 is 64.8%, 71.3% with a 10% retry budget. Large tier 4,000 a minute; peak 2,430 is 60.8%, 66.8% with the budget; at 50% escalation 4,860, which is 121.5%. Refill at 80%: 200 and 53.3 a second.
Latency: 10.0 + 0.2 + 1.0 + 3.0 + 6.0 + 0.3 + 1.0 = 21.5 s of 30 s. After the large call times out, 30 - 14.2 - 6 = 9.8 s; minus a 1.3 s reserve (0.3 + 1.0), 8.5 s. The 1.5 s in the capacity table is the typical small attempt, the 3.0 s here its timeout.
Redelivery: 32 instances x 25 records = 800 events, 800 x 2.5 s = 2,000 worker-seconds. Inbox keys: every new event id, 1,200 x 0.9 = 1,080 a second x 86,400 x 14 days = 1,306,368,000 keys, 130.6 GB at 100 bytes. Topic retention 7 days, longest worker pause 3 days, keys kept 14 days.
Sagas: 3% of post-dedupe events, so 81 a second at peak and 32.4 on average. Three steps at 0.98 each: complete 0.941192, abort 0.058808, expected compensations 0.058016 per saga: 4.70 a second at peak, 162,408 a day on average. Approvals: 0.1% of sagas, 0.0324 a second, 2,799 a day, 46.7 h at 60 s, 7.8 reviewers at 6 productive hours.
Costs: small call (300 fresh, 500 cached at 0.1, 60 output at 4) is 590 token equivalents, 0.00059 CU; large 10 times, 0.0059; scorer 0.02 of large, 0.000118. Cascade 0.00059 + 0.000118 + 0.25 x 0.0059 = 0.002183. Per 1,000 events: 100 duplicates, 828 rules, 18 hits, 54 model-lane; model 0.117882, tools 27 x 3 x 0.0005 = 0.0405, retries 0.007919, people 0.027 x 0.15 = 0.00405, infrastructure 0.08: 0.250351; 0.2581 at 97% success. Naive 1,000 x 0.0059 = 5.9. Monthly base 3,110,400 thousand x 0.250351 = 778,692 CU; low and high are 0.5 and 2 times that. At 85% coverage 0.358655 per 1,000, and 2,231,122 CU in the high case. Cap 1,200,000, soft limit 960,000.
Gates: 2,813 of 3,000 has a lower bound of 0.9300; a true 95% passes 99.9% of the time and a true 94% passes 72.0%. Detectable drop at 95%: 3.4 points at 300 items, 5.0 at 150. Ten classes at 5% each: 1 - 0.95^10 = 40.1%. Zero duplicates in 10,000: 1 - 0.05^(1 / 10,000) = 0.03%. Canary: 224.5 items, 249.5 corrected, so 250; 1% of keys is 55,987 model-lane events a day, 0.5% judged is 280 a day, 0.89 days; block with 231 or fewer (size 4.7%, power 81.0% at a true 91%). A 4-point drop: 55,987 x 0.04 = 2,239 a day.
Batch and poll: 25 records x 9 s worst case (3 s small timeout plus 6 s large timeout) = 225 s, against the 300 s default.

Key takeaways

A full design is a set of views tied together by numbers: every requirement has a measurement, and every view answers one reviewer question.
The binding constraint shapes the design: authority in Design A (fail closed, approval bound to the call), volume in Design B (a funnel that keeps the model off the hot path).
The model's output is a request to check. Design A has a person approve the risky action, and Design B lets the model return only a class.
One retry owner, one deadline, a defined degraded mode and a defined honest failure for every part, written down before the incident.
Gates decide on a confidence bound and name the smallest drop they can see: 370 of 400 in Design A, 2,813 of 3,000 in Design B.
Cost is cost per successful task, decomposed. Design B's cost and capacity both depend on rules coverage, so coverage is monitored.
Classification follows the intended purpose, so a change of purpose returns to the registry. Nothing on this page is legal advice.

Self-check

1. Design A's release set shows 366 of 400 acceptable and the requirement is "at least 90%". A manager says 91.5% clears it. What do you answer?

Not demonstrated. The one-sided 95% lower bound is 88.9%, below 0.90, so the data cannot rule out a true rate under the bar. The proof gate needs 370 of 400, and a system whose true rate is 92% passes it only 40.0% of the time, so the choices are a larger set, a better system or a regression gate.

2. A team lead wants Design A to filter by department after retrieval, "to keep recall high". What is wrong with that?

A finance user may see 36,000 of 240,000 chunks (15%), so a post-filtered top 40 holds about 6 allowed chunks, and forbidden chunks were in memory before they were removed. The filter belongs inside the search, built from the verified identity, with a test that tries to cross departments.

3. A letter was approved, the tool call times out and is retried. What stops two letters, and who retries?

The retry reuses the idempotency key derived from the call hash, and the Case system, which performs the effect, returns the first result instead of sending again. If it did not honour the key, the call would end unconfirmed for a person to check. Only the Orchestrator's dispatcher retries, once, with the SDK's retries set to 0. The approval is bound to the hash, so a changed letter or recipient is a different call and needs a new approval.

4. In Design B the rules stop resolving 92% of events and fall to 85% at peak. What happens, and what do you do?

The model lane receives 303.75 events a second against a capacity of 204.8, so lag grows, and the model term per 1,000 events rises by 1.875 times to 0.2210 CU. Alert when coverage falls below 90%, park P2 and P3 classes on the delay topic, fix the rules first and add partitions last, because the large-tier quota would not carry the load anyway.

5. A product owner wants to reuse Design A's copilot to rank customer claims for fraud review. What does the course say you do first?

Treat it as a change of intended purpose: the system returns to Classified before release. Page 13's classification attaches to the use, not the model, so the registry entry, the counsel note and the Annex III reading are redone. The controls of Design A help, but they do not answer the classification question.

Ask about this article

Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

AI Architect (Advanced): Operate, Secure and Govern· Page 7 of 7

PreviousCloud Platforms for AI: Networking, Identity, Keys, Logs Finish · Test your knowledgeTest your knowledge — AI Architect (Advanced) quiz Final step · Practice examTake the AI Architect (Advanced) practice examA longer, scenario-based exam. Every wrong answer tells you which lesson and section to revise.

Scaled AI Weekly

Enjoyed this? Get more like it
every Monday.

Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

Get new articles by email

Free. We email a confirmation link first, and you can unsubscribe from any email.

Join engineers building production AI systems

Comments

Back to all guides

Capstone: Two Full AI System Designs

At a glance

How to read a full design

Design A: A regulated-industry knowledge and actions copilot

Requirements

Classification and governance reasoning

Context and containers

The Orchestrator

Runtime: the normal path and the failure path

Trust boundaries and deployment

Data model

Decisions

Evaluation and release

Cost envelope

Failure table

Threat model

Design B: A high-volume event-driven agent platform

Requirements

Context and the funnel

Containers

Runtime: the normal path and two failure paths

Capacity arithmetic

Latency budget

Trust boundaries and deployment

Data model

Decisions

Evaluation and release

Cost envelope

Failure table

Threat model

Comparing the two designs

Design-review checklist

Assumptions

Key takeaways

Self-check

Further reading

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it
every Monday.

Comments

At a glance

How to read a full design

Design A: A regulated-industry knowledge and actions copilot

Requirements

Classification and governance reasoning

Context and containers

The Orchestrator

Runtime: the normal path and the failure path

Trust boundaries and deployment

Data model

Decisions

Evaluation and release

Cost envelope

Failure table

Threat model

Design B: A high-volume event-driven agent platform

Requirements

Context and the funnel

Containers

Runtime: the normal path and two failure paths

Capacity arithmetic

Latency budget

Trust boundaries and deployment

Data model

Decisions

Evaluation and release

Cost envelope

Failure table

Threat model

Comparing the two designs

Design-review checklist

Assumptions

Key takeaways

Self-check

Further reading

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it every Monday.

Comments

Enjoyed this? Get more like it
every Monday.