AI Architect Mindset: Requirements, ADRs, Design Reviews

Two habits help. First, mark each constraint fixed or negotiable and name who can change it. "No data outside the region" is fixed by a contract; "finish by the end of the quarter" may be negotiable with a smaller scope. Second, write down constraints that are really preferences in disguise: "we prefer open-source" is a preference until a policy makes it a rule. Disguised preferences are the most common reason a design review stalls, because one side treats a preference as a law.

Build vs buy vs configure

There are three answers, not two. Build means writing the capability yourself on lower-level components (a model API, a vector database, your own orchestration). Buy means adopting a product or managed service whose behaviour the vendor controls. Configure means using a capability you already own, such as the search in your current document system or a feature of your current platform, and adjusting it without new software. Configure is the option teams forget, and it is usually the cheapest to test.

Figure 3. Build, buy or configure

How to read this: the order is cheapest to test first, and a failed trial moves you down the tree with evidence instead of opinion.

Criteria worth writing down for each option:

• Fit to the requirements. Run the same evaluation set against each option. A vendor demo is not a test.
• Total cost over the life of the system. Include licences, engineering time, running costs, evaluation and the people who maintain it, over three years rather than three months. A build that looks cheap in month three can be dominated by maintenance later; the contract search case shows the arithmetic.
• Control and portability. How hard is it to leave? What happens if the vendor changes a model or price?
• Time to first value. Configure is usually the quickest to start and build the slowest. If the requirement is time-bound, this can decide the matter.
• Data handling. What leaves your boundary, and who is accountable for it?
• Maintenance load. Who tests after a model update, and who answers the pager?

The arguments pull both ways. Buying spreads maintenance across a vendor's customers; building thin layers around model APIs limits lock-in and keeps control of the data path. Which weighs more depends on your constraints. This page's default is to start with configure, then buy, and to build only what a requirement or constraint forces, but to write the ADR so the decision can be reversed when the evidence changes.

Architecture Decision Records

An architecture decision record captures one significant decision, so a person who joins in two years can see why the system looks the way it does. The adr.github.io site defines an architectural decision as "a justified design choice that addresses a functional or non-functional requirement" that is architecturally significant, and an ADR as something that "captures a single AD and its rationale". The collection of ADRs in a project is its decision log. The site credits Michael Nygard's 2011 post with popularising the idea.

Nygard's post, "Documenting Architecture Decisions" (Cognitect, 15 November 2011), explains the problem: "One of the hardest things to track during the life of a project is the motivation behind certain decisions." A new person on the project has two choices, to blindly accept a decision or blindly change it, and changing it without understanding its motivation or consequences "could mean damaging the project's overall value". Nygard also observes that "Large documents are never kept up to date.", so he proposes small ones. An ADR is a short text file; each record describes a set of forces and a single decision made in response to them. It has five parts:

The post says the whole record should fit on one or two pages, and suggests keeping the files in the project repository (at doc/arch/adr-NNN.md, in a lightweight markup such as Markdown) so they travel with the code.

Figure 4. The life of an ADR

How to read this: the status values are the four from Nygard's post, and nothing is deleted, a superseded record stays so readers know it was once the decision.

The post's rule for reversals is the part teams skip: when a decision is reversed, the old ADR is kept and marked superseded ("It's still relevant to know that it was the decision, but is no longer the decision."). If your team also needs a value for a proposal that was dropped, add one and define it in your template; the post does not prescribe it.

When to write one. Write an ADR when a decision is expensive to reverse, when it constrains later choices, or when someone is likely to ask "why did you do it this way?" For AI systems that includes: choice of model and provider, build vs buy vs configure, retrieval strategy, where humans approve, what is logged, and which data may reach which model. Do not write one for choices that a one-line pull request comment explains.

When to supersede. Write a new ADR, do not edit the old one, when the context has changed in a way that would change the decision: a new model tier changes the quality-cost balance, a provider changes terms, a requirement is tightened. Editing the old record erases the history that makes the log useful. Set a review trigger in the Decision or Consequences ("revisit when the smaller tier reaches the bar on the release set") so supersession is planned.

A worked ADR for a model choice

This is an illustrative ADR for a ticket-summarisation feature. The model tiers are placeholders ("small tier", "large tier"), not real products, because model names and prices are volatile facts. The measurements are invented for the example; the interval method is shown in the Worked example.

ADR 12: Use the large model tier for ticket summaries

Status: Accepted (2026-10-21). Supersedes: none.

Context
Support agents need a three-sentence summary of long ticket threads. The
requirements: at least 92% of summaries rated acceptable on the 300-example
release set (rubric v3); p95 latency at most 4 s at the gateway; tickets may
contain customer personal data, so only providers on the approved list may
be used. About 150,000 summaries a month are expected. Two tiers from the
approved provider were evaluated on the release set:
  small tier: 272 of 300 acceptable (90.7%), p95 1.8 s, relative cost 1
  large tier: 286 of 300 acceptable (95.3%), p95 3.1 s, relative cost 6
With a normal-approximation 95% interval the pass rate of the small tier lies
between about 87.4% and 94.0%, so the data do not prove it is below the bar, but its measured
rate is under it. The large tier's lower bound is about 92.9%.

Decision
We will use the large tier for ticket summaries behind the model gateway,
with the prompt and rubric versions recorded on every request.

Consequences
Positive: the quality requirement is met with margin and latency is within budget.
Negative: the cost per summary is about six times the small tier's.
Neutral: the model gateway lets us switch tiers per request type later.
Review trigger: re-run the release set when the provider releases a new
small tier, or when monthly cost exceeds the approved cap.

Note what the ADR does. It names the requirement it is answering, shows the evidence, admits the uncertainty and states a trigger for revisiting. A reader can disagree with the decision, but cannot say they do not know why it was made.

Design reviews

A design review is a meeting with a defined purpose: decide whether a design is ready to build (or to launch), record the decision and the conditions. It is not a status update, not a demo and not a place to redesign from scratch. The ADR is the input; the decision log is the output.

Who attends. Keep it small enough to decide: the author (owns the design and the ADR); an architect or senior engineer chairing; a product owner (who owns the requirements); a security or privacy reviewer when data sensitivity or tool access is in scope; an operations or reliability engineer; and the person who owns evaluation. A scribe records decisions. Anyone else reads the pre-read and sends comments.

Agenda (60 minutes, illustrative). Five minutes for context and the decision requested. Ten minutes on requirements and constraints, checking each is testable. Twenty minutes walking the diagrams, from context to the runtime sequence to the failure paths. Fifteen minutes on risks and the draft ADRs. Ten minutes to decide, assign actions and set dates.

Pre-read. One to two pages plus diagrams, sent two working days before: the requirements, the constraints, the options considered with the chosen one, the evaluation evidence, and the open questions. A review that spends its first twenty minutes reading is a review that will end undecided.

Figure 5. A design review from request to decision log

How to read this: the architect is the hinge between the author's evidence and the decision log, and every branch ends with a written entry.

Questions an architect asks. These are the ones that expose most gaps:

1. What requirement does this component exist to satisfy, and how will we know it does?
2. What is the worst plausible output, and who is harmed if it reaches a user?
3. What happens when the model provider is slow, down or returns something malformed?
4. What is the cost per request at expected volume, and at ten times that volume?
5. Which data goes to which component, and where does it get logged?
6. What can the system do without a human approving it, and what is the maximum damage of one wrong action?
7. How do we detect that quality has dropped after a model or prompt change?
8. What is the rollback, and how long does it take?
9. What is the simplest non-AI option, and why is it not enough?
10. Which decision here is hardest to reverse, and is it written down?

A good outcome is one of three recorded results: approved (ADRs set to accepted), approved with conditions (each condition has an owner and a date), or not approved (with the specific evidence missing). A review that ends with "looks fine, let's keep going" has not produced a decision. The meeting also fails if the author leaves unsure what to change.

Case files

• Claims triage copilot: requirements nobody wrote down. In a composite scenario, an insurer's pilot looked good in demos but could not be called a success or a failure. Pattern: requirements definition, written as testable statements with a labelled evaluation set.
• Contract search: build, buy or configure. In a composite scenario, a legal team began building and then discovered the three-year cost. Pattern: build vs buy vs configure, compared on total cost with a simple non-AI baseline.

Decision framework

1. If a requirement has no number and no measurement point, then do not start building against it, because the team cannot tell whether it succeeded; revisit when the product owner can state what result would make them say no.
2. If a constraint is marked fixed, then exclude options that break it before comparing anything else, because a better option that violates a contract is not an option; revisit when the owner of the constraint changes it in writing.
3. If an owned capability could meet the requirement, then trial it first, because it is the cheapest to test and keeps data inside the boundary; revisit when the trial misses the bar on the evaluation set.
4. If a decision is expensive to reverse or constrains later choices, then write an ADR before the build starts, because the reasons are easiest to recall now; revisit at the review trigger written in the record.
5. If a design review ends without a recorded outcome, then treat the design as not approved, because unrecorded approvals are remembered differently by each attendee; revisit at the next scheduled review.
6. If the risk of one wrong output is high and hard to detect, then add a human approval step and test it as a requirement, because tolerance is the owner's decision and a filter is not proof; revisit when evidence shows the error rate and impact are both lower than assumed.

Comparison of the main options

Failure scenarios to design for

1. A requirement exists only in conversation. Behaviour: if a requirement is not in the written requirements sheet, it does not count as accepted for release; the design review checks the sheet, not memory.
2. The evaluation set is too small to support the claim. Behaviour: the release report states the pass rate with its confidence interval, and a result whose interval straddles the bar is reported as "not demonstrated", not as a pass.
3. The model or prompt changes after approval. Behaviour: any change to the model, prompt or retrieval re-runs the release set; a drop below the bar blocks promotion and reverts to the previous version.
4. Constraints change mid-project (for example a data-residency rule). Behaviour: the owner raises it, the affected ADRs move to a review state, and a superseding ADR is drafted before the build continues.
5. The review is skipped under deadline pressure. Behaviour: launch without a recorded review outcome is not permitted; a shorter review with the same questions is the minimum.
6. The decision log goes stale. Behaviour: each ADR has an owner and a review trigger; a quarterly pass marks accepted records as still valid, deprecated or superseded.

Worked example

A support-summary feature has the illustrative requirements below (a different feature from the ADR example, so the numbers differ). Five calculations show how to turn requirements into numbers.

1. Error budget. An availability objective of 99.5% over a 30-day window on 400,000 requests allows (1 − 0.995) × 400,000 = 0.005 × 400,000 = 2,000 failed requests in the window. The SRE book describes an error budget as a rate at which the SLOs can be missed, tracked on a daily or weekly basis. A "failure" here is a request that returns no valid summary within the latency limit.

2. Latency budget. The requirement is p95 end-to-end of 6 s. Allocate it across the stages and keep slack.

Figure 6. A latency budget on the request path

How to read this: each stage carries a time budget, and the slow-provider branch shows the behaviour agreed in advance, a fallback rather than a hang.

The stage budgets are 0.2 + 0.8 + 0.2 + 3.8 + 0.4 + 0.2 = 5.6 s, which leaves 6.0 − 5.6 = 0.4 s of slack. Percentiles of separate stages do not add up exactly, so this is a planning budget. Measure the end-to-end percentile in a load test.

3. Cost per request. A monthly budget of 6,000 currency units over 400,000 requests allows 6,000 / 400,000 = 0.015 per request. The token allowance follows from the price: allowed tokens per request = 0.015 / price per token. Get the price per token from the cost estimator for the model and date you use, and include retries and embedding calls in the count.

4. Does a pass rate support a quality bar? With the normal-approximation interval, the 95% bounds of a pass rate are p ± 1.96 × √(p(1 − p) / n). Suppose 180 of 200 summaries pass. Then p = 0.90, the standard error is √(0.9 × 0.1 / 200) = √0.00045 = 0.0212, and 1.96 × 0.0212 = 0.0416. The interval is about 85.8% to 94.2%. A requirement of "at least 90%" is met by the point estimate, but the data cannot rule out a true rate near 86%. The lower bound is always below the observed rate, so to show a lower bound of at least 90% you need an observed rate above 90% and a sample large enough for the margin to fit under it; a bigger sample at exactly 90% only narrows the interval. This normal approximation is simple and is poorer near 0% and 100% or at small samples; use an exact or Wilson interval there.

5. Zero failures is not zero risk. Suppose none of 80 adversarial cases produced a leak. If the true leak rate were p, the chance of seeing zero in 80 independent cases is (1 − p)^80. Setting that equal to 0.05 gives p = 1 − 0.05^(1/80) = 0.0368. So zero hits in 80 cases is consistent, at 95% confidence, with a true rate up to about 3.7%. That is why a risk requirement needs both a test set big enough for its tolerance and controls that do not depend on the test passing.

Assumptions: 400,000 requests per month, a 30-day window, a 6,000 currency-unit budget, 200 and 80 as example sample sizes, all stage times and the 99.5% objective chosen for illustration, independent cases, and a 95% confidence level with z = 1.96.

Requirements as an executable acceptance check

Requirements that live in a document drift. Put the thresholds in code that runs against the evaluation results, so a change that breaks one fails the build. The sketch below checks quality (with the lower bound from the worked example), p95 latency (nearest-rank) and mean cost against the thresholds. It needs only the standard library.

import math

REQ = {"quality_lower_bound": 0.85, "latency_p95_s": 5.0, "cost_mean": 0.20}

def percentile(values, q):
    ordered = sorted(values)
    return ordered[math.ceil(q * len(ordered)) - 1]  # nearest rank

def lower_bound(passes, n, z=1.96):
    p = passes / n
    return p - z * math.sqrt(p * (1 - p) / n)

def check(results):
    """results: list of dicts with ok (bool), latency_s (float), cost (float)."""
    n = len(results)
    got = {
        "quality_lower_bound": lower_bound(sum(r["ok"] for r in results), n),
        "latency_p95_s": percentile([r["latency_s"] for r in results], 0.95),
        "cost_mean": sum(r["cost"] for r in results) / n,
    }
    got["pass"] = (got["quality_lower_bound"] >= REQ["quality_lower_bound"]
                   and got["latency_p95_s"] <= REQ["latency_p95_s"]
                   and got["cost_mean"] <= REQ["cost_mean"])
    return got

With 400 results where 353 pass, latencies evenly spaced from 1.0 to 4.9 s and a cost of 0.12 each, check reports a lower bound of about 0.851, a p95 of 4.7 s and a pass. At 352 passes the lower bound falls to about 0.848 and the check fails: the point estimate (88%) looks fine and the requirement still is not demonstrated, which is the behaviour you want a gate to have. Wire the function into CI so a prompt or model change cannot merge without the numbers.

Common mistakes

• Writing requirements as adjectives. "Fast", "accurate" and "safe" cannot fail a test. Replace each with an indicator, a threshold and a measurement point.
• Setting the target from the prototype's result. The SRE book warns against picking a target based on current performance. The target comes from the user's need; the prototype tells you how far away you are.
• Treating "buy" as the end of design work. A product still needs requirements, an evaluation on your data, a data-flow review and an exit plan.
• Editing an old ADR instead of superseding it. The edit erases why the earlier choice was reasonable, which is the information the next team needs.
• Using the design review as a demo. A demo shows the happy path. A review checks the failure paths, the evidence and the rollback.
• Hiding preferences as constraints. "We cannot use that vendor" needs a named owner and a reason, or it is a preference and should be weighed as one.

Try it

Time: 20 to 40 minutes. Pick one AI feature you work on or have been asked to scope (or use "a chatbot that answers HR policy questions" if you have none).

1. Write one requirement for each of the five kinds (quality bar, latency, cost, risk, data sensitivity) in testable form: a threshold, a measurement point and a check.
2. List three constraints and mark each fixed or negotiable, with the owner.
3. Sketch the build, buy and configure options in one table row each, naming one simple non-AI option.
4. Write a one-page ADR for the choice you would make, using the five sections, with a status and a review trigger.
5. Draft five questions you would ask in a review of your own design.

Done when each requirement names a number and a place where it is measured, the ADR states its status and a review trigger, and a colleague who was not involved can say what result would make you reverse the decision.

Key takeaways

• A requirement is useful only if a test can fail it: state the indicator, the threshold, the window and where it is measured.
• Use the SLI and SLO vocabulary from the SRE book for latency and cost, and extend it to quality with a rubric and a fixed evaluation set.
• Risk is probability combined with consequence, and the tolerance is an owner's documented decision, not something the model or the framework supplies.
• Separate constraints from preferences, mark which are fixed, and name who can change them.
• Evaluate configure, then buy, then build, on the same evaluation set and over a three-year cost.
• Write an ADR for decisions that are costly to reverse; supersede rather than edit; set a review trigger.
• A design review is complete only when it records one of three outcomes with owners and dates.

Self-check

Further reading

• Michael Nygard, Documenting Architecture Decisions, Cognitect, 15 November 2011.
• Architectural Decision Records, the community site that defines the terms used in this lesson.
• Google, Site Reliability Engineering, chapter "Service Level Objectives".
• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023; see also the NIST AI RMF page for the Generative AI Profile and the current revision status. This is a voluntary framework and not legal advice.

Last verified: 2026-10-21

Next

Continue to page 3: Diagramming AI systems, which teaches how to draw the context, component and runtime views that a design review needs.