Skip to content
    All guides

    Forward Deployed Engineer

    Latency, Capacity and Availability Targets for AI Design

    Turn fast and always on into numbers: a latency budget per hop, peak concurrency from Little's law, downtime per month, and rules for degraded service.

    By Published 9 min read

    Share

    This lesson is part of AI Solution Architect: Design Method and Patterns. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: how to turn "it must be fast and always on" into a latency budget per hop, a peak concurrency figure, an availability target in downtime minutes, and a rule for when a dependency is down.

    At a glance

    Figure 1. The request path with a budget on each hop (illustrative seconds, 95th percentile)

    flowchart LR accTitle: Request path with a latency budget per hop accDescr: A request goes from the client app to the API gateway with a budget of 0.2 seconds. The gateway fans out to the retriever at 0.5 seconds and the order lookup tool at 0.8 seconds in parallel. Both feed the model call at 4.0 seconds. The guardrail takes 0.3 seconds and the reply 0.2 seconds. The total is 5.5 seconds against a target of 6.0. A["Client app"] --> B["API gateway: 0.2 s"] B --> C["Retriever: 0.5 s"] B --> D["Tool server, order lookup: 0.8 s"] C --> E["Model gateway, model call: 4.0 s"] D --> E E --> F["Policy engine, guardrail: 0.3 s"] F --> G["Reply: 0.2 s"]

    How to read this: the two reads run in parallel, so only the slower one counts; 0.2 + 0.8 + 4.0 + 0.3 + 0.2 = 5.5 s, leaving 0.5 s of a 6.0 s target as reserve.

    • A target is a number, a percentile and a measurement point.
    • Give every hop a budget share and a timeout.
    • Little's law gives average concurrency; the ceiling is the number of users.
    • A higher availability target can cost far more: choose the lowest the customer can live with.
    • Write down what the assistant does when each dependency is down.

    Scenario

    Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, is designing its Order and Policy Assistant for 24 customer-service agents. The Head of Customer Operations says: "It must be fast and always on." The platform team asks what load the order management system (OMS) should expect. All numbers are illustrative.

    What you will be able to do

    • Split a latency target across hops and set a timeout on each.
    • Estimate peak concurrency with Little's law and name the ceiling.
    • Convert an availability target into downtime minutes for a stated window.
    • Write a degradation rule for each dependency.

    Prerequisites

    Integrating with the customer's order and policy systems defines the hops; AI Architect: reliability and scaling covers timeouts, retries and fallbacks.

    Core ideas

    Latency: one budget, one share per hop

    Figure 2. Latency budget sheet (Lumen values are illustrative allocations, not measurements)

    Hop Budget p95 (s) Timeout (s) If slow or failing Measured p95
    API gateway 0.2 1.0 Error to the agent ____
    Retriever (policy search) 0.5 1.0 Skip the policy part, say so ____
    Tool server (order lookup) 0.8 1.2 Skip the order part, say so ____
    Model call 4.0 5.0 Show cited passages only ____
    Guardrail 0.3 0.5 Block the reply ____
    Reply 0.2 0.5 Error to the agent ____
    End to end 6.0 6.5 Honest error ____

    How to read this: fill the last column with your own measurements; the model row is likeliest to be wrong, so measure it in Run your service against a real model.

    Predict first. At month-end the order system is slow and the lookup takes 3.5 s instead of 0.8 s. Ignoring timeouts for now, which hop breaks the 6.0 s target, and by how much?

    Answer

    The order lookup. It is now the slower parallel read: 0.2 + 3.5 + 4.0 + 0.3 + 0.2 = 8.2 s, which is 2.2 s over. The model call did not change.

    Percentiles of separate hops do not add, so the sum is a plan: measure the end-to-end p95 too. Google's SRE book warns that a 100 ms average can hide "1% of requests" taking "5 seconds".

    Capacity: concurrency from usage, then the ceiling

    Little's law: the average number in a system, L, equals the average arrival rate, λ, times the average time in the system, W. Little's 2011 retrospective states L = λW and proves a finite-interval form.

    Figure 3. Sizing the Lumen assistant (all inputs illustrative)

    Quantity Formula Value
    Peak-hour requests 6,720 / 21 days = 320 a day, x 20 percent, x 3 requests 192
    Arrival rate λ 192 / 3,600 s 0.0533 per second
    Average in flight L λ x 5.5 s 0.29
    If calls hang to 30 s (no timeout) λ x 30 s 1.6
    Hard ceiling 24 agents x 1 request each 24
    Ceiling rate 24 / 5.5 s 4.36 per second

    How to read this: the average is tiny, so the real questions are the ceiling and the limits of the systems you call.

    Do not size a limit from the average 0.29. With one open request per agent, 24 is the most in flight: ask the platform team whether the gateway and the OMS accept 24 concurrent calls and about 4.4 a second. Read model provider limits from your own account.

    Availability: a target, a window and a downtime figure

    The SRE book defines a service level indicator (SLI) as "a carefully defined quantitative measure" of service level and a service level objective (SLO) as "a target value or range of values" measured by an SLI. It advises avoiding absolutes such as "always" available.

    Figure 4. Downtime allowed by an availability target

    Target 30-day month (43,200 min) Staffed hours (168 h a month)
    99 percent 432 min (7.2 h) 100.8 min
    99.5 percent 216 min (3.6 h) 50.4 min
    99.9 percent 43.2 min 10.08 min
    99.95 percent 21.6 min 5.04 min

    How to read this: each cell is the window times one minus the target, for example 43,200 x 0.005 = 216, so the window changes the answer by a factor of 4.3.

    The SRE book says cost "does not increase linearly as reliability increments": an increment "may cost 100x more than the previous increment". The unreliability a target allows is the error budget: at 99.5 percent over staffed hours, 50.4 minutes a month.

    NFR STATEMENT
SLI:      share of requests that return a cited or honest degraded reply within 6.0 s
Window:   staffed hours, 168 h a month
SLO:      99.5 percent (50.4 minutes of error budget a month)
Measured: at the API gateway, per request
Degraded: good only if it says it is degraded; its rate is tracked separately

    Degradation: decide before the outage

    Figure 5. What the assistant does when a dependency is down

    flowchart TD accTitle: Degradation rules by failed dependency accDescr: If the model is down, show the cited policy passages and the raw order fields. If retrieval is down, answer order questions only and point to the policy library. If the order system is down, answer policy questions only and say order status is unavailable. If the gateway or identity provider is down, agents use the old process. S{"Which dependency failed?"} S -->|"Model"| A["Show cited passages and order fields, no generated answer"] S -->|"Retriever"| B["Order lookup only, point to the policy library"] S -->|"Order system"| C["Policy answers only, say order status is unavailable"] S -->|"Gateway or SSO"| D["Agents use the old process"]

    How to read this: each branch keeps what works and says what is missing; none guesses.

    Decision exercise. The lookup overshoots by 2.2 s at month-end. Choose one: (A) raise the target to 8.5 s; (B) enforce the 1.2 s order-call timeout from Figure 2 and answer the policy part alone when it fires; (C) drop the guardrail to save 0.3 s.

    Model answer

    Choose B: 0.2 + 1.2 + 4.0 + 0.3 + 0.2 = 5.9 s. Combined questions lose their order part while the OMS is slow; the agent checks it directly. A moves the target without the sponsor. C still totals 7.9 s and removes the check on unsupported answers.

    Worked example

    Assumptions (illustrative): 6,720 assisted contacts a month, 21 working days, 8 staffed hours a day, a peak hour holding 20 percent of a day, 3 requests per contact, the Figure 2 budgets.

    • Chain check: gateway 99.9, OMS 99.9, retrieval 99.9 and model service 99.5 percent, all hard dependencies in series. The product, 0.999 x 0.999 x 0.995 x 0.999, is 0.9920, so 99.20 percent. This is a derivation, not a source claim: independent failures multiply.
    • Closing the gap to 99.5: Figure 5 means an OMS outage costs only the order part of a reply, so the SLI counts a labelled degraded reply as good. Put that in the proposal as an assumption with an owner.

    At the customer: "Will it always be on?" Say: "No system is. We agree 99.5 percent over staffed hours, 50.4 minutes a month, and the assistant says which part is down."

    Practice task

    Produce the targets sheet for Lumen.

    1. Copy Figure 2 and fill the measured column with your own p95 values.
    2. Recompute L at the average peak and at the ceiling.
    3. Write the NFR statement and one degradation row per dependency.

    Check yourself: if your measured sum passes 6.0 s, change a timeout or the scope before the target.

    Decide first, then reveal.

    Targets and degradation calls (interactive when JavaScript is on)

    The sponsor asks for 99.9 percent. How much downtime is that in a 30-day calendar month?

    • 7.2 hours
    • 43.2 minutes
    • 21.6 minutes
    Answer

    43.2 minutes. 43,200 minutes x 0.001 = 43.2. 7.2 hours is the 99 percent figure.

    The order system is down and an agent asks about the 30-day return rule. What should the assistant do?

    • Refuse everything until the OMS is back
    • Show the last cached order status
    • Answer from the policy library with its citation and say order status is unavailable
    Answer

    Answer from the policy library with its citation and say order status is unavailable. A policy answer does not need the OMS, so the rule keeps what still works. A cached status could mislead.

    A colleague sizes the OMS connection limit from the average in flight, 0.29. What is wrong?

    • It is an average: 24 agents can each have a request open, so the ceiling is 24
    • Nothing: Little's law gives the exact peak
    • Triple it for safety, giving 0.87
    Answer

    It is an average: 24 agents can each have a request open, so the ceiling is 24. Little's law gives an average, and the number of users caps what can be in flight. Tripling it is still far below 24.

    Exercise

    Write the availability row for the year after the pilot: choose a target, state the window, and compute the downtime and error budget in minutes.

    Acceptance criteria:

    • The target, window, SLI and measurement point are stated.
    • The downtime is computed with the arithmetic shown, for calendar and staffed hours.
    • A degradation rule covers the retriever failing.

    Done when the Head of Customer Operations can read the row and see what a bad month looks like in minutes.

    Hint 1: choose the window first

    Decide whether the clock runs every minute or only staffed hours, then multiply by one minus the target.

    Hint 2: say what counts as good

    Say whether a labelled degraded reply counts as good, and name the number that tracks how often replies are degraded.

    Common mistakes

    • A target without a percentile. "Fast" cannot fail; "p95 under 6.0 s at the gateway" can.
    • Sizing from the average. Check the ceiling.
    • Promising "always on". Offer a target and a window.
    • Copying others' latency or limits. Measure your own.

    Knowledge check

    1. Why can you not add hop p95 values to get the end-to-end p95?

    Percentiles of separate hops do not add, so the sum is only a plan: measure the end-to-end p95.

    2. Why does the Lumen sizing use 24 and not 0.29?

    0.29 is an average, so bursts can exceed it; 24 is the most that can be in flight.

    3. What do you get by moving from 99.9 to 99.95 percent, and what does it cost?

    Allowed downtime halves from 43.2 to 21.6 minutes a month. The SRE book says an increment in reliability may cost far more than the previous one.

    4. Why must an availability target name its window?

    99.5 percent is 216 minutes over a 30-day month and 50.4 over 168 staffed hours, so an unnamed window hides a fourfold difference.

    5. Four hard dependencies give 99.20 percent. How does the design meet 99.5?

    Series availability is the product, below its weakest part. Degradation rules let the assistant answer part of a request, and the SLI counts a labelled degraded reply as good.

    You can now

    • Add the targets sheet to your design pack.
    • Split a latency target into hop budgets and timeouts.
    • Estimate average and ceiling concurrency.
    • Convert an availability target into minutes for a stated window.
    • Write a degradation rule for each dependency.

    Further reading

    Last verified: 2026-11-04

    Next

    AI Discovery Workshop: Questionnaire and Stakeholder Map

    Enroll to keep reading

    Lesson 6 of 6 · AI Solution Architect: Design Method and Patterns

    Turn fast and always on into numbers: a latency budget per hop, peak concurrency from Little's law, downtime per month, and rules for degraded service.

    In the rest of this lesson

    • Worked example
    • Practice task
    • Exercise
    • Common mistakes
    • Knowledge check
    • You can now

    Enrolling is free. It unlocks every lesson in AI Solution Architect: Design Method and Patterns, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    AI Solution Architect: Design Method and Patterns· Page 6 of 6
    PreviousIntegrating AI With Customer Order and Policy Systems
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides