Deadlines, one retry owner, rate limits, jittered backoff, circuit breakers, bulkheads, queues and hedging for LLM calls, with the arithmetic to size each.
What you'll learn: how to wrap a model call in the defences that keep an AI system up when the provider is slow, limited or down: deadlines, a single retry owner, a client-side token bucket, jittered backoff, a circuit breaker, bulkheads, queues and hedged requests. You will size each one with numbers you can recompute, and see which parts are provider-specific.
At a glance
Figure 1. The defence layers around one model call
How to read this: solid arrows are the live path, dotted arrows are queued work and fallbacks, and the box is the Model gateway where the per-call defences sit in order.
Every call carries a deadline from the user's patience; each timeout is a slice of what is left.
Exactly one layer retries; an SDK's built-in retries count as a layer.
A client-side token bucket keeps you under the provider's limit instead of discovering it through 429 errors.
A queue absorbs bursts of non-urgent work; it cannot fix an average rate above capacity.
Scale stateless parts out; the provider quota and stateful stores are shared ceilings.
The real-world problem
Composite scenario, not a real company. A retailer's shopping assistant calls one provider. At a sales peak the provider returns rate-limit errors, three layers retry, autoscaling adds workers that send even more calls, and a batch summariser takes every free slot so chat waits. Users give up, and no component reports an outage: each did what it was configured to do.
After this page you will be able to
Classify a model-call failure and say which layer may retry it, and which failures must never be retried.
Size a client-side token bucket from a requests-per-minute and a tokens-per-minute limit, and read a trace of it.
Compute full-jitter backoff windows, a circuit breaker's open-period saving and a queue's drain time.
Split a latency deadline across a call chain and decide whether a retry still fits.
State when a hedged request helps and what it costs.
5xx; Anthropic documents 500 api_error and 529 overloaded_error
Usually
Model gateway, with backoff and a breaker
Timeout
No answer in the deadline; Anthropic documents 504 timeout_error
Unknown whether the work ran
A read-only call, or one with an idempotency key
Spend cap
Anthropic: a 429 with no retry-after and code enforced_spend_limit_reached
No, until access resumes
Nobody; degrade and alert
Malformed output
HTTP 200, but the schema fails
Not a transport fault
The step's owner, once, with the error
Mid-stream failure
An error after a 200 in a server-sent event stream
Partial text exists
Nobody silently
Refusal or truncation
200 with a stop reason (field name varies)
Not transient
The step's owner decides
How to read this: the right column is the point; a failure class decides which layer may act, and some rows allow nobody.
Signals are as of 2026-10-22 and differ by provider: see Anthropic's errors page and OpenAI's rate limits guide (a Retry-After header "when present"). The timeout row is the dangerous one: it does not say whether the first attempt happened, which is why idempotency keys exist.
Try classifying three failures.
Which layer acts on this failure? (interactive when JavaScript is on)
A call returns 429 with a retry-after header of 12 seconds. What do the provider docs say?
Retry at once with the SDK default
Wait at least 12 seconds before retrying
Open the circuit breaker for an hour
Switch to another model for good
Answer
Wait at least 12 seconds before retrying. Anthropic's rate-limits page says to wait that many seconds when the header is present; earlier retries will fail.
A 429 says the organisation reached its monthly spend cap and has no retry-after header. What should the retry owner do?
Retry with full jitter until it works
Stop retrying, serve the degraded path and alert someone who can raise the cap
Add more workers
Raise the timeout
Answer
Stop retrying, serve the degraded path and alert someone who can raise the cap. Anthropic documents that a spend-cap 429 keeps failing until access resumes (provider-specific, as of 2026-10-22).
The model returns HTTP 200 and JSON that fails your schema. Which statement is right?
A transport failure, so the Model gateway retries it
An output failure: one bounded repair attempt by the step's owner, then a fallback
It should open the circuit breaker
Pass it downstream
Answer
An output failure: one bounded repair attempt by the step's owner, then a fallback. The transport worked, so a transport retry or a breaker would be misled. A repair attempt costs tokens, so cap it.
Timeout budgets
Figure 3. Should this attempt start, and with what timeout?
flowchart TD
accTitle: Gate before each attempt
accDescr: Before each attempt the caller checks the time left in the deadline, whether the time left is enough for one useful attempt, and whether tokens and retry budget remain. If any check fails it goes to the fallback. Otherwise it sends with a timeout equal to the smaller of the p99.9 for the call class and the time left minus a reserve.
Start(["Attempt about to start"])
Q1{"Time left in the deadline?"}
Q2{"Enough for one useful attempt?"}
Q3{"Bucket tokens and retry budget left?"}
Run["Send with timeout = smaller of the p99.9 for this call class and time left minus a reserve"]
Fail["Skip to fallback or honest error"]
Start --> Q1
Q1 -->|"none"| Fail
Q1 -->|"some"| Q2
Q2 -->|"no"| Fail
Q2 -->|"yes"| Q3
Q3 -->|"no"| Fail
Q3 -->|"yes"| Run
How to read this: three gates, all leading to the same fallback; only a call that passes every gate is sent.
The AWS Builders' Library says to set a timeout on any remote call, chosen from the downstream latency: pick an acceptable false-timeout rate, "such as 0.1%", and use that percentile (p99.9). Too high wastes resources; too low raises retry traffic.
One retry owner
Figure 4. Retry ownership across the layers, including the hidden one
How to read this: solid arrows are requests, dotted arrows are errors travelling up, and each box states its retry rule.
Anthropic's errors page says its official SDK retries transient failures "twice by default", honouring retry-after, with max_retries to configure or disable it. An SDK under a gateway that also retries is a second owner. Published bounds:
Control
What it bounds
Published figure
Per-request attempts
Tries on one request
Google SRE book: up to three attempts, then let the failure bubble up
Per-client retry budget
Retries as a share of calls
SRE book: retry only while the retry ratio is below 10%; with only the three-attempt limit load grows "just below 3X", with this budget added "just 1.1x"
Per-process budget
Retries per minute
SRE book (cascading failures): "only allow 60 retries per minute in a process"
AWS, for low-cost operations: "retry at a single point in the stack"
Check yourself: your Model gateway makes up to 3 attempts and its SDK also makes 3 attempts (the first try plus two retries). How many calls can one failing request send to the provider?
9 (3 x 3). The SDK's retries sit under the gateway's, so attempts multiply as in the billing-dispute case. Set the SDK's retries to 0 so the gateway is the only owner.
Fallbacks and degraded modes
Figure 5. A fallback ladder, from best answer to honest failure
flowchart TD
accTitle: Fallback ladder for a failing model call
accDescr: The primary model call fails on a rate limit, server error, timeout or open breaker. The next rungs are the same prompt on a second deployment or provider with its own quota, a smaller model with the same output schema, a stored answer labelled with its age, accepting the job to deliver later, and finally an honest error with a retry time.
P["Primary model call"]
S["Second deployment or provider, own quota"]
M["Smaller model, same output schema"]
C["Stored answer labelled with its age"]
Q["Accept the job and deliver later"]
E["Honest error with a retry time"]
P -->|"429, 5xx, timeout, breaker open"| S
S -->|"also failing"| M
M -->|"also failing"| C
C -->|"no stored answer"| Q
Q -->|"queue full"| E
How to read this: each arrow is a step down in quality, taken only when the rung above cannot answer in time.
A fallback needs its own quota (a second deployment on the same limit helps nothing), its own check on a golden set because outputs change, and regular exercise. Tell the user when the answer is degraded.
Capacity and Little's law
Figure 6. Little's law for model calls
Quantity
Formula
Example (illustrative)
Calls in flight
L = λ x W
20 requests per second x 6 s per call = 120
Workers needed
L / calls per worker / target utilisation
120 / 10 / 0.7 = 17.1, so 18
In flight when calls fail fast
λ x 2 s
20 x 2 = 40
In flight when calls hang to a 30 s timeout
λ x 30 s
20 x 30 = 600
How to read this: the last two rows are the same traffic; only the timeout changed, and the held connections rose 15 times.
Little's 1961 abstract defines L (mean units in the system), λ (arrival rate) and W (mean time in the system), and shows L = λW if the means are finite, the processes strictly stationary and the arrival process metrically transitive with nonzero mean; his 2011 retrospective proves a finite-interval form that needs no stationarity. It is an average, so size for the peak.
Scaling the stateless parts and protecting the stateful ones
Figure 7. What scales out and what is a shared ceiling
How to read this: boxes marked scale out can be cloned; the others are limits that every clone shares.
Adding replicas multiplies the call rate against one quota. The Azure queue-based load leveling page warns that autoscaling "without bounding consumers' aggregate downstream rate only moves the overload to downstream dependencies". As of 2026-10-22 Anthropic lets you set lower per-workspace limits and OpenAI defines limits per organisation and project, which makes them provider-side bulkheads. Anthropic also documents acceleration limits: a sharp rise in usage can return 429s, so ramp traffic gradually.
Case files
Three composite case files put these defences under pressure.
A token bucket drips full at a steady rate. Each request takes a token; an empty bucket makes the request wait or fail. A full bucket lets a burst through, and the long-run rate is the drip.
Mechanism
Stripe's 2017 engineering post describes it for its API: "you take tokens on each request, and slowly drip more tokens into the bucket", and an empty bucket rejects the request. Anthropic's rate-limits page says its API "uses the token bucket algorithm", so capacity "is continuously replenished up to your maximum limit, rather than being reset at fixed intervals", and that 60 requests per minute might be enforced as 1 per second.
Figure 8. A client-side token bucket in front of a provider
sequenceDiagram
accTitle: Client-side token bucket in front of the provider
accDescr: A Worker asks the Model gateway to send a request. The gateway holds a token bucket. If a token is available the request goes to the LLM provider, which answers with usage headers or, if it still limits, returns a 429 with a retry-after value that makes the gateway pause. If the bucket is empty the gateway refuses locally without calling the provider and the Worker queues or degrades.
autonumber
participant W as Worker
participant M as Model gateway
participant L as LLM provider
W->>M: request, estimated 1500 tokens
Note over M: Token bucket - refill 5 per second, capacity 10
alt bucket has a token
M->>L: forward the request
alt inside the provider limit
L-->>M: 200 with usage and remaining-limit headers
M-->>W: answer
else the provider still limits
L-->>M: 429 with a retry-after value
Note over M: Stop sending until the retry-after time has passed
M-->>W: overload status with the wait time
end
else bucket is empty
M-->>W: refused locally, no call made
Note over W: Queue it, wait or degrade
end
How to read this: the outer alt is the local limiter, the inner alt is the provider's own limit; a local refusal costs nothing, a 429 counts against you.
The bucket starts full (capacity 10) and gains 5 tokens a second, never above capacity.
A request takes one token (for a tokens-per-minute limit, its estimated tokens); no token means refuse locally.
A 429 with retry-after pauses sending for that long.
Refill is the limit divided by 60 (300 per minute is 5 per second). Set the capacity to 5: how many of the 12 arrivals at second 2 get through?
Token bucket: refill 5 per second, capacity 10 (interactive when JavaScript is on)
Second 0
0 arrived, 0 accepted, 0 rejected, 10 tokens left
Second 1
0 arrived, 0 accepted, 0 rejected, 10 tokens left
Second 2
12 arrived, 10 accepted, 2 rejected, 0 tokens left
Second 3
6 arrived, 5 accepted, 1 rejected, 0 tokens left
Second 4
6 arrived, 5 accepted, 1 rejected, 0 tokens left
Second 5
6 arrived, 5 accepted, 1 rejected, 0 tokens left
Second 6
3 arrived, 3 accepted, 0 rejected, 2 tokens left
Second 7
0 arrived, 0 accepted, 0 rejected, 7 tokens left
Second 8
9 arrived, 9 accepted, 0 rejected, 1 tokens left
Of 42 arrivals, 37 are accepted: 2 are refused in the burst at second 2 and 1 in each of seconds 3 to 5.
Check yourself: in the trace, why are only 5 requests accepted in each of seconds 3, 4 and 5 although 6 arrive?
The burst at second 2 emptied the bucket, so only the refill of 5 tokens per second is available. Sustained traffic above the refill is always refused, however big the capacity.
A fixed-window counter resets at each boundary, which Cloudflare's engineering post calls "not terribly accurate" because spikes pass at the boundary. Its sliding-window counter weights the previous window: with a limit of 50 per minute, 42 requests last minute, 18 so far and the window 15 s old, the rate is 42 x (60 - 15) / 60 + 18 = 49.5.
Figure 9. The same limit in provider documentation, as of 2026-10-22
Topic
Anthropic
OpenAI
Measured in
Requests, input tokens and output tokens per minute, per model class
Requests, tokens and images per minute; requests and tokens per day
Scope
Organisation, with optional lower workspace limits
Organisation and project, not user
What counts
For most models only uncached input tokens; output tokens as produced; max_tokens not used
Calculated as "the maximum of max_tokens and the estimated number of tokens"
After a limit
429; wait for retry-after when present
429; Retry-After "when present"
Failed requests
Not stated on the page read
"unsuccessful requests contribute to your per-minute limit"
How to read this: the providers count differently, so a limiter built for one is wrong for the other; check your provider's current page.
Worked numbers (illustrative limits): with 500 requests and 200,000 tokens per minute and 1,500 tokens per request, the binding limit is min(500, 200,000 / 1,500) = min(500, 133) = 133 requests per minute. Anthropic's own example: a 2,000,000 limit with an 80% cache hit rate handles 2,000,000 / (1 - 0.8) = 10,000,000 input tokens per minute.
What this means for architects
What changes
What you do
A burst hits 429 inside a minute under the limit
Smooth with a client-side bucket: refill = limit / 60, capacity = the burst you accept
The tokens limit binds before the request limit
Budget tokens, not requests; set max_tokens near the expected reply
Cached tokens may not count (Anthropic)
Plan with the hit rate, and alert when it drops
One organisation limit is shared by all features
Use workspace or project limits as provider-side bulkheads
Responses carry remaining-limit headers
Feed them to the bucket so it tracks the provider
Under the hood: Backoff with jitter
Intuition
If 100 clients fail together and all wait 1 second, they all return together and fail again. Jitter makes each wait different, so the returns spread out and the provider sees a trickle.
Mechanism
The AWS Builders' Library says backoff alone is not enough: "If all the failed calls back off to the same time, they cause contention or overload again". Jitter "adds some amount of randomness to the backoff to spread the retries around in time". Capping the backoff leaves "all of the clients ... retrying constantly at the capped rate", so limit the retry count too.
The code in AWS's simulator repository for its jitter blog post defines, with v = min(cap, base x 2^n) (its first retry is n = 1; this page counts from 0):
Variant
Sleep before retry n
No jitter
v
Full jitter
uniform(0, v)
Equal jitter
v / 2 + uniform(0, v / 2)
Decorrelated
min(cap, uniform(base, previous sleep x 3)), previous starts at base
Figure 10. Three clients return after one 429
sequenceDiagram
accTitle: Synchronised retries versus jittered retries
accDescr: Three Workers receive a 429 from the LLM provider at the same moment. Without jitter all three retry together after one second and are limited again. With full jitter each waits a different time between zero and one second, so the retries arrive spread out and the provider can accept them.
autonumber
participant A as Worker A
participant B as Worker B
participant C as Worker C
participant L as LLM provider
A->>L: request at t = 0
B->>L: request at t = 0
C->>L: request at t = 0
L-->>A: 429
L-->>B: 429
L-->>C: 429
alt No jitter
Note over A,C: All three wait exactly 1 s
A->>L: retry at t = 1.0 s
B->>L: retry at t = 1.0 s
C->>L: retry at t = 1.0 s
L-->>A: 429 again
else Full jitter
Note over A,C: Each waits a random time from 0 to 1 s
C->>L: retry at t = 0.2 s
A->>L: retry at t = 0.6 s
B->>L: retry at t = 0.9 s
L-->>C: 200
end
How to read this: both branches start from the same three failures; the branch shows whether the second wave lands as one spike or as three separate calls.
Compute the window v from the retry number and the cap.
Draw the wait from the variant's range.
If the server sent retry-after, wait that long plus the jittered wait (a design choice; the provider pages only say to wait that long).
Stop at the attempt limit or when the wait would pass the deadline.
Figure 11. First retry of 100 clients that failed together (base 1 s)
Variant
Wait range
Expected retries per 100 ms
Simulation (seed 7 per variant)
No jitter
exactly 1 s
100 in one bucket
100
Full jitter
0 to 1 s
100 / 10 = 10
peak 13
Equal jitter
0.5 to 1 s
100 / 5 = 20
peak 24
Decorrelated
1 to 3 s
100 / 20 = 5
peak 9
How to read this: the expected value divides 100 clients over the 100 ms slices in the wait range; the simulation shows the random peak above it.
AWS's simulation of 100 clients contending to update one object found that jitter cut the call count "by more than half" and, against un-jittered backoff, improved completion time; it concluded that jittered backoff "should be considered a standard approach for remote clients".
What this means for architects
What changes
What you do
Many workers share one limit
Use full or decorrelated jitter in every retry loop
Capped backoff still retries forever
Set an attempt limit and a retry budget; AWS limits retries and handles the failure earlier in the architecture
retry-after is present
Treat it as a floor and add jitter on top
Cron jobs and refresh timers start together
Jitter them too; AWS suggests a consistent per-host offset so problems are reproducible
A spend-cap or 400 response arrives
Do not retry; backoff only helps transient faults
Under the hood: The circuit breaker state machine
Intuition
A breaker stops calling a failing dependency, so callers fail fast instead of piling up behind it. After a pause it lets a few calls through to see whether the dependency is back.
Mechanism
The Azure Architecture Center describes a proxy implemented as a state machine. In Closed, requests pass and failures are counted; if they exceed "a specified threshold within a given time period" the breaker opens and starts a time-out timer. In Open, requests fail immediately. When the timer expires it becomes Half-Open, where "a limited number of requests" pass; if they succeed it closes and resets the counter, and if any fails it reopens and restarts the timer.
Figure 12. Circuit breaker states
stateDiagram-v2
accTitle: Circuit breaker states
accDescr: The breaker starts Closed. In Closed it counts recent failures and stays Closed below the threshold. When failures reach the threshold within the window it moves to Open. After the time-out timer expires it moves to Half-Open. In Half-Open a required number of consecutive trial successes returns it to Closed, and any failure returns it to Open and restarts the timer.
[*] --> Closed
Closed --> Closed: failures below the threshold
Closed --> Open: failures reach the threshold within the window
Open --> HalfOpen: time-out timer expires
HalfOpen --> Closed: required consecutive trial successes
HalfOpen --> Open: any trial fails and the timer restarts
How to read this: each arrow names its trigger; Open is the only state that never calls the dependency.
Step through one lifecycle with illustrative thresholds: 5 failures in 30 s, a 20 s open period and 3 trial calls.
Circuit breaker lifecycle for one model deployment (interactive when JavaScript is on)
Provider recovers
Closed, counting. Calls pass through; failures in a 30 s window are counted against a threshold of 5.
Threshold reached. The fifth failure inside the window opens the breaker and starts a 20 s timer.
Open, failing fast. For 20 s every call goes at once to the fallback. At 40 calls per second that is 800 calls the provider never sees.
Half-open, probing. The timer expires and 3 trial calls pass; everything else still takes the fallback.
Closed again. All 3 trials succeed, so the breaker closes and the counter resets.
Provider still failing
Open, timer running. Every call takes the fallback until the 20 s timer ends.
Half-open, probing. The timer expires and the first trial call passes.
A trial fails. The dependency is still unhealthy, so the breaker reopens and restarts the timer.
Longer pause. Azure notes the timer can grow, for example from a few seconds to a few minutes, if the fault persists.
The saving in the first variant is 40 x 20 s = 800 calls. Azure lists the choices the numbers hide: which exceptions count (more time-outs than hard failures may be needed to trip), how fast Open becomes Half-Open ("a circuit breaker can fluctuate" if too quick), and one breaker per independent resource. AWS adds that breakers "introduce modal behavior into systems that can be difficult to test" and can lengthen recovery.
What this means for architects
What changes
What you do
One shared breaker hides which dependency failed
Key breakers by provider, model and region
Not every error is a dependency fault
Count 5xx and timeouts, not 400 or schema failures. For 429 use the token bucket; Azure's own example trips on 429, so this is a design choice
Open must return something
Wire it to the fallback ladder in Figure 5, labelled as degraded
A breaker changes behaviour silently
Emit an event on every state change and alert on Open, as Azure suggests; inject faults to test all three states
Under the hood: Bulkheads and queue-based load leveling
Intuition
A ship's bulkheads keep a leak in one compartment from sinking the ship. A bulkhead gives each feature its own pool of capacity. A queue is a buffer that lets a fast producer and a slower consumer work at their own paces.
Mechanism
The Azure Bulkhead page describes partitioning so that "resources used to call one service don't affect the resources used to call another". It adds that AI and inference workloads "often require strict bulkheads because of deployment-level quotas and concurrency limits", and suggests isolating model deployments per workload or tenant. The Queue-Based Load Leveling page puts "a queue between the task and the service", which handles messages "at its own pace".
Figure 13. Per-feature pools in front of one provider
How to read this: each pool is a hard cap on concurrent calls for one feature; the queue holds batch work that does not fit.
Figure 14. A queue with a deadline stamp
sequenceDiagram
accTitle: Queue-based load leveling with a deadline stamp
accDescr: The Client app asks the Orchestrator to summarise many documents. The Orchestrator puts jobs on the Event bus with a deadline stamp and replies that the work is accepted. A Worker takes jobs at its own pace, skips any whose deadline has passed, calls the LLM provider within its token bucket, and acknowledges on success or leaves the job for retry or a dead-letter queue on failure. The client polls for status.
autonumber
participant C as Client app
participant O as Orchestrator
participant B as Event bus
participant W as Worker
participant L as LLM provider
C->>O: summarise 500 documents
O->>B: enqueue jobs with a deadline stamp
O-->>C: accepted, job id
loop at the Worker's own pace
W->>B: take the next job
Note over W: Skip it if the deadline stamp has passed
W->>L: call within the token bucket
alt success
L-->>W: summary
W->>B: acknowledge
else failure
W->>B: leave for retry or the dead-letter queue
end
end
C->>O: poll job status
How to read this: the client learns at once that the work is accepted, and it drains at the Worker's rate, not the arrival rate.
Worked numbers (illustrative; the last three rows use a separate queue served at 8 calls per second):
Question
Calculation
Result
Chat slots needed
4 requests per second x 3 s
12 of 14, 2 spare
Batch throughput
6 slots / 5 s per call
1.2 calls per second
500 documents
500 / 1.2
417 s, about 7 minutes
Backlog after a 60 s burst at 12 per second
(12 - 8) x 60
240 queued
Wait of the last arrival
240 / 8
30 s
Drain once arrivals fall to 5 per second
240 / (8 - 5)
80 s
What this means for architects
What changes
What you do
A slow feature can hold every slot
Give each feature a pool sized with Little's law plus headroom; queue or reject beyond it
Queues hide an average overload
Alert on queue depth and age; a queue helps only while average arrivals stay below the service rate
A job can wait past its usefulness
Stamp a deadline in the message; AWS's load-shedding article notes of queued requests that a server "has no idea how long the request was in the queue"
Autoscaled consumers raise the downstream rate
Bound the total consumer rate with the token bucket
A user is waiting synchronously
Do not queue; Azure lists a low-latency synchronous caller as a poor fit
Under the hood: Hedged requests and tail latency
Intuition
Fan out to many calls and the slowest one sets your latency. A hedged request sends a second copy of a slow-looking request to another replica and uses whichever answers first.
Mechanism
In "The Tail at Scale" (Communications of the ACM, February 2013), Dean and Barroso note that if each server has a 99th-percentile latency of one second and a request needs 100 such servers in parallel, "63% of user requests will take more than one second" (check, assuming independent servers: 1 - 0.99^100 = 0.634).
Hedging sends to the replica believed best, then "falls back on sending a secondary request after some brief delay". One variant waits until the first request has been outstanding longer than "the 95th-percentile expected latency", limiting the extra load to "approximately 5%".
Figure 15. A hedged request after the p95 delay
sequenceDiagram
accTitle: Hedged request after the p95 delay
accDescr: The Orchestrator sends a request to LLM provider A and waits for the hedge delay, the 95th percentile for this call class. If A answers first no second request is sent. If A is still outstanding the Orchestrator sends the same idempotent request to LLM provider B, uses whichever answers first, and cancels the other.
autonumber
participant O as Orchestrator
participant A as LLM provider A
participant B as LLM provider B
O->>A: primary request
Note over O: Wait for the hedge delay, the p95 for this call class
alt A answers before the delay
A-->>O: answer, no second request
else A is still outstanding
O->>B: hedged request, same idempotent input
alt B answers first
B-->>O: answer
O-)A: cancel
else A answers first
A-->>O: answer
O-)B: cancel
end
end
How to read this: the second request exists only on the slow branch, so the expected extra load is about the share of requests slower than the delay.
The paper's benchmark reads 1,000 values from a BigTable table spread over 100 servers: a hedge after a 10 ms delay cut the 99.9th-percentile latency "from 1,800ms to 74ms while sending just 2% more requests". It works because latency is "often not inherent in the particular request but rather due to other forms of interference", and it suits read-style, loosely consistent work.
Derived for model calls (this page's arithmetic, not the paper's): an agent step with 5 parallel calls, each slow 1% of the time, is slow in 1 - 0.99^5 = 4.9% of steps. A hedge on 5% of requests that both run to completion adds up to 5% tokens; whether cancelling stops generation and billing is provider-specific, so test it.
Check yourself: a request waits on 100 servers that are each slow 1% of the time. What share of requests are slow, and which assumption does that rely on?
About 63% (1 - 0.99^100 = 0.634). It assumes servers are slow independently; the paper states the 63%, not that assumption.
What this means for architects
What changes
What you do
Hedging duplicates work
Hedge only idempotent reads, never a step with side effects, and send the same idempotency key
Long outputs are slow by nature
A hedge cannot shorten a long reply; set the delay per request class and exclude long generations
A hedge adds load and tokens
Cap hedges as a share of calls and price them with the cost estimator
Slowness from rate limiting is not interference
Do not hedge while limited or when the breaker is not Closed; hedges count against the limit
A second provider changes the answer
Evaluate the hedge target on the golden set before it can win a race
Under the hood: Timeout budgets across a call chain
Intuition
A deadline is one clock for the whole request. Every hop gets what is left, not a fresh timeout. A retry is allowed only if the time left can hold a useful attempt.
Mechanism
The Google SRE book's cascading-failures chapter advises deadline propagation: "Rather than inventing a deadline when sending RPCs to backends, servers should employ deadline propagation". Its example: a 30-second deadline with 7 seconds spent in one server leaves 23 seconds for the downstream call. It warns that no deadline, or a very high one, lets work keep consuming resources after it stopped mattering, and that servers should check the remaining deadline before each stage.
Figure 16. One 10 s deadline across the chain
sequenceDiagram
accTitle: One deadline across a call chain
accDescr: The API gateway sets a 10 second deadline. The Orchestrator spends 0.8 seconds on the Retriever and 1.2 seconds on the Tool server, then calls the Model gateway with 7.9 seconds left and a 5 second timeout. The first model attempt times out, leaving 2.9 seconds, of which 0.5 is reserved for the output check, so 2.4 seconds remain. If that is enough for the call class the Orchestrator retries with a 2.4 second timeout, otherwise it uses the fallback.
autonumber
participant G as API gateway
participant O as Orchestrator
participant R as Retriever
participant T as Tool server
participant M as Model gateway
G->>O: request after 0.1 s of gateway work, deadline 10 s from the start
O->>R: retrieve, 0.8 s used
O->>T: tool call, 1.2 s used
O->>M: generate, 7.9 s left, timeout 5 s
M-->>O: timeout after 5 s
Note over O: 2.9 s left, 0.5 s reserved for the output check, 2.4 s available
alt 2.4 s is enough for this call class
O->>M: retry with timeout 2.4 s
else not enough
O->>O: use the fallback now
end
How to read this: the numbers in each message are what remains of the one deadline, so the retry's timeout is smaller than the first attempt's.
The outermost layer fixes the deadline from the user's patience.
Each hop passes the remaining time down and sets its timeout to at most that.
Before each attempt, check time left (Figure 3), and reserve time for the steps after the model call.
Set the Model stage to 10 s (two 5 s attempts): how far over the deadline is the total?
Latency budget for one answer (interactive when JavaScript is on)
Gateway and auth
0.1 s
Retrieve
0.8 s
Tool call
1.2 s
Model
5 s
Output check
0.5 s
Total
7.6 s of 10 s
Slack
2.4 s
The stages total 7.6 s of 10 s, so 2.4 s of slack. With a second 5 s attempt the total is 12.6 s, over by 2.6 s. That is why the retry in Figure 16 gets 2.4 s, not 5 s.
Check yourself: the first model attempt times out at 5 s, with 7.9 s left at its start and 0.5 s reserved for the output check. How long can a retry run?
7.9 - 5 - 0.5 = 2.4 s. If the model's typical time for this call class is longer than that, go straight to the fallback.
What this means for architects
What changes
What you do
Inner timeouts can add up past the outer one
Propagate the remaining deadline; never give an inner layer a fixed timeout larger than what is left
A retry may not fit
Retry only if time left minus the reserve can hold a typical attempt; else fall back
Slow failures hold resources
A 30 s timeout holds 600 calls at 20 per second against 40 at 2 s
Streaming has two clocks
Set a time-to-first-token timeout and a total deadline (streaming)
Decision framework
Figure 17. Which defence handles this failure?
flowchart TD
accTitle: Decision tree for a failed model call
accDescr: A failed call is first checked for permanence such as a 400, 401, 403, spend cap or a schema that can never be valid, which fails now. Otherwise the caller checks time left for a useful attempt, whether this layer is the retry owner, whether retry budget and tokens remain, and whether the breaker is open. Any no leads to a fallback or an overload status upward. If all pass it waits with full jitter, at least the retry-after value, and retries.
Start(["A model call failed"])
P{"Permanent: 400, 401, 403, spend cap, never-valid schema?"}
T{"Time left for a useful attempt?"}
O{"Is this layer the retry owner?"}
B{"Retry budget and tokens left?"}
C{"Breaker open?"}
Now["Fail now and fix the cause"]
FB["Fallback or honest error"]
Up["Return an overload status upward"]
Retry["Wait with full jitter, at least retry-after, then retry"]
Start --> P
P -->|"yes"| Now
P -->|"no"| T
T -->|"no"| FB
T -->|"yes"| O
O -->|"no"| Up
O -->|"yes"| B
B -->|"no"| FB
B -->|"yes"| C
C -->|"yes"| FB
C -->|"no"| Retry
How to read this: follow the first matching answer; only one exit in five is a retry.
If
Then
Because
Revisit when
Several layers could retry
One owner per failure class; SDK retries to 0
Attempts multiply across layers
You add a layer or change SDK
Bursts meet a per-minute limit
A token bucket sized to the limit in the Model gateway
Providers enforce short intervals
The provider changes how it counts
Many clients retry on one signal
Full or decorrelated jitter and an attempt limit
Synchronised retries return as one spike
Traces show correlated retries
One feature can starve another
A bulkhead pool per feature, sized with Little's law
A shared pool lets the slowest feature hold every slot
The traffic mix changes
Work can wait
Queue it with a deadline stamp and bounded consumers
A queue absorbs bursts, not average overload
Queue age passes the deadline
The tail is slow on idempotent reads
Hedge after the p95 delay, with a cost cap
Interference-driven slowness often vanishes on a second try
Limits or long outputs cause it
Comparison of the main options
Technique
Protects against
Cost or risk
Use for LLM calls when
Deadline and timeouts
Hung calls holding resources
Too low raises retries; too high holds slots
Always, from a measured percentile
Jittered retry
Transient faults
Adds load; needs a budget
The call is safe to repeat
Client-side token bucket
429 in bursts
Needs the right unit
A per-minute limit binds
Circuit breaker
A dependency that stays down
Modal behaviour, hard to test
A provider or region fails for minutes
Bulkhead
One feature starving others
Idle capacity in unused pools
Features differ in urgency
Queue-based load leveling
Bursts of non-urgent work
Latency grows if arrivals stay above service rate
Nobody waits synchronously
Hedged request
Slow tail
Extra load and tokens
Idempotent reads, interference-driven slowness
Failure scenarios to design for
Failure
What the system does
Why
Provider returns 429 for a minute
The bucket pauses to retry-after, one owner retries with jitter, queued work waits
Retries must not add to the limit that caused them
Spend cap reached
No retries; degraded path and an alert to the budget owner
The error keeps failing until access resumes
Provider region down for 10 minutes
The breaker opens on 5xx and timeouts, calls go to the second deployment, a status banner shows
Failing fast protects threads; the fallback keeps answering
Queue age passes the job deadline
Jobs past their stamp are skipped and reported
Late answers cost tokens and help nobody
A stream fails after the first token
Keep the partial text, mark it incomplete, offer an explicit retry
The 200 was sent and shown text cannot be retracted
Autoscaling adds workers during a limit event
The aggregate bucket caps total calls and scale-out ramps gradually
More replicas share one quota
Worked example
Assumptions: illustrative figures for the retailer's assistant. Peak 20 requests per second; mean model latency 6 s; 1,500 tokens per request; each worker handles 10 concurrent calls; target utilisation 70%; provider limits of 1,500 requests and 2,000,000 tokens per minute; a 10% retry budget.
Step
Calculation
Result
Calls in flight
20 x 6
120
Workers
120 / 10 / 0.7 = 17.1
18
Requests per minute
20 x 60
1,200 (80% of 1,500)
Tokens per minute
1,200 x 1,500
1,800,000 (90% of 2,000,000)
Headroom before the token limit
2,000,000 / 1,800,000
1.11 times
Bucket refill
1,500 / 60 requests; 2,000,000 / 60 tokens
25 per second; 33,333 per second
Tokens with a 10% retry budget
1.1 x 1,800,000
1,980,000 (99%)
Held calls, 2 s fail-fast against a 30 s hang
20 x 2; 20 x 30
40 against 600
The token limit binds first and, with the retry budget, reaches 99%: almost no room to retry or absorb a 1.2 times burst. The answer is a token-sized bucket, a smaller retry budget or a higher limit, and a queue for non-urgent work, not more workers.
Code sample: one retry owner with a token bucket, full jitter and a deadline
This dependency-free Python sketch takes a clock and sleep function so it can be tested; it was run with a fake clock.
import random, time
classTokenBucket:
def__init__(self, rate, capacity, clock=time.monotonic):
self.rate, self.capacity, self.clock = rate, capacity, clock
self.tokens, self.last = capacity, clock()
deftake(self, n=1):
now = self.clock()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
ifself.tokens >= n:
self.tokens -= n
returnTruereturnFalseclassTransient(Exception):
"""429, 5xx or timeout; retry_after is the server's wait in seconds, if sent."""def__init__(self, retry_after=None):
self.retry_after = retry_after
defcall_with_retries(send, bucket, deadline, base=1.0, cap=30.0, attempts=3,
sleep=time.sleep, clock=time.monotonic):
"""The only retry owner. send(timeout) raises Transient. None means use the fallback."""for n inrange(attempts):
remaining = deadline - clock()
if remaining <= 0ornot bucket.take(): # retries spend tokens tooreturnNonetry:
return send(timeout=remaining)
except Transient as e:
wait = random.uniform(0, min(cap, base * 2 ** n)) # full jitterif e.retry_after isnotNone:
wait += e.retry_after # the server's wait is a floorif clock() + wait >= deadline:
returnNone
sleep(wait)
returnNone
Every attempt takes a token, because OpenAI's page says unsuccessful requests count toward the limit.
The timeout passed to send is the time left, never a constant; retry-after is a floor with the jittered wait added on top.
With a fake clock and random.seed(46), a first failure carrying retry_after=2.0, a second failure and a success, it called at 0 s, 2.89 s and 3.69 s with timeouts of 10, 7.11 and 6.31 s.
Production use also needs the SDK's retries off, a retry budget across calls, retry-after parsing and a token count for take.
Common mistakes
SDK retries left on under a retrying gateway. Set the SDK's retries to 0.
Retrying without jitter. Add full jitter and an attempt limit to every loop.
Sizing for requests when tokens bind. Compute min(request limit, token limit / tokens per request).
One timeout for the whole chain. Pass the remaining deadline and reserve time after the model.
A single global circuit breaker. Key it by provider, model and region.
Autoscaling workers against a fixed quota. Bound the aggregate rate and ramp gradually.
Try it
Time: 30 to 40 minutes. Take a model call path you know, or the retailer in the worked example.
List every layer that can retry, including SDKs, and name one owner per failure class.
Split a deadline across the chain with a reserve, and check that a retry still fits.
Compute calls in flight with Little's law and find the binding limit.
Pick a bucket refill and capacity, and a breaker key, threshold and open period.
Done when every layer has a stated retry rule, every timeout is a slice of one deadline, and each number in your design can be recomputed from stated assumptions.
Key takeaways
Classify failures first: each class allows a different response, and some allow no retry.
One layer owns retries, and an SDK's default retries count as a layer.
A client-side token bucket turns bursts into waiting; whether it counts requests or tokens depends on the provider.
Jitter spreads synchronised clients and a breaker stops calls to a dead dependency; both need an attempt limit and a fallback.
Bulkheads and queues keep features apart and absorb bursts, not an average rate above capacity.
Hedging cuts the tail only for idempotent, interference-driven slowness, and costs extra requests.
Scaling out stateless workers multiplies calls against shared quotas; bound the aggregate rate.
Self-check
1. A request fails with a timeout and the call has a side effect. Why is retrying it unsafe, and what makes it safe?
A timeout does not say whether the first attempt ran, so a retry can do the work twice. An idempotency key honoured where the effect happens makes it safe; a read-only call is already safe.
2. A gateway makes 3 attempts over an SDK that also makes 3 by default. How many provider calls can one request cause, and what is the fix?
3 x 3 = 9. Switch the SDK's retries off and keep one retry owner with a budget.
3. A token bucket has refill 5 per second and capacity 10. Twelve requests arrive in one second after a long idle period. How many are accepted?
10, the full bucket's capacity. The other 2 are refused or queued.
4. Why does full jitter reduce the retry spike of 100 failed clients with a 1 s window?
Without jitter all 100 retry at 1 s. With full jitter about 100 / 10 = 10 fall in each 100 ms slice.
5. A request has an 8 s deadline. Earlier stages took 1.5 s, the first model attempt timed out after 4 s, and 0.5 s is reserved for the output check. How long can a retry run?
8 - 1.5 - 4 - 0.5 = 2.0 s. If a typical attempt takes longer, go straight to the fallback.
Lesson 2 of 7 · AI Architect (Advanced): Operate, Secure and Govern
Deadlines, one retry owner, rate limits, jittered backoff, circuit breakers, bulkheads, queues and hedging for LLM calls, with the arithmetic to size each.
In the rest of this lesson
Under the hood: Rate limiting algorithms
Under the hood: Backoff with jitter
Under the hood: The circuit breaker state machine
Under the hood: Bulkheads and queue-based load leveling
Under the hood: Hedged requests and tail latency
Under the hood: Timeout budgets across a call chain
Enrolling is free. It unlocks every lesson in AI Architect (Advanced): Operate, Secure and Govern, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.