Reliability and Scaling for AI Systems

• Provider slowdown at peak and graceful degradation. Problem: one shared pool let a slow provider freeze order status at a sales peak. Pattern: bulkheads, a queue for non-urgent work, a degraded mode. See Bulkheads and queue-based load leveling.
• Retry storm after a rate limit, fixed with jitter. Problem: synchronised retries after 429 responses made the limit event worse. Pattern: jittered backoff, a token bucket sized to the tokens-per-minute limit, one retry owner. See Backoff with jitter.
• Slow tail latency and hedged retrieval requests. Problem: one slow index shard made many answers late. Pattern: a timeout budget and hedged reads of an index replica, with a cost check; the model call is not hedged. See Hedged requests and tail latency.

Under the hood: Rate limiting algorithms

Intuition

A token bucket drips full at a steady rate. Each request takes a token; an empty bucket makes the request wait or fail. A full bucket lets a burst through, and the long-run rate is the drip.

Mechanism

Stripe's 2017 engineering post describes it for its API: "you take tokens on each request, and slowly drip more tokens into the bucket", and an empty bucket rejects the request. Anthropic's rate-limits page says its API "uses the token bucket algorithm", so capacity "is continuously replenished up to your maximum limit, rather than being reset at fixed intervals", and that 60 requests per minute might be enforced as 1 per second.

Figure 8. A client-side token bucket in front of a provider

How to read this: the outer alt is the local limiter, the inner alt is the provider's own limit; a local refusal costs nothing, a 429 counts against you.

1. The bucket starts full (capacity 10) and gains 5 tokens a second, never above capacity.
2. A request takes one token (for a tokens-per-minute limit, its estimated tokens); no token means refuse locally.
3. A 429 with retry-after pauses sending for that long.

Refill is the limit divided by 60 (300 per minute is 5 per second). Set the capacity to 5: how many of the 12 arrivals at second 2 get through?

Of 42 arrivals, 37 are accepted: 2 are refused in the burst at second 2 and 1 in each of seconds 3 to 5.

A fixed-window counter resets at each boundary, which Cloudflare's engineering post calls "not terribly accurate" because spikes pass at the boundary. Its sliding-window counter weights the previous window: with a limit of 50 per minute, 42 requests last minute, 18 so far and the window 15 s old, the rate is 42 x (60 - 15) / 60 + 18 = 49.5.

Figure 9. The same limit in provider documentation, as of 2026-10-22

How to read this: the providers count differently, so a limiter built for one is wrong for the other; check your provider's current page.

Worked numbers (illustrative limits): with 500 requests and 200,000 tokens per minute and 1,500 tokens per request, the binding limit is min(500, 200,000 / 1,500) = min(500, 133) = 133 requests per minute. Anthropic's own example: a 2,000,000 limit with an 80% cache hit rate handles 2,000,000 / (1 - 0.8) = 10,000,000 input tokens per minute.

What this means for architects

Under the hood: Backoff with jitter

Intuition

If 100 clients fail together and all wait 1 second, they all return together and fail again. Jitter makes each wait different, so the returns spread out and the provider sees a trickle.

Mechanism

The AWS Builders' Library says backoff alone is not enough: "If all the failed calls back off to the same time, they cause contention or overload again". Jitter "adds some amount of randomness to the backoff to spread the retries around in time". Capping the backoff leaves "all of the clients ... retrying constantly at the capped rate", so limit the retry count too.

The code in AWS's simulator repository for its jitter blog post defines, with v = min(cap, base x 2^n) (its first retry is n = 1; this page counts from 0):

Figure 10. Three clients return after one 429

How to read this: both branches start from the same three failures; the branch shows whether the second wave lands as one spike or as three separate calls.

1. Compute the window v from the retry number and the cap.
2. Draw the wait from the variant's range.
3. If the server sent retry-after, wait that long plus the jittered wait (a design choice; the provider pages only say to wait that long).
4. Stop at the attempt limit or when the wait would pass the deadline.

Figure 11. First retry of 100 clients that failed together (base 1 s)

How to read this: the expected value divides 100 clients over the 100 ms slices in the wait range; the simulation shows the random peak above it.

AWS's simulation of 100 clients contending to update one object found that jitter cut the call count "by more than half" and, against un-jittered backoff, improved completion time; it concluded that jittered backoff "should be considered a standard approach for remote clients".

What this means for architects

Under the hood: The circuit breaker state machine

Intuition

A breaker stops calling a failing dependency, so callers fail fast instead of piling up behind it. After a pause it lets a few calls through to see whether the dependency is back.

Mechanism

The Azure Architecture Center describes a proxy implemented as a state machine. In Closed, requests pass and failures are counted; if they exceed "a specified threshold within a given time period" the breaker opens and starts a time-out timer. In Open, requests fail immediately. When the timer expires it becomes Half-Open, where "a limited number of requests" pass; if they succeed it closes and resets the counter, and if any fails it reopens and restarts the timer.

Figure 12. Circuit breaker states

How to read this: each arrow names its trigger; Open is the only state that never calls the dependency.

Step through one lifecycle with illustrative thresholds: 5 failures in 30 s, a 20 s open period and 3 trial calls.

The saving in the first variant is 40 x 20 s = 800 calls. Azure lists the choices the numbers hide: which exceptions count (more time-outs than hard failures may be needed to trip), how fast Open becomes Half-Open ("a circuit breaker can fluctuate" if too quick), and one breaker per independent resource. AWS adds that breakers "introduce modal behavior into systems that can be difficult to test" and can lengthen recovery.

What this means for architects

Under the hood: Bulkheads and queue-based load leveling

Intuition

A ship's bulkheads keep a leak in one compartment from sinking the ship. A bulkhead gives each feature its own pool of capacity. A queue is a buffer that lets a fast producer and a slower consumer work at their own paces.

Mechanism

The Azure Bulkhead page describes partitioning so that "resources used to call one service don't affect the resources used to call another". It adds that AI and inference workloads "often require strict bulkheads because of deployment-level quotas and concurrency limits", and suggests isolating model deployments per workload or tenant. The Queue-Based Load Leveling page puts "a queue between the task and the service", which handles messages "at its own pace".

Figure 13. Per-feature pools in front of one provider

How to read this: each pool is a hard cap on concurrent calls for one feature; the queue holds batch work that does not fit.

Figure 14. A queue with a deadline stamp

How to read this: the client learns at once that the work is accepted, and it drains at the Worker's rate, not the arrival rate.

Worked numbers (illustrative; the last three rows use a separate queue served at 8 calls per second):

What this means for architects

Under the hood: Hedged requests and tail latency

Intuition

Fan out to many calls and the slowest one sets your latency. A hedged request sends a second copy of a slow-looking request to another replica and uses whichever answers first.

Mechanism

In "The Tail at Scale" (Communications of the ACM, February 2013), Dean and Barroso note that if each server has a 99th-percentile latency of one second and a request needs 100 such servers in parallel, "63% of user requests will take more than one second" (check, assuming independent servers: 1 - 0.99^100 = 0.634).

Hedging sends to the replica believed best, then "falls back on sending a secondary request after some brief delay". One variant waits until the first request has been outstanding longer than "the 95th-percentile expected latency", limiting the extra load to "approximately 5%".

Figure 15. A hedged request after the p95 delay

How to read this: the second request exists only on the slow branch, so the expected extra load is about the share of requests slower than the delay.

The paper's benchmark reads 1,000 values from a BigTable table spread over 100 servers: a hedge after a 10 ms delay cut the 99.9th-percentile latency "from 1,800ms to 74ms while sending just 2% more requests". It works because latency is "often not inherent in the particular request but rather due to other forms of interference", and it suits read-style, loosely consistent work.

Derived for model calls (this page's arithmetic, not the paper's): an agent step with 5 parallel calls, each slow 1% of the time, is slow in 1 - 0.99^5 = 4.9% of steps. A hedge on 5% of requests that both run to completion adds up to 5% tokens; whether cancelling stops generation and billing is provider-specific, so test it.

What this means for architects

Under the hood: Timeout budgets across a call chain

Intuition

A deadline is one clock for the whole request. Every hop gets what is left, not a fresh timeout. A retry is allowed only if the time left can hold a useful attempt.

Mechanism

The Google SRE book's cascading-failures chapter advises deadline propagation: "Rather than inventing a deadline when sending RPCs to backends, servers should employ deadline propagation". Its example: a 30-second deadline with 7 seconds spent in one server leaves 23 seconds for the downstream call. It warns that no deadline, or a very high one, lets work keep consuming resources after it stopped mattering, and that servers should check the remaining deadline before each stage.

Figure 16. One 10 s deadline across the chain

How to read this: the numbers in each message are what remains of the one deadline, so the retry's timeout is smaller than the first attempt's.

1. The outermost layer fixes the deadline from the user's patience.
2. Each hop passes the remaining time down and sets its timeout to at most that.
3. Before each attempt, check time left (Figure 3), and reserve time for the steps after the model call.

Set the Model stage to 10 s (two 5 s attempts): how far over the deadline is the total?

The stages total 7.6 s of 10 s, so 2.4 s of slack. With a second 5 s attempt the total is 12.6 s, over by 2.6 s. That is why the retry in Figure 16 gets 2.4 s, not 5 s.

What this means for architects

Decision framework

Figure 17. Which defence handles this failure?

How to read this: follow the first matching answer; only one exit in five is a retry.

Comparison of the main options

Failure scenarios to design for

Worked example

Assumptions: illustrative figures for the retailer's assistant. Peak 20 requests per second; mean model latency 6 s; 1,500 tokens per request; each worker handles 10 concurrent calls; target utilisation 70%; provider limits of 1,500 requests and 2,000,000 tokens per minute; a 10% retry budget.

The token limit binds first and, with the retry budget, reaches 99%: almost no room to retry or absorb a 1.2 times burst. The answer is a token-sized bucket, a smaller retry budget or a higher limit, and a queue for non-urgent work, not more workers.

Code sample: one retry owner with a token bucket, full jitter and a deadline

This dependency-free Python sketch takes a clock and sleep function so it can be tested; it was run with a fake clock.

import random, time

class TokenBucket:
    def __init__(self, rate, capacity, clock=time.monotonic):
        self.rate, self.capacity, self.clock = rate, capacity, clock
        self.tokens, self.last = capacity, clock()

    def take(self, n=1):
        now = self.clock()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

class Transient(Exception):
    """429, 5xx or timeout; retry_after is the server's wait in seconds, if sent."""
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

def call_with_retries(send, bucket, deadline, base=1.0, cap=30.0, attempts=3,
                      sleep=time.sleep, clock=time.monotonic):
    """The only retry owner. send(timeout) raises Transient. None means use the fallback."""
    for n in range(attempts):
        remaining = deadline - clock()
        if remaining <= 0 or not bucket.take():       # retries spend tokens too
            return None
        try:
            return send(timeout=remaining)
        except Transient as e:
            wait = random.uniform(0, min(cap, base * 2 ** n))     # full jitter
            if e.retry_after is not None:
                wait += e.retry_after                             # the server's wait is a floor
            if clock() + wait >= deadline:
                return None
            sleep(wait)
    return None

• Every attempt takes a token, because OpenAI's page says unsuccessful requests count toward the limit.
• The timeout passed to send is the time left, never a constant; retry-after is a floor with the jittered wait added on top.
• With a fake clock and random.seed(46), a first failure carrying retry_after=2.0, a second failure and a success, it called at 0 s, 2.89 s and 3.69 s with timeouts of 10, 7.11 and 6.31 s.
• Production use also needs the SDK's retries off, a retry budget across calls, retry-after parsing and a token count for take.

Common mistakes

• SDK retries left on under a retrying gateway. Set the SDK's retries to 0.
• Retrying without jitter. Add full jitter and an attempt limit to every loop.
• Sizing for requests when tokens bind. Compute min(request limit, token limit / tokens per request).
• One timeout for the whole chain. Pass the remaining deadline and reserve time after the model.
• A single global circuit breaker. Key it by provider, model and region.
• Autoscaling workers against a fixed quota. Bound the aggregate rate and ramp gradually.

Try it

Time: 30 to 40 minutes. Take a model call path you know, or the retailer in the worked example.

1. List every layer that can retry, including SDKs, and name one owner per failure class.
2. Split a deadline across the chain with a reserve, and check that a retry still fits.
3. Compute calls in flight with Little's law and find the binding limit.
4. Pick a bucket refill and capacity, and a breaker key, threshold and open period.

Done when every layer has a stated retry rule, every timeout is a slice of one deadline, and each number in your design can be recomputed from stated assumptions.

Key takeaways

• Classify failures first: each class allows a different response, and some allow no retry.
• One layer owns retries, and an SDK's default retries count as a layer.
• A client-side token bucket turns bursts into waiting; whether it counts requests or tokens depends on the provider.
• Jitter spreads synchronised clients and a breaker stops calls to a dead dependency; both need an attempt limit and a fallback.
• Bulkheads and queues keep features apart and absorb bursts, not an average rate above capacity.
• Hedging cuts the tail only for idempotent, interference-driven slowness, and costs extra requests.
• Scaling out stateless workers multiplies calls against shared quotas; bound the aggregate rate.

Self-check

Further reading

• AWS Builders' Library, Timeouts, retries and backoff with jitter, and the AWS Architecture Blog, Exponential Backoff and Jitter.
• Google SRE book, Handling overload and Addressing cascading failures.
• Azure Architecture Center patterns: Circuit Breaker, Bulkhead and Queue-Based Load Leveling.
• Dean and Barroso, "The Tail at Scale", Communications of the ACM 56(2), 2013.
• Provider documentation, as of 2026-10-22: Anthropic rate limits and errors, and OpenAI rate limits.

Last verified: 2026-10-22

Next

Continue to Cost and unit economics for AI systems, which prices the retries, caches and fallbacks designed here.