Timeouts, Retries, Exponential Backoff and Jitter

What you'll learn: By the end of this lesson you will be able to pick a timeout from measured latency, decide which failures are safe to retry, write the full, equal and decorrelated jitter rules, explain why stacked retries multiply load and how a retry budget caps it, propagate one deadline through every attempt, and run a Java lab that shows each of these on a simulated clock.

A restaurant's phone line is busy at six o'clock. A hundred hungry people hang up and redial. Those who redial at once collide again; those who redial every minute on the minute collide every minute. The line clears only when callers wait different lengths of time, and some give up.

Reliability basics and resilience patterns introduce timeouts and retries in a paragraph each, and ShopStream Stage 9 applies them to one payment call. This lesson is the dedicated treatment: the numbers behind a timeout, the four waiting rules, the budget and the deadline. The server side of overload is in load shedding and retry storms.

Verified with Java 25.0.3 (the lab), on 2026-11-08, against the Amazon Builders' Library, the AWS Architecture Blog, the Google SRE book and RFC 9110.

The core idea

A retry is extra load that you choose to send to a service that has just failed, so it must be bounded in time, in count and in rate. A timeout bounds how long one attempt may take. Backoff and jitter bound when the next attempt happens. A retry limit and a budget bound how many attempts there are. A deadline bounds the whole call. Each answers a different question, and leaving one out is how an ordinary slowdown becomes an outage.

Three examples run through the lesson, each labelled:

• Minimal example: the Java lab in "Build it": a hundred clients, four waiting rules, and one call with a deadline.
• Realistic example: ShopStream's courier assignment calling a route-estimate provider, in "In practice".
• Failure example: a timeout that hid a completed charge, so the retry charged twice (lab part 4).

Timeouts: how long is the answer still useful?

A timeout is the longest a client waits for a call. Without one, each waiting call holds a thread, a connection and memory, and the Amazon Builders' Library notes the server can run out of resources when callers wait too long.

Its method: choose an acceptable rate of false timeouts, calls that would have succeeded had the client waited a little longer, such as 0.1 percent, and take the matching percentile of the downstream latency, here p99.9. Cautions from the same article: internet clients need worst-case network time added; if the p99.9 is close to the median, add padding; set both a connection and a request timeout; and know what the timer covers. Its example is a 20 ms timeout that began failing after deployments because the timer included secure-connection setup.

A timeout is per attempt, but the user waits for the whole call, so the useful bound is a deadline: an absolute time after which the answer is no longer useful. Every attempt gets the smaller of its own timeout and the time left, and the deadline travels downstream. The Google SRE book says servers should pass a deadline down the call tree instead of inventing new timeouts, so downstream servers can drop work whose deadline has passed.

Figure 1. One call, one deadline, three attempts. How to read this: the numbers are from part 3a of the lab (seed 7). The third attempt is not given the usual 800 ms but only what is left, 358 ms.

Which failures to retry

Retrying is safe only when repeating the request does no harm. RFC 9110 defines a method as idempotent if the intended effect on the server of several identical requests is the same as that of one. PUT, DELETE and the safe methods are idempotent; POST is not. The RFC says a client should not automatically retry a non-idempotent request unless it has some way to know the request is effectively idempotent or that the original was never applied, and that a proxy must not.

A timeout does not tell you the work did not happen. The server may have applied the change and only the reply was lost. The Builders' Library's view is that APIs with side effects are not safe to retry unless they provide idempotency. The usual form is an idempotency key: a token the client generates once per logical operation and sends with every attempt, so the server can recognise a repeat and return the first result (idempotent and commutative operations has the design). And some errors never pass: the SRE book says not to retry permanent errors or malformed requests (a 400 or 404), because neither will ever succeed.

Figure 2. Five gates between a failure and a retry. How to read this: the lab's RetryExecutor implements these gates in this order, and each "no" ends the call with its own status.

Exponential backoff and jitter

Exponential backoff makes each wait longer than the last: base, twice the base, four times, up to a cap. The AWS Architecture Blog post "Exponential Backoff And Jitter" simulated many clients contending to update one item and found that capped backoff alone reduces each client's work only slightly, because the clients still call in clusters: started together with the same rule, they retry together.

Jitter adds randomness so the retries spread out. The post compares three variants, which the lab implements with these definitions:

The post's conclusions, as read from its text: with no jitter the work and time are the worst; full and equal jitter make about the same number of calls; decorrelated makes more in its model; equal jitter was the weakest of the three jittered rules; and "the return on implementation complexity of using jittered backoff is huge." The lab is a simpler model and orders the three jitter rules differently, so treat the order as model-dependent and the gap between jitter and no jitter as the robust result.

Jitter is not only for retries. The Builders' Library recommends it for timers, periodic jobs and delayed work, because clients on a once-a-minute schedule can align early in each minute and daily jobs just after midnight.

Retry layers and budgets

Retries repeated at each layer multiply. The Builders' Library's example is five layers each retrying three times, multiplying the load on a failing database by 243; the Google SRE book's is four attempts at each of three layers, which can create 64 attempts (4^3). The Builders' Library's rule for most calls: retry at a single point in the stack.

Figure 3. The multiplication. How to read this: each layer treats the failure of the layer below as a reason to try again, so the counts multiply: 4 x 4 x 4 = 64. The lab counts this by running the nested loops, and prints 64 for three layers of three retries and 243 for five layers of two.

A retry budget limits the total, not just the count per request. The SRE book's cascading-failures chapter suggests a server-wide budget (its example: 60 retries per minute in a process); its overload chapter describes a per-client budget, where a request is retried only while retries are below 10 percent of requests, with at most three attempts per request and retries only at the layer immediately above the one that rejects. The Builders' Library describes a local token bucket that lets every call retry while tokens remain, which AWS added to its SDKs in 2016.

The lab's RetryBudget is a token bucket: each new request adds a tenth of a token, each retry spends one. Against a dependency that always fails, 1,000 requests with up to three retries each make 4,000 attempts without a budget and 1,109 with it (109 retries allowed, 997 refused). When the dependency is healthy the budget is untouched.

Trade-offs

Build it

The lab is examples/backend/labs/retry-backoff-jitter. It has no real clock and no Math.random: a ManualClock moves only when told to, and SeededRandom (SplitMix64, a small published generator) makes every "random" wait a pure function of the seed. The heart of the waiting rule is this method:

public long nextWaitMillis(int retryNumber, long previousWait, SeededRandom rng) {
    long ceiling = ceilingMillis(retryNumber);
    return switch (jitter) {
        case NONE -> ceiling;
        case FULL -> rng.betweenInclusive(0, ceiling);
        case EQUAL -> {
            long half = ceiling / 2;
            yield half + rng.betweenInclusive(0, half);
        }
        case DECORRELATED -> {
            long upper = Math.max(baseMillis, previousWait * 3);
            yield Math.min(capMillis, rng.betweenInclusive(baseMillis, upper));
        }
    };
}

Steps.

1. Run the tests: waits stay inside each rule's bounds, a retry never sleeps past the deadline, permanent errors are not retried, the budget never exceeds its tokens, and the same seed replays the same trace.
2. Run the demo and compare it with the output below.

Expected output, captured from a real run (Java 25.0.3, 2026-11-08):

Retry, backoff and jitter lab. All times are simulated milliseconds; nothing here reads a real clock or Math.random.
Every random number comes from SeededRandom (SplitMix64), so each line below is the same on every run.

Part 1. The herd: 100 clients all call at t=0. The server accepts 10 calls per 100 ms window and refuses the rest at once.
Backoff: base 100 ms, cap 3200 ms, no retry limit. Fastest possible finish: 900 ms (10 windows).
                       | seed 7                             | mean of seeds 1 to 20                 
rule                   | calls refused peak 2nd+  finished |   calls peak 2nd+  finished   refused
--------------------------------------------------------------------------------------------------------
exponential, no jitter |   550     450        90  15900 ms |   550.0      90.0  15900 ms     450.0
full jitter            |   406     306        53   2067 ms |   408.1      56.1   2285 ms     308.1
equal jitter           |   394     294        69   2647 ms |   391.4      66.4   2418 ms     291.4
decorrelated jitter    |   281     181        48   1850 ms |   279.2      50.7   1693 ms     179.2

Part 2. Retries stack: the dependency is down, so every attempt fails. Attempts that reach it per user request.
setup                                                  attempts
---------------------------------------------------------------
1 layer, no retries                                           1
1 layer, 3 retries                                            4
3 layers, 3 retries each (gateway, service, client)          64
5 layers, 2 retries each                                    243
1 layer, 3 retries, 10 percent budget: 1000 requests       1109
  of which retries allowed 109, retries refused by the budget 997, attempts per request 1.109
1 layer, 3 retries, no budget: 1000 requests               4000

Part 3. One call with a 2000 ms deadline. Policy: attempt timeout 800 ms, up to 4 retries, full jitter (base 100, cap 1000), seed 7.
 a) The callee is slow twice (5000 ms), then healthy (200 ms):
  attempt 1 at t=0    timeout 800 ms: timed out (outcome unknown), then waited 39 ms
  attempt 2 at t=839  timeout 800 ms: timed out (outcome unknown), then waited 3 ms
  attempt 3 at t=1642 timeout 358 ms: ok
  result: SUCCEEDED after 3 attempt(s), finished at t=1842 ms
 b) The callee is down (every call takes 5000 ms): the deadline ends the call.
  attempt 1 at t=0    timeout 800 ms: timed out (outcome unknown), then waited 39 ms
  attempt 2 at t=839  timeout 800 ms: timed out (outcome unknown), then waited 3 ms
  attempt 3 at t=1642 timeout 358 ms: timed out (outcome unknown)
  result: DEADLINE after 3 attempt(s), finished at t=2000 ms
 c) The callee answers a permanent error (HTTP 400 style) in 10 ms: no retry.
  attempt 1 at t=0    timeout 800 ms: permanent error
  result: PERMANENT_ERROR after 1 attempt(s), finished at t=10 ms

Part 4. A timeout does not mean the work did not happen. A payment provider charges on arrival, and its first reply takes 900 ms.
The caller times out at 500 ms and retries once (full jitter, seed 7). The second reply takes 100 ms.
  idempotency key none        -> status SUCCEEDED, attempts 2, card charged 2 time(s)
  idempotency key order-8812  -> status SUCCEEDED, attempts 2, card charged 1 time(s)

Reading part 1. Without jitter the clients stay in lockstep: 10 get in per round, 90 retry together, then 80, and so on, 550 calls in all, finishing at 15,900 ms because the waits keep doubling up to the cap. Every jittered rule needs far fewer calls (281 to 406) and finishes in about 1.7 to 2.4 seconds on average; the busiest later window falls from 90 calls to between 48 and 69. Decorrelated jitter makes the fewest calls here, the reverse of the blog's finding for its model: this server refuses instantly and holds no queue, so do not carry the ordering over to a real system.

Reading parts 2 to 4. Layers multiply (64, 243); the budget holds 1,000 requests to 1,109 attempts. In part 4 the timed-out charge still happened, so only the idempotency key prevents the double charge.

Break it. In Backoff.nextWaitMillis, change case FULL -> rng.betweenInclusive(0, ceiling); to case FULL -> ceiling;: full jitter turns into plain exponential backoff, and two tests fail (2 of 30 failed when tried, then the file was restored).

What the model leaves out. Real networks and threads, queueing inside the server, correlated failures, connection setup.

The other side of this topic

On the front end the same failure appears as an app that retries when the network drops, or a user who taps "retry" five times. Cancelling a request that is no longer needed, and retrying with the same idempotency key, belong to the client. The Frontend System Design track plans a lesson on this; it is not published yet, so there is no link.

In practice: route estimates for courier assignment

This is a realistic scenario, not a case history.

This is an educational scenario on ShopStream, the course's grocery marketplace; nothing in it describes a real company. Every figure other than the shared ones (peak of about 58 checkouts per second) is illustrative.

The situation

When an order is placed, an assignment worker asks an external route provider for a delivery-time estimate, then assigns a courier. At peak that is about 58 estimate calls per second. The provider has a 20-second outage during dinner peak and afterwards answers 429 Too Many Requests whenever a burst arrives. The workers' retries arrive in bursts, so the 20-second outage turns into minutes of 429s, and orders wait for couriers.

Starting architecture

Figure 4. Two retry layers and no jitter. How to read this: both the worker and the SDK retry, with no jitter and no budget. After a short outage the 40 workers wake at the same moments (1, 2, 4 seconds), each making up to four SDK attempts: the calls arrive in clusters, and the provider rate-limits the clusters.

Options considered

The decision

Option B, with a cached zone-level estimate as the fallback when the budget refuses (see graceful degradation and fallbacks). The worker owns retries; the SDK's are off. Per-attempt timeout 800 ms, from the provider's measured p99.9 (illustrative), full jitter with base 200 ms and cap 10 s, at most three retries, a budget of 10 percent of calls and a deadline of five seconds from order placement. The estimate is a read, so it is safe to repeat; the later "assign courier" write carries an idempotency key made from the order id.

Rollout and verification

Turn the SDK retries off first in a canary and watch attempts per order; then ship jitter, then the budget. Rehearse an outage with a stub provider that answers slowly. The evidence: attempts per order at or below 1.1 on a healthy provider, provider calls per second flat during recovery instead of in waves, and no order waiting past the deadline.

The 2 a.m. failure

The retries are fixed, but a timer causes the next incident. A cron job re-enqueues every unassigned order from the last ten minutes at exactly 02:05:00. After a 20-second outage the budget ran dry and many orders were passed on unassigned; at 02:05:00 the sweep puts them all back at once, and the provider returns 429 to the burst. The page reads "provider 429 rate above 20 percent". Check the attempts-per-order panel, the provider's 429 rate over time and the sweep's schedule. Mitigate by pausing the sweep; fix it by jittering its start and re-enqueuing in small batches, the Builders' Library's point about timers and periodic jobs.

Architect's notes

• Pick the retry layer on purpose. List every layer between the user and the dependency (client, gateway, service, SDK, driver) and turn retries off everywhere but one.
• Measure attempts per request. It shows a retry storm starting; alarm when it rises above about 1.1 (illustrative).
• Unknown outcome is a state. After a timeout on a write, keep the order pending and reconcile.
• Jitter your timers too. Anything on a schedule synchronises with other schedules.

Common mistakes

• Retrying at every layer. The attempts multiply.
• Backoff without jitter. The clients stay in lockstep.
• Retrying non-idempotent calls without a key. The card may have been charged.
• Retrying permanent errors. A 400 will not become a 200.
• A timeout longer than the caller's deadline. The work continues for nobody.

Key takeaways

• A timeout comes from data: pick a false-timeout rate and use the matching percentile.
• Retry only errors that may pass and operations that are safe to repeat; use idempotency keys for writes.
• Backoff spaces retries; jitter desynchronises callers. Both are needed.
• Retries at several layers multiply (64 for three layers of four attempts); retry at one point and use a budget.
• A deadline is the whole call's bound; each attempt gets the smaller of its timeout and the time left.
• Everything on a schedule needs jitter, not only retries.

Practice exercises

Exercise 1 (30 min). A gateway, a service and a client library each make 3 attempts (2 retries). How many attempts reach the database for one user request? Which layer keeps its retries?

Exercise 2 (45 min). Run the lab and read part 1. Why does the busiest later window fall from 90 calls with jitter? Change the cap from 3,200 to 800 and describe what happens to the no-jitter row.

Exercise 3 (1 hour). Add a Retry-After hint to Callee.Response, make the executor wait at least that long and write a test.

Related reading

• Reliability Basics: the first treatment of timeouts, retries and idempotency.
• Resilience Patterns: breakers, bulkheads and shedding together.
• ShopStream Stage 9: Resilience: the payment call with a timeout and one retry.
• Backpressure: what a server does when callers send too much.
• Load Shedding and Retry Storms: the server side of a retry storm.
• Idempotent and Commutative Operations: making repeats harmless.

Official documentation

• Amazon Builders' Library: Timeouts, retries and backoff with jitter.
• AWS Architecture Blog: Exponential Backoff And Jitter.
• Google SRE book: Addressing Cascading Failures.
• Google SRE book: Handling Overload.
• RFC 9110: HTTP Semantics: section 9.2.2, idempotent methods.