Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Reliability and scaling for AI systems. For the model stage's own tail (prefill and output length), see In-product assistant latency.
Context
- A help-desk software product has an answer panel for support agents who are talking to a customer.
- Each question searches a knowledge index split into 8 shards, reranks the passages, then streams a model answer.
- The whole path has a 6 s deadline, because an agent will not wait longer with a customer on the line.
- Every shard has a read-only replica.
Problem statement
Shard read latency (assumed): median 0.15 s, 95th percentile 0.5 s, 98th 1.0 s, 99th 3.0 s. The request waits for its slowest shard.
| Stage | Typical request | Request hit by a slow shard |
|---|---|---|
| Authenticate and route | 0.1 s | 0.1 s |
| Retrieval (slowest of 8 shards) | 0.5 s | 3.0 s |
| Rerank | 0.3 s | 0.3 s |
| Model | 4.2 s | 4.2 s |
| Total against a 6 s deadline | 5.1 s | 7.6 s, 1.6 s over |
With 8 shards, 1 - 0.99^8 = 7.7% of requests wait for a shard at or beyond its 99th percentile, even though each shard is slow only 1% of the time.
Goal: at most 1% of requests miss a shard or the deadline, with at most 5% extra load.
Constraints
- Retrieval is read-only. The model call is not free: it costs tokens and counts against the provider's limits.
- The 6 s deadline is fixed.
- Answers must say when a shard was missed.
- No writes happen on this path.
Why the traditional approach fell short
Keyword search and a cached FAQ answer fast but miss paraphrased questions. Waiting for every shard ties each request to its slowest part. A longer timeout only moves the deadline: the agent is still waiting. Averages look fine here, since the typical request takes 5.1 s, and only the tail reveals the problem.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Non-AI: serve cached exact-match FAQ answers and keyword search for the most common questions | Kept as the last fallback. Instant, but no open questions |
| B. Cut retrieval at 1.0 s and answer from the shards that replied | Kept as a backstop. Alone it leaves 1 - 0.98^8 = 14.9% of requests without a shard's passages |
| C. Hedge shard reads to the replica after 0.5 s, inside a timeout budget | Accepted: removes most of the missing shards for about 5% more reads |
| D. Hedge the model call too | Rejected: its 95th percentile is 4.0 s, so no time is left for a second call, and a duplicate costs tokens |
Solution architecture
Figure 1. Hedged reads on the retrieval stage only
How to read this: the dotted arrow is the only hedge. The model call has one attempt and its own budget.
Figure 2. One shard read with a hedge
How to read this: the outer alt is the normal case, the inner alt is the hedge, and the note is the backstop when both reads are slow.
How AI solved it, step by step
Under the hood
The effect and the remedy both come from The Tail at Scale (Dean and Barroso, 2013).
| Point | What the paper says | Here |
|---|---|---|
| Fan-out amplifies the tail | If each server is slow 1 in 100 times, a request over 100 servers is slow 63% of the time | 8 shards give 7.7% (derived here) |
| What a hedged request is | Send the same request to a second replica after a brief delay, use the first reply, cancel the rest | Hedge to the replica |
| Delay choice | Wait for the 95th-percentile latency, which limits extra load to about 5% | Hedge at 0.5 s |
| Why it works | The latency is often caused by interference, not by the request itself | Assumes independent slowness |
| Where it fits | Replicas serving largely read-only data | Reads only |
AWS adds that APIs with side effects "aren't safe to retry unless they provide idempotency", and a hedge is a duplicate request. Mechanism: Under the hood: Hedged requests and tail latency.
Arithmetic: a shard read is still unfinished at 1.0 s only if the primary is (2%) and the hedge, sent at 0.5 s, is still out after 0.5 s (5%). Independent: 0.02 x 0.05 = 0.1% per shard, 1 - 0.999^8 = 0.80% per request.
What went wrong / failure modes
| Failure | Designed mitigation |
|---|---|
| Both replicas are slow for the same reason, such as a hot partition | The hedge cannot help. Alert on hedge win rate and fix the partition |
| An overloaded index gets 5% more reads at the worst time | Cap hedges at 10% of reads and stop hedging when the index is saturated |
| A cancel is not honoured, so work is duplicated | Count duplicate work in the trace and mark hedges as lower priority |
| A "read" has a side effect, such as an audit write | Hedge only reads known to be idempotent |
| The 95th percentile drifts as load changes | Recompute the delay from a rolling window |
| A shard is missed anyway | Show "answer based on 7 of 8 sections" and offer the FAQ fallback |
Results
These are illustrative targets, not measurements.
| Measure | Before | Target design |
|---|---|---|
| Requests with a shard slower than 1.0 s (missing under a 1.0 s cut) | 14.9% | 0.80% |
| Extra shard reads | 0 | 8 x 0.05 = 0.4 per request, +5% |
| Slow shard resolved in | 3.0 s or timed out | about 0.5 + 0.15 = 0.65 s |
| Worst-case total | 7.6 s | 5.6 s of 6 s, 0.4 s slack |
Tradeoffs and when NOT to use this approach
- Hedging spends capacity to buy latency. At high utilisation the extra reads can create the slowness they cure.
- If slowness comes from the request itself, a long query for instance, a duplicate is slow too.
- Hedging the model call doubles a paid generation and uses the provider's limit; prefer a smaller model fallback.
- If the median is already too slow, hedging does nothing. Fix the common case first.
Design rules extracted
- Fan-out multiplies the tail: with n parallel calls, a 1% slow rate becomes 1 - 0.99^n per request.
- Give every stage a budget that sums inside the deadline, with slack.
- Hedge only idempotent reads, after the 95th percentile, and cancel the loser.
- Check cost before hedging: the added share is about the share of calls that pass the hedge delay.
- Trace whether each hedge fired and won.
Assumptions
- Shard read latency: p50 0.15 s, p95 0.5 s, p98 1.0 s, p99 3.0 s. Model stage budget 4.2 s with its own 95th percentile at 4.0 s.
- The typical request's retrieval time is taken as 0.5 s. Totals: 0.1 + 0.5 + 0.3 + 4.2 = 5.1 s; 0.1 + 3.0 + 0.3 + 4.2 = 7.6 s; budget 0.1 + 1.0 + 0.3 + 4.2 = 5.6 s, slack 0.4 s.
- 1 - 0.99^8 = 7.7%; 1 - 0.98^8 = 14.9%; 1 - 0.999^8 = 0.80%: derived here, not from the paper, assuming independent shards. Hedged unfinished at 1.0 s: 0.02 x 0.05 = 0.001, assuming the two reads are independent, which correlated slowness would break.
- The hedge delay is the 95th percentile, so about 5% of shard reads hedge: 8 x 0.05 = 0.4 extra reads on a base of 8. A hedge's own time is taken as the median, 0.15 s.
- The 10% hedge cap, 8 shards, 6 s deadline and the product are invented.
- References as of 2026-10-22: Dean and Barroso, "The Tail at Scale", Communications of the ACM 56(2), 2013 (https://research.google/pubs/the-tail-at-scale/) and https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter.
Last verified: 2026-10-22