What you'll learn: By the end of this lesson you will be able to tell throughput from goodput, explain how retries and a queue form a feedback loop that keeps a service down after its trigger has gone, choose where and how to shed load (at the door, by deadline, by priority), say what a retry budget and a client-side throttle add, watch a deterministic simulation show a storm outlast a one-second slowdown, and name the way shedding can hide an overload from an autoscaler.
A nightclub has room for 200. A good doorman counts people in and, at 200, turns the next ones away politely, so everyone inside has a good night. A bad doorman lets everyone in, the floor fills, nobody can reach the bar, and the fire marshal closes the club. There is a second failure that matters here: a bad bar. Orders pile up, guests who waited too long walk out, and the bartender keeps pouring drinks for people who left. The work is done and nobody is served.
Servers behave the same way. When demand exceeds capacity, either the server refuses some work on purpose, or it accepts everything, runs late on everything and spends effort on callers who gave up. This lesson is about the first choice and the loop that makes the second one worse. Backpressure covers bounded queues, rejecting, waiting and credits in general and has its own lab; this lesson does not repeat that. It focuses on retries, deadlines and priorities, and its lab is separate (see "Build it"). The timing and waiting rules of retries are in timeouts, retries, backoff and jitter, and the toolbox overview is resilience patterns.
Verified with Java 25.0.3 (the lab), on 2026-11-08. The sources are the Amazon Builders' Library, the Google SRE book and RFC 9110.
The core idea
Overload is dangerous because it feeds itself. A server over capacity answers slowly; slow answers turn into client timeouts; timeouts turn into retries; retries add load. The Builders' Library describes it this way: an overload creates its own feedback loop that results in overload as a steady state. Load shedding cuts that loop at its first step by refusing the excess immediately, so the work that is accepted still finishes in time.
Three examples run through the lesson, each labelled:
- Minimal example: the Java simulation in "Build it": a one-second slowdown, with and without shedding and retries.
- Realistic example: ShopStream's inventory reservation under a slow database, in "In practice".
- Failure example: shedding that kept CPU low, so the autoscaler never added servers.
Throughput is not goodput
The Builders' Library defines throughput as the total number of requests per second sent to the server, and goodput as the subset handled without errors. In its wording, goodput counts only answers that arrive in time for the client to use them. A server can sit at 100 percent CPU with high throughput and almost no goodput: every request is slow, every client has timed out, and each late reply is, in the article's words, like a tree falling in the forest.
The same article gives a simple measure of how fast this goes wrong: when the median latency equals the client timeout, half of the requests time out. And its guidance for testing is the target to design for: the ideal load test result is for goodput to plateau when the service is close to fully utilised, not to fall as load keeps rising.
The retry storm loop
drop expired work"] -.->|breaks the loop here| Q
Figure 1. The loop. How to read this: the trigger appears once, on the left. Everything on the right is the system feeding itself, and it can continue after the trigger ends. The dashed arrow is where shedding cuts in: before the work enters the queue.
The Google SRE book names the same pattern. Its cascading-failures chapter defines a cascading failure as one that grows over time as a result of positive feedback, and calls overload the most common cause. It describes work that has stopped being useful too: queued requests consume memory and add latency, and servers should drop work whose deadline has passed.
Where to shed, and by what rule
Shedding is deciding, per request, to say no. Four rules appear again and again, and they combine.
- Refuse at admission. When in-flight work or queue length passes a limit, reject immediately. The SRE book describes rejecting with HTTP 503 when too many requests are in flight. RFC 9110 defines 503 as "the server is currently unable to handle the request", due to temporary overload or scheduled maintenance, and allows a
Retry-Afterheader with a delay in seconds or a date. A fast refusal costs the server almost nothing and tells the client early. - Drop expired work. If the deadline travels with the request, a server can skip a request whose caller has already left, at no cost, instead of doing the work for nobody. The Builders' Library states the same rule for queues: cap how long a request may wait before it is discarded.
- Prefer important work. The SRE book's overload chapter describes four criticality levels (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE); when a task is itself overloaded, it rejects requests of lower criticality sooner. The Builders' Library gives a simpler example: a load balancer's health-check ping is the most important request a server receives, and a call that finishes an earlier one (an
end()after astart()) should come before new ones. - Choose the order of the queue. The SRE book notes that switching from first-in first-out to last-in first-out, or using the CoDel algorithm, can drop requests that the clients have probably abandoned.
Figure 2. Two places to say no. How to read this: the first no is at the door, before the queue, based on limits and priority. The second is at the head of the queue, for work whose caller is gone. Both are cheap; the expensive mistake is to do the work.
Figure 3. The life of one request under shedding. How to read this: the caller gets an answer in time, an immediate refusal, or (when its deadline passes in the queue) nothing from the server because it has already moved on. In the third case the server spends no worker time.
What the callers should do
Shedding works only if callers do not answer a refusal with a flood of retries. The SRE book's overload chapter gives three rules for callers. Allow at most three attempts per request, and let the failure bubble up after that. Keep a per-client retry budget, so a request is retried only while retries are below 10 percent of requests. Retry only at the layer immediately above the one that is rejecting. It also describes a client-side throttle: each client tracks how many requests it has made and how many the backend accepted over the last two minutes, and once requests exceed K times the accepts (it prefers K of 2), the client rejects some requests itself without sending them. A backend can also return a distinct "overloaded; don't retry" answer when a retry would not help, to prevent a combinatorial explosion of retries up the call tree.
Trade-offs
| Policy | Gain | Cost | Wrong when |
|---|---|---|---|
| Refuse at admission | Cheap, early, keeps latency bounded | Some callers get errors that a longer wait would have turned into answers | The limit is set below real capacity, so you refuse work you could serve |
| Drop expired work | No effort spent on callers who left | The deadline must reach the server | Callers cannot send deadlines |
| Priority shedding | Protects checkout over browsing | A policy to maintain and test | Everything is labelled critical |
| Client-side throttle and retry budget | Stops the loop at the source | Some retries that would have worked are not made | One client, nothing to share a budget with |
| Absorb with a big queue | Smooths short bursts | Long waits, stale work, memory | Overload lasts longer than the callers' timeouts |
Build it
The lab is examples/backend/labs/load-shedding-retry-storm. The backpressure lab compares bounded queues, rejecting, blocking and credits for one producer; this one asks a different question: what happens when callers time out and retry, and which rules keep the system alive afterwards. A server has 4 workers and a 30 ms service time (capacity 133 requests per second). A new user request arrives every 10 ms (100 per second). For one second, from t=1000, an attempt takes 90 ms, so capacity falls to 44 per second: the trigger. Callers wait 500 ms. The simulation moves a manual clock one millisecond at a time with a fixed order of events, so every number repeats. The server's choice is made in one method:
private boolean admit(Req req) {
return switch (policy) {
case NO_SHED -> true;
case BOUNDED, BOUNDED_EXPIRE -> queue.size() < sc.queueLimit();
case PRIORITY -> queue.size() < (req.critical ? sc.queueLimit() : sc.lowPriorityQueueLimit());
};
}Steps.
- Run the tests: every attempt ends in exactly one place, a healthy system loses nothing to shedding, bounded queues never exceed their limit, the unshed server stays down after the trigger when callers retry, and priority protects critical requests.
- Run the demo and compare with the output below.
Expected output, captured from a real run (Java 25.0.3, 2026-11-08):
Load shedding and retry storms lab. All times are simulated milliseconds; no real clock, threads or randomness.
A server has 4 workers and a 30 ms service time: capacity 133 requests/s. A new user request arrives every 10 ms (100/s).
The trigger: from t=1000 to t=1999 the service time is 90 ms, so capacity falls to 44 requests/s for one second.
Callers time out after 500 ms. Queue limit 40 (priority policy: 20 for low priority). Every 5th request is critical.
Success rate = share of the user requests that STARTED in a window and finally succeeded.
Part 1. Healthy system, no trigger: shedding must cost nothing (every row should read 100.0).
server policy 0-999 1000-1999 2000-3999 4000-7999 tries/req
--------------------------------------------------------------------------------------------------
no shedding 100.0% 100.0% 100.0% 100.0% 1.00
bounded queue 100.0% 100.0% 100.0% 100.0% 1.00
bounded + drop expired 100.0% 100.0% 100.0% 100.0% 1.00
priority + drop expired 100.0% 100.0% 100.0% 100.0% 1.00
Part 2. The trigger with callers that retry 3 times (100, 200, 400 ms backoff). Four server policies.
server policy 0-999 1000-1999 2000-3999 4000-7999 tries/req
--------------------------------------------------------------------------------------------------
no shedding 100.0% 62.0% 1.0% 0.0% 2.22
bounded queue 100.0% 100.0% 100.0% 100.0% 2.02
bounded + drop expired 100.0% 100.0% 100.0% 100.0% 1.74
priority + drop expired 100.0% 99.0% 100.0% 100.0% 2.22
Part 3. The same trigger, same two servers, three caller behaviours.
server / callers 0-999 1000-1999 2000-3999 4000-7999 tries/req
--------------------------------------------------------------------------------------------------
no shedding / no retries 100.0% 54.0% 100.0% 100.0% 1.00
no shedding / 3 retries each 100.0% 62.0% 1.0% 0.0% 2.22
no shedding / 3 retries, 10% budget 100.0% 66.0% 94.0% 100.0% 1.15
bounded + drop expired / no retries 100.0% 57.0% 100.0% 100.0% 1.00
bounded + drop expired / 3 retries each 100.0% 100.0% 100.0% 100.0% 1.74
bounded + drop expired / 3 retries, 10% budget 100.0% 61.0% 100.0% 100.0% 1.14
Part 4. Where the work went (whole run, callers retry 3 times). Attempts = every call the server saw.
server policy attempts refused dropped on time too late wasted work maxQ
-----------------------------------------------------------------------------------------------
no shedding 2716 0 0 164 1308 39780 ms 1510
bounded queue 1139 299 0 800 40 1740 ms 40
bounded + drop expired 985 157 6 800 22 1200 ms 40
priority + drop expired 1064 256 0 799 9 810 ms 21
Part 5. A harsher trigger (150 ms per attempt for one second, capacity 27/s), callers do not retry. Success rate of requests that started in t=1000 to 1999.
server policy critical other
------------------------------------------------
bounded + drop expired 35.0% 40.0%
priority + drop expired 60.0% 30.0%Reading part 2. Without shedding and with retrying callers, a one-second slowdown never ends: requests that start in the following two seconds succeed 1.0 percent of the time and, from t=4000, none do, although the server is back to full speed. Part 4 shows why: the queue reached 1,510, the server did 1,308 late replies against 164 on time, and wasted 39,780 ms of worker time on callers who had left. Every bounded policy recovered fully.
Reading part 3. The loop needs both halves. The same unshed server recovers when callers do not retry (100 percent from t=2000) and nearly recovers when they retry under a 10 percent budget (94 percent, then 100). A server that refuses early and drops expired work stays healthy even with naive retries. A nuance the table also shows: with a fast-refusing server, retries help during the trigger itself (100 percent with retries against 57 percent without), because a refusal costs nothing and the trigger ends within the retry window. Retries are not wrong; unbounded ones against a server that keeps stale work are.
Reading part 5. With a harsher trigger and no retries, priority shedding lifts the success of critical requests from 35 to 60 percent at the cost of the others (40 down to 30). That is the point: when you cannot serve everyone, choose who.
Break it. In Simulation.admit, change the bounded rule to true: the bounded rows behave like the unshed row and three tests fail (3 of 14 failed when tried, then the file was restored).
What the model leaves out. Network time, several services, jitter, breakers, cancelling queued attempts, autoscaling. The percentages describe this model with these parameters only.
In practice: inventory reservation under a slow database
This is a realistic scenario, not a case history.
This is an educational scenario on ShopStream, the course's grocery marketplace. Nothing here describes a real company. Besides the shared peak of about 58 checkouts per second, every figure is illustrative.
The situation
Checkout reserves stock from the Inventory service before charging. Stores also push bulk stock and price updates to Inventory, which shares its database. A large update from one store slows the database for 30 seconds. Inventory's calls to the database take ten times longer, its request queue grows, and Checkout, which waits up to two seconds and retries twice, sends more calls. Checkout's success rate drops and does not recover when the update finishes.
Starting architecture
Figure 4. One queue for two kinds of work. How to read this: the store updates and the checkout reservations share a queue and a database. A slow database makes both wait; the retries then pile onto the same queue.
Options considered
| Option | What it fixes | Cost | Risk |
|---|---|---|---|
| A. Scale Inventory out | More workers | Money, and the database is the limit | The database slows further under more clients |
| B. Bounded admission, deadline from Checkout, priority for reservations, retry budget in Checkout | Stops the loop and protects checkout | A policy to write and test | Wrong limits refuse work that could be served |
| C. Move store updates to a separate service and database | Removes the shared bottleneck | A project | Not ready for the next sale |
The decision
Option B now, option C later. Inventory admits at most a set number of in-flight calls (set from a load test, because the limit must track the database's real capacity), refuses store updates first, answers with 503 and Retry-After, and skips any request whose deadline has passed. Checkout sends a deadline with each reservation, retries at most once and only within a 10 percent budget, and shows "we are confirming your stock" instead of an error when the budget is spent. Both sides alert on goodput, not on CPU.
Rollout and verification
Ship the deadline header and the expiry check first, since they change nothing for a healthy system (the lab's part 1 shows this property). Then the admission limit in log-only mode for a week, then enforcing for store updates, then for everything. Rehearse by slowing the database in a test environment for 30 seconds and confirm that Checkout's success rate returns to normal within seconds of the end of the slowdown.
The 2 a.m. failure
Overnight, a store update slows the database again. The shedding works, Inventory stays at modest CPU, and checkout stays healthy, but the page reads "reservation latency p99 above 1 s for 20 minutes". The autoscaler scales on CPU, shedding keeps CPU low, and the instance count never rises while the database is the real limit and the refused store updates are accumulating. The Builders' Library warns that, if misconfigured, load shedding can disable reactive autoscaling. First checks: refusals per second by priority, goodput against offered load, and the autoscaler's metric. Mitigation: raise the limit only if the database has room, and pause the bulk jobs. Fix: scale on a signal that includes refusals or queue age, and cap the store updates' share with their own quota.
Architect's notes
- Measure goodput, not throughput. Count the responses that arrived in time and succeeded.
- Shed before the work, not after. A request refused at the door costs nothing; a request dropped after processing wasted the whole effort.
- Send deadlines. Without them the server cannot tell stale work from live work.
- Tell callers how to behave. A 503 with
Retry-After, a retry budget and a client throttle turn a refusal into a pause instead of a flood. - Test above the breaking point. Check that goodput stays flat as offered load keeps rising.
Common mistakes
- Unbounded queues. They hide overload and create stale work.
- Retrying every failure three times at every layer. See the timeouts and retries lesson.
- Shedding on CPU alone. A server can be overloaded by waiting, with low CPU.
- Setting the limit below real capacity. You refuse work you could have done.
- Giving everything the highest priority. Priority only helps when some work is sheddable.
- Forgetting the autoscaler. Shedding changes the metrics it watches.
Key takeaways
- Goodput, not throughput, is what overload damages.
- Retries and queues form a loop that can outlast the trigger; refuse at the door to break it.
- Drop work whose caller has left; pass deadlines to make that possible.
- Shed by priority: when you cannot serve all, choose who.
- Callers need a retry budget and a throttle so that refusals lead to pauses.
- Check the effect on autoscaling and alerting.
Practice exercises
Exercise 1 (30 min). In part 2 of the output, compare "no shedding" and "bounded queue" for 2000-3999. Name the number in part 4 that explains the difference.
Exercise 2 (45 min). Change the trigger's slow service time from 90 ms to 60 ms and re-run. Does the unshed server still fail to recover? Explain with capacity and offered load.
Exercise 3 (1 hour). Do "Break it" and say which promise each failing test protects.
Exercise 4 (1 hour). Add last-in first-out service when the queue is more than half full and compare goodput.
Related reading
- Backpressure: bounded queues, rejecting and credits, with its own lab.
- Timeouts, Retries, Exponential Backoff and Jitter: the caller's side.
- Resilience Patterns: breakers, bulkheads and shedding together.
- ShopStream Stage 9: Resilience: the checkout under a slow dependency.
- Rate Limiting: per-caller limits.
- Latency vs Throughput: why latency climbs near capacity.
Official documentation
- Amazon Builders' Library: Using load shedding to avoid overload.
- Google SRE book: Addressing Cascading Failures.
- Google SRE book: Handling Overload.
- RFC 9110: HTTP Semantics: 503 Service Unavailable and Retry-After.