What you'll learn: By the end of this lesson you will be able to rank a feature's dependencies by what a failure costs the user, choose among five kinds of fallback (a default, a stale copy, a simpler computation, queue-and-promise, a second provider), say when a fallback is more dangerous than an error, design the degraded mode so users and operators can see it, keep it tested instead of trusting it, and explain why a degradation switch needs an owner, a kill switch and an alert.
A restaurant's fish supplier does not deliver. The owner can close for the night, or print a smaller menu and tell the waiters which dishes are off. Most guests prefer the smaller menu, provided the waiters know about it before the guests order. The decision was made when the menu was designed, not at seven in the evening with the dining room full.
Graceful degradation is that decision made in advance for software: when something it depends on fails or runs out of capacity, the product offers a simpler service instead of none. It builds on circuit breakers and bulkheads, which detect and contain a failure but leave open what the user sees. ShopStream Stage 9 ranks four features for dropping; this lesson is the dedicated treatment: the kinds of fallback, the risks of each, how to make the smaller service visible and how to keep it working. It pairs with load shedding and retry storms (what to refuse under overload) and timeouts and retries.
Verified against the Google SRE book and the Amazon Builders' Library on 2026-11-08. This lesson has no lab.
The core idea
Every dependency call needs three answers written down before it fails: what the user sees, how long that answer stays acceptable, and who is told. Without them, a failure turns into whatever the code happens to do: a spinner, a stack trace, or a silently wrong number. With them, a failure becomes a lesser but honest service.
Three examples run through the lesson, each labelled:
- Minimal example: a product page whose review service is down: hide the reviews.
- Realistic example: ShopStream's price display and delivery estimate, in "In practice".
- Failure example: a fallback that was worse than no fallback (from the Builders' Library) and a switch nobody had tested.
Five kinds of fallback
A fallback is the alternative path taken when the preferred one fails. They differ in how much they promise.
| Kind | What the user gets | Example | Main risk |
|---|---|---|---|
| Default or empty | A neutral value | An empty "recommended for you" list | Looks broken if the page has no room for the gap |
| Stale copy | The last known answer | Yesterday's catalogue entry from a cache | Wrong data presented as right |
| Simpler computation | A cheaper answer | Plain text ranking instead of personalised ranking | Lower quality, still real data |
| Queue and promise | "We have your request, we will confirm" | An order accepted and processed later | The promise must be kept |
| Second provider | The same answer from elsewhere | A backup payment or map provider | Two systems to run and keep in step |
The Google SRE book describes the third kind as graceful degradation, one step beyond load shedding: serve a lower-quality answer, for example by searching only a subset of the data held in memory, or by using a cheaper ranking algorithm, instead of dropping the request. It adds a caution that applies to all five: graceful degradation should not trigger very often, and the code path you never use is the code path that often does not work.
for the user's goal?"} O -->|yes| D["Hide it or show a default"] O -->|no| S{"Is an old answer safe
within an age limit?"} S -->|yes| ST["Serve the stale copy,
label its age"] S -->|no| C{"Is a simpler computation
good enough?"} C -->|yes| SC["Use it, mark it as approximate"] C -->|no| Q{"Can the work wait?"} Q -->|yes| QP["Queue it, promise a confirmation,
reconcile later"] Q -->|no| E["Fail fast with a clear error"]
Figure 1. A decision for each dependency, made in advance. How to read this: start at the top for one call, not for the system. The first question separates features the user can live without; the next two ask whether an answer that is old or approximate is still true enough; the last two are about promising later or telling the truth now.
When a fallback is worse than an error
A fallback runs only when something is already wrong, so it runs rarely, under stress and with the least testing. The Builders' Library article "Avoiding fallback in distributed systems" says that fallback logic is hard to test, because failure modes such as overload are hard to reproduce, and that distributed fallback strategies often have latent bugs that appear months or years after they were added. Its example is an early Amazon retail incident: each server had a cache with a fallback to query the database directly; the caches failed together, the database was overwhelmed, and the site and fulfilment network went down. In the article's words, the fallback strategy itself amplified the problem and was worse than no fallback at all.
The lesson from that example is general. A fallback that moves load from one component to another must be sized for the load it will receive, and a cache is not a safe design if the backing store cannot survive without it. The article's alternatives are worth knowing:
- Improve the primary path instead of adding a second one, for example by using a more inherently available store.
- Let the caller handle errors, for example with the caller's retries.
- Push data ahead of time, so the request path has fewer moving parts. The article's example is credentials pushed to instances that stay valid for hours.
- Turn fallback into failover: "A service must run both the fallback and the non-fallback logic continuously", so neither is ever dormant.
It also notes that retries can quietly become a fallback: they may kick in during scenarios that the team has never tested, so Amazon monitors retry rates and alarms on frequent retries.
Making the smaller service visible
A degraded service that looks normal misleads. Three audiences need to know.
The user. Say what is reduced and, for stale data, how old it is: "Prices were last updated 12 minutes ago and are confirmed at checkout". Hide controls that cannot work rather than letting them fail.
The operator. The SRE book says to monitor and alert when too many servers enter a degraded mode, and to design a way to turn complex degradation off quickly or tune its parameters. Complex shedding and degradation can cause problems themselves.
The caller of your API. Mark degraded responses in the body or a header so that clients can decide. A stale price flagged as stale can be shown with a caveat; the same price unflagged cannot.
Figure 2. One degraded response. How to read this: the page does not wait for the failing dependency beyond its short timeout. It answers with older data, says how old it is and flags the response. Had the cache entry been older than the limit, the page would have shown "price at checkout" instead.
Figure 3. Three modes, not two. How to read this: a service that flips straight back to normal the moment a dependency answers once will flap, and the returning load can break the dependency again. The recovering state ramps back slowly; the SRE book makes the same point about cold caches, advising a slow ramp so that caches can warm up.
Ranking dependencies
For each feature, list what it calls and rank every call on two axes: how much the user's goal depends on it, and how wrong a fallback answer could be.
| Dependency | If it fails | Fallback | Correctness risk |
|---|---|---|---|
| Reviews service | Page has no reviews | Hide the section | None |
| Image CDN | No product photo | Placeholder image | None |
| Pricing service | Cannot show a current price | Cached price with an age label; checkout always re-checks | Medium: a stale price can mislead |
| Payment provider | Cannot charge now | Queue the order as pending, or fail with a clear message | High: money moves |
| Stock service | Cannot show exact stock | Show "available" only; reserve at checkout | Medium |
The rule the table encodes: degrade freely where the answer is decoration, carefully where it is information and never where it is a commitment. A price shown is information; a price charged is a commitment, and it must come from the authoritative service.
Trade-offs
| Choice | Gain | Cost | Wrong when |
|---|---|---|---|
| Hide the feature | Simple, safe | Smaller product | The feature is the reason for the visit |
| Serve stale data | Page stays complete | Possible wrong answer | The data is a price, balance or stock count used to commit |
| Simpler computation | Real data, lower quality | A second code path to maintain | You cannot test it regularly |
| Queue and promise | Hides outages from users | A promise to keep, pending state, support work | The result cannot be delivered later |
| Second provider | Survives a provider outage | Contract, reconciliation, constant testing | Failover is never exercised |
The other side of this topic
On the front end the same decision is made in the interface: a section that hides itself when its data fails to load, a skeleton that turns into a message, a cached page shown while offline. The Frontend System Design track plans a lesson on graceful degradation in the UI; it is not published yet, so there is no link. The back end's job is to send enough information (stale, partial, degraded) for the interface to decide.
In practice: price display and delivery estimate
This is a realistic scenario, not a case history.
This is an educational scenario on ShopStream, the course's grocery marketplace; nothing in it describes a real company. Every figure other than the shared ones (about 2,300 browsing requests per second at peak) is illustrative.
The situation
The product page calls three things to render: Catalogue (name and photo), Pricing (the current price and any offer) and Delivery estimates (a time window from a route provider). All three are in the request path. When Pricing is slow, the whole page is slow, and a slow page at 2,300 requests per second ties up threads fast. The team must decide what the page shows when Pricing or Delivery estimates fail.
Starting architecture
Figure 4. A page that fails when any call fails. How to read this: all three calls sit on the path of the page. The amber boxes are the ones the team chose to protect against; Catalogue is the page's reason to exist and stays strict.
Options considered
| Option | Behaviour | Cost | Risk |
|---|---|---|---|
| A. Fail the page if any call fails | Simple | Pricing trouble means no page | Largest blast radius |
| B. Per-call timeouts, and a ladder: estimate hidden, price served stale with an age label, Catalogue strict | The page survives both failures | Cache, labels, a degraded-mode flag in the response | A stale price misleads if unlabelled |
| C. Replicate Pricing in a second region | Removes the single dependency | Large | Two copies to keep consistent |
The decision
Option B. The delivery estimate is decoration: when its call fails or exceeds a short timeout, the page shows "estimate at checkout". The price is information: the product page may show a cached price if it is younger than a limit (assume 15 minutes, illustrative), labelled "Price as of 14:05, confirmed at checkout". Checkout never uses the cached price: it calls Pricing and, if Pricing is down, shows a clear message and keeps the cart. The response carries a degraded list naming which parts are stale or missing, so the interface and the dashboards can see it.
Rollout and verification
Ship the flag and the labels first with the fallbacks switched off. Turn them on per dependency in a canary. Rehearse: in a test environment make Pricing return errors, then make it slow, and check the page, the labels and the degraded count. Evidence you want: page success rate during the test, the share of responses flagged degraded, and the age of the cached prices shown. Add a recurring test that exercises the fallback on a schedule, so it is never dormant.
The 2 a.m. failure
Pricing fails overnight. The on-call engineer sets the "serve cached prices" switch, and nothing changes: pages still time out. The page reads "product page p99 above 3 s". The cause is in the configuration: a rename earlier in the week changed the switch's key in one region, so the new key is ignored there, and the code path behind the switch has not run in production since the change. The SRE book's warning applies exactly: the code path you never use is the code path that often does not work. First checks: the degraded-mode metric (it stayed at zero, so the switch did nothing), the configuration key in that region, and the pricing error rate. Mitigation: set the key in the right place or restart with the right configuration. Follow-up fixes: the scheduled fallback test from the rollout, an alert when the switch is on and the degraded count is zero, and an owner for the switch.
Architect's notes
- Write the user-facing sentence for each failure first. If nobody can write it, the fallback does not exist.
- Classify every field as decoration, information or commitment. Degrade the first freely, the second with labels, and never the third.
- Size the fallback for the load it receives. A cache fallback that sends everything to the database needs a database that can take it.
- Run the fallback regularly. A path that runs only in disasters is the one most likely broken.
- Put a kill switch on every complex mode and alert when it is on.
Common mistakes
- Stale data without a label. The user cannot tell it from fresh.
- A fallback that increases load on a failing system. See the Amazon example above.
- Degrading a commitment such as the amount charged.
- A switch with no owner and no test. It breaks silently.
- Flipping back at the first healthy answer. Use a recovering state and ramp up.
- Forgetting the hidden fallback. Retries that start at a failure are a fallback too.
Key takeaways
- Decide in advance what the user sees when each dependency fails.
- Five kinds of fallback: a default, a stale copy, a simpler computation, queue-and-promise, a second provider.
- Degrade freely for decoration, carefully for information, never for commitments.
- A fallback is code that runs rarely: test it on a schedule and give it a kill switch.
- Mark degraded answers for users, operators and API callers.
- A fallback must not move more load onto a component than it can take.
Practice exercises
Exercise 1 (30 min). List the dependencies of a page in a system you know. For each, write the user-facing sentence for its failure and say whether it is decoration, information or commitment.
Exercise 2 (45 min). For ShopStream's pricing fallback, choose an age limit for the cached price. What evidence would let you lengthen or shorten it?
Exercise 3 (1 hour). Design the scheduled test that proves the cached-price fallback works, and the alert that fires when the switch is on but the degraded count is zero.
Exercise 4 (1 hour). The Builders' Library lists ways to avoid a fallback. For the delivery estimate, argue for one of them over hiding the estimate.
Related reading
- Resilience Patterns: breakers, bulkheads and shedding.
- ShopStream Stage 9: Resilience: the ranked degradation ladder for four features.
- Load Shedding and Retry Storms: refusing work under overload.
- Timeouts, Retries, Exponential Backoff and Jitter: bounding the calls a fallback replaces.
- Cache Stampede and Hot Keys: what happens when many caches miss at once.
Official documentation
- Amazon Builders' Library: Avoiding fallback in distributed systems.
- Google SRE book: Addressing Cascading Failures: load shedding and graceful degradation.
- Google SRE book: Handling Overload.
- Amazon Builders' Library: Timeouts, retries and backoff with jitter: on circuit breakers and retries.