System Design: Resilience Patterns

Retries with budgets

Retries help with brief, random failures such as a dropped packet or a restarting instance. They hurt when the failure is overload, because they add load.

The flowchart reads as a series of gates; a retry happens only if every gate passes. The rules behind it:

• Retry at one layer only. If the client, the gateway and the service each make up to three attempts, one user action can become 27 calls. Retry closest to the failure and let the others fail through.
• Use exponential backoff with jitter so retries from many clients do not arrive in synchronised waves. Honour Retry-After when present.
• Use a retry budget. Allow retries to make up at most a fixed fraction of traffic, for example 10 percent of recent requests (an illustrative assumption). When the budget is spent, stop retrying. This keeps the worst case near 1.1 times normal load rather than 4 times.

Circuit breakers

A circuit breaker wraps calls to a dependency and watches their outcomes. Like the electrical device, it trips to stop damage, then tests whether it is safe to resume.

• Closed: calls pass through and failures are counted over a sliding window.
• Open: the failure rate (or slow-call rate) crossed a threshold. Calls fail immediately without touching the dependency, protecting your threads and giving it room to recover.
• Half-open: after a wait period, a limited number of trial calls are allowed. Success closes the breaker; any failure reopens it.

Tuning matters. Count slow calls as failures (a 20-second success is not healthy). Require a minimum number of calls in the window, or one failed request on a quiet service trips the breaker. Keep one breaker per dependency, or per endpoint, not one global breaker.

What happens when the breaker is open is a product decision, the fallback: cached data, a default value, an empty list or a clear error. Choose a fallback that is safe. A stale price may be unacceptable; "no recommendations right now" almost always is acceptable.

Bulkheads

A bulkhead isolates resources so one consumer cannot starve the others. Typical forms: separate thread or connection pools per dependency (recommendations get 10 threads; when those are stuck, checkout has its own pool); separate instances for critical and non-critical traffic; per-tenant quotas; and cell-based architecture, where the whole stack is deployed as several independent cells so a bad deploy or poisoned request takes down one twentieth of users, not everyone.

Back-pressure and load shedding

When demand exceeds capacity, something has to give; the only choice is whether it gives deliberately or by collapsing. Back-pressure pushes the signal upstream, "slow down, I am full". Bounded queues are the simplest form: when full, the producer is blocked or rejected instead of growing memory without limit. See message queues for how queue depth becomes a health signal.

Load shedding is deliberately dropping some work to keep serving the rest. A server at 100 percent CPU that tries to serve everyone serves no one well: latency rises, clients time out, and work done for timed-out requests is wasted.

Good shedding is selective (drop recommendations before checkout), early (before spending CPU), explicit (503 or 429 with Retry-After so clients back off), and based on concurrency or queue time, not CPU alone.

Hedged requests

Tail latency is a real problem at scale. If one backend call in a hundred is slow, a page that makes 50 calls will hit a slow one about 40 percent of the time (1 − 0.99^50, approximately). A hedged request attacks the tail: send the request to one replica, and if no reply arrives within, say, the p95 latency, send a second copy to another replica, use whichever answers first, and cancel the loser. It costs a few percent extra load, so hedge only idempotent reads, cap hedges under the same budget as retries, and never hedge when the system is overloaded.

Testing resilience: chaos and game days

All the settings above are guesses until tested. Chaos engineering means injecting controlled failures to check that the system behaves as designed: killing instances, adding latency to a dependency, blocking a network path. A good experiment states a steady-state hypothesis ("checkout success stays above 99.5 percent when the recommendation service is slow"), starts with a small blast radius and a kill switch, observes against your SLOs, fixes what it learns, then widens scope.

A game day is a scheduled exercise where the team rehearses an incident, including detection, escalation, communication and rollback, not only the technical failure. Many teams find their runbooks and alerts are the broken part.

In practice: a retail checkout during a partial outage of recommendations and the payment provider

The situation

A mid-sized retailer runs its web shop on Kubernetes with about ten engineers on the checkout team. The checkout service calls three things in the request path: an internal recommendation service (backed by Redis and a feature store), an inventory service on PostgreSQL, and an external payment provider over HTTPS. Traffic at peak is around 300 checkouts per minute (an assumption). In the last incident the recommendation service slowed to 20 seconds per call; the checkout service's single HTTP client pool filled, checkout p99 went from about 1.5 seconds to over 20, and the success rate fell below 80 percent for 25 minutes even though inventory and payments were healthy. Two weeks later the payment provider had a partial outage with p99 latency of 8 seconds, and client retries from the mobile app and the gateway multiplied traffic. Constraints: no budget for a second payment provider this quarter, a Black Friday freeze in six weeks, and a rule that a customer is never charged twice.

Starting architecture

Options considered

The decision

Rollout and verification

Start in staging by injecting latency on the recommendation service and confirming checkout success holds. Run breakers in shadow mode (log what they would have done) for a week, then canary at 5 percent behind a feature flag. Move retries last: enable the service-level retry budget before disabling gateway retries, or you briefly have none. Dashboards that prove it worked: checkout success rate and p99, breaker state per dependency, pool utilisation per dependency, shed requests by priority, and retry ratio (target under 10 percent of requests). Finish with a game day that slows recommendations for ten minutes and a staged payment-provider brownout, with a hypothesis of "checkout success stays above 99 percent".

The 2 a.m. failure

The page reads "Payment breaker open for 5 minutes, checkout success 91 percent". The provider's API is timing out for roughly one request in three. The dangerous part is not the timeout; it is that a timed-out charge may still have succeeded. The checkout service must not retry blindly or tell the customer it failed.

First three checks: the breaker and pool dashboard to see which dependency is tripped; the provider's status page plus a synthetic probe latency to confirm the fault is external; and the retry-ratio panel to make sure the retry budget is holding. Calls the open breaker rejects never reached the provider, so those are known failures and safe to retry later; only calls that timed out are unknown. Mitigation: leave the breaker open so threads stay free, let half-open probes run at low rate, show customers a "payment pending, we will confirm by email" state, and let the reconcile job query the provider (by idempotency key or merchant reference, if the provider supports that lookup) and complete or cancel each pending order. Follow-up fix: shorten the half-open probe interval during partial brownouts, add an alert on the count of pending-payment orders older than 15 minutes, and take the second-provider decision to the quarterly review with the incident data.

Architect's notes

• Measure the dependency graph before choosing patterns. List every call in the request path, mark critical or optional, and record each one's real p99. Most teams discover that two or three optional calls sit on the critical path.
• In design review, ask what the user sees when each dependency is down. If nobody can answer in one sentence, the fallback does not exist yet.
• A breaker without an owner decays. Thresholds drift as traffic changes; assign the owning team, review trips monthly and page on breaker state only for critical dependencies.
• The late-discovered cost is idle capacity. Per-dependency pools and standby cells hold unused resources; state that cost up front as an insurance premium.
• Unknown outcome is its own state. For money and inventory, model "pending" explicitly and reconcile; do not map a timeout to "failed".
• Defer hedging and cells. They pay off at high scale or strict tail targets; breakers, budgets and shedding come first.

Common mistakes

• Retrying at every layer. Multiplication turns three retries into dozens of calls. Retry in one place and use a budget.
• No timeout, or one huge default timeout. Threads pile up behind a slow dependency. Derive timeouts from measured latency and the remaining deadline.
• A circuit breaker with no meaningful fallback. It only turns slow errors into fast errors. Decide what the user sees and make it safe.
• Shared pools for critical and optional work. Isolate with bulkheads.
• Unbounded queues. They hide overload and create huge latency. Bound them and shed when full.
• Hedging non-idempotent requests. You may charge a card twice. Hedge only safe reads.
• Never rehearsing failure. Untested failover and runbooks often do not work. Run game days.

Key takeaways

• Slow can be worse than down. A slow dependency exhausts callers' resources and spreads failure upstream.
• Contain the blast radius. Bulkheads and cells keep one failure in one compartment.
• Propagate a deadline. A timeout budget stops services doing work nobody is waiting for.
• Retries are load. Use backoff, jitter, a single retry layer and a retry budget.
• Circuit breakers need fallbacks. Count slow calls, set a minimum volume and keep one breaker per dependency.
• Shed deliberately. Bounded queues, priority-based shedding and fast 503 or 429 responses beat collapse.
• Hedge carefully. It trims tail latency for idempotent reads at the cost of a little extra load.
• Test it. Chaos experiments and game days show what your configuration really does.

Practice exercises

Exercise 1 — Starter (30 min): A request passes through a gateway, service A, service B and a database. Each layer retries up to three times (that is, up to four attempts) on failure. If the database is completely down, how many attempts reach it for one user request? Then redesign the retry policy so the worst case is at most two attempts at the database, and say which layer keeps its retries.

Exercise 2 — Intermediate (2 hours): Draw the dependency graph of a system you know (or the checkout example). Mark each dependency as critical or optional. For each optional one, write the timeout, the fallback and which pool it uses. Then simulate on paper what happens if one optional dependency begins responding in 20 seconds, with and without a bulkhead, using an assumed request rate and thread pool size.

Exercise 3 — Advanced (half day): Write a game-day plan for a service of your choice. Include a steady-state hypothesis with an SLO-based metric, three experiments of increasing blast radius (latency injection, instance loss, dependency outage), the abort conditions, who is on call, and what you expect to observe for each. Afterwards write a short template for the post-experiment review with columns for expected, observed and action.

Related reading

• System Design: Reliability Basics — the foundations of timeouts, retries and idempotency.
• AI System Resilience — applying these patterns to model calls, which are slow and failure-prone.
• System Design: Observability and SLOs — the signals that tell you when a breaker or shedder should act.
• System Design: Rate Limiting — controlling demand at the edge before it needs shedding.
• System Design: Multi-Region and Disaster Recovery — resilience at the scale of a whole region.