System Design: Reliability Basics

The request needs all four tiers, so multiply the combined column. The first three cost about 0.0027% together (0.0001% + 0.0001% + 0.0025%). The login service costs 0.1%. The total is roughly 99.9% (about 99.897% before rounding).

The lesson is in the shape of the result. After you add redundancy, the remaining single dependency dominates. To improve the system further, add a fallback for login, for example letting signed-in users keep browsing, rather than a fourth app server.

Timeouts: never wait forever

When service A calls service B and B stops responding, A has to decide how long to wait. If it waits forever, A's threads pile up behind the stuck calls, A is exhausted too, and its own callers start waiting. One slow component can freeze a whole chain.

The fix: every network call needs a timeout. Pick it from the latency you expect. If a call normally takes 20 ms and its 99th percentile is 100 ms, a few hundred milliseconds is reasonable and 30 seconds is not. Use separate timeouts for connecting (fast: a machine is either there or it is not) and for reading the response (longer). And respect the caller's own deadline: if the user's browser gives up after 10 seconds, the backend should not keep working for 60.

Retries, backoff and jitter

Many failures are brief: a lost packet, a restarting server, a leader election. Trying again a moment later often works. That is a retry, and it is powerful and dangerous.

Why naive retries make outages worse

Suppose a database starts responding slowly. Clients time out and retry immediately, so the database now receives the original load plus the retries, gets slower, and triggers more retries. This feedback loop is a retry storm, and it can keep a system down long after the original problem is gone.

It compounds across layers. If the app tier makes up to 3 attempts against the service tier, and the service tier makes up to 3 against the database, one user request becomes 3 × 3 = 9 database calls. A third layer makes it 27, and the system hammers the database hardest when it is weakest.

Exponential backoff with jitter

• Exponential backoff: wait longer after each failure, for example 100 ms, 200 ms, 400 ms, up to a cap such as 10 seconds. This gives the struggling service room to recover.
• Jitter: add randomness to the wait. Without it, a thousand clients that failed together retry together in synchronised waves.

A common recipe is "full jitter": wait a random time between zero and the current backoff ceiling.

import random, time

def call_with_retry(operation, max_attempts=4, base=0.1, cap=10.0):
    for attempt in range(max_attempts):
        try:
            return operation()
        except TransientError:
            if attempt == max_attempts - 1:
                raise
            ceiling = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, ceiling))

The decision of whether to retry at all is a short flowchart. Each question below must say yes before the client tries again.

Read it top to bottom: most failures leave through the left-hand exits. Only transient errors on safe operations, within budget, earn another attempt. In practice:

• Limit attempts. Two or three retries is typical.
• Retry at one layer, usually the outermost one that can do it safely.
• Consider a retry budget, for example "retries may be at most 10% of normal traffic", so a failing dependency cannot be flooded.

Idempotency: making retries safe

A retry raises an awkward question. The first attempt timed out, but did it run? Maybe the server did the work and only the reply was lost. Retrying "charge the customer" might do it twice.

An operation is idempotent if doing it several times has the same effect as doing it once. Setting a profile name to "Asha" is idempotent. Adding 1 to a counter is not. For operations that are not naturally idempotent, use an idempotency key: the client generates a unique ID for the intent and sends it with every attempt. The server remembers keys it has processed and returns the original result instead of repeating the work. Our API design lesson shows this over HTTP.

The key is what turns the second attempt into a lookup. Without it, the second request is a second charge.

Health checks and graceful degradation

A health check is a small endpoint, such as /healthz, that load balancers and orchestrators call regularly. A server that fails several checks in a row is taken out of rotation, and a platform like Kubernetes can restart it. Distinguish two kinds:

• Liveness: "is the process running at all?" A failure means restart it.
• Readiness: "is it ready to take traffic?" A failure means stop sending requests but leave it alone, for example while it warms its cache.

Keep health checks cheap and honest. A check that always says "ok" is useless, and a check that queries five downstream services can turn one outage into a total outage when every server marks itself unhealthy at once.

When a non-essential dependency fails, the system should do less, not nothing. This is graceful degradation, the "shorter menu" from our restaurant.

Decide in advance which features are essential and make the code able to switch the others off. A circuit breaker (a component that stops calling a failing service for a while) is a common tool; Advanced: Resilience Patterns covers it.

Backups and restore testing

Redundancy protects you from hardware dying. It does not protect you from a bad deployment that corrupts data, an engineer who runs the wrong delete command, or ransomware, because replication faithfully copies the damage to every replica. For that you need backups: separate, point-in-time copies kept where the damage cannot reach.

1. Keep copies in a different failure domain (a set of things that can fail together, such as one account, zone or region), ideally a different account and region, with restricted delete permissions.
2. Know your targets. The recovery point objective (RPO) is how much data you can afford to lose, measured in time. The recovery time objective (RTO) is how long you can afford to be down. Backup frequency follows RPO; restore speed follows RTO.
3. Test restores. A backup you have never restored is a hope, not a backup. Schedule drills, time them, and check the restored data is usable.

In practice: a checkout that depends on a payment gateway

The situation

An online retailer with eight engineers and one on-call rotation runs checkout as a Java service on Kubernetes, backed by PostgreSQL and Redis (for carts). Card payments go through a third-party payment gateway over HTTPS. Assume 40 orders per minute on a normal day and 300 on promotion days, a gateway that normally answers in about 1 second at p99, and a contract promising 99.9% availability.

The pain, all assumed: when the gateway slows down, the checkout success rate falls from 99% to about 70%, the checkout pods run out of threads within minutes, and support sees roughly 0.3% of orders charged twice. The defaults explain it. The gateway client has a 30 second timeout and 3 retries, and the ingress (Nginx) has been configured to retry failed requests once, even POSTs (Nginx does not retry POSTs by default). Nothing carries an idempotency key.

Starting architecture

One request can become 2 × 4 = 8 gateway calls, each holding a thread for up to 30 seconds, and none of them is safe to repeat.

Options considered

The decision

Rollout and verification

The new order lifecycle makes the uncertain case explicit instead of guessing:

Ship it behind a feature flag in four steps:

Prove it with dashboards, not hope: checkout success rate, gateway call p99, circuit breaker state, the count and age of payment_unknown orders, retries as a share of traffic, and a duplicate-charge report built from gateway settlement files. The target is no duplicate charges in the canary and payment_unknown older than 15 minutes at zero. Before full rollout, run a game day that makes the gateway slow in staging.

The 2 a.m. failure

The page reads "Gateway p99 above 5 s and breaker open for 5 minutes". The gateway is degraded, so charge calls time out; each timed-out call may or may not have charged the card, so those orders land in payment_unknown. Once the breaker is open, new attempts are refused before any call is sent, so their outcome is known: nothing was charged.

First three checks: the gateway status page, kubectl get pods -n checkout (are pods healthy, or only the dependency?), and SELECT status, count(*) FROM orders WHERE created_at > now() - interval '30 minutes' GROUP BY status. Mitigation: leave the breaker open, show "confirming your payment" for orders already in payment_unknown and "payment temporarily unavailable, please retry" for refused attempts, pause promotional emails, and let the reconciliation job settle the unknown orders when the gateway returns. Do not disable retries by hand, and do not mark orders failed to clear the queue. Follow-up: give the payment client its own thread pool (a bulkhead) so a slow gateway cannot starve cart requests, and add an alert on the age of the oldest payment_unknown order.

Architect's notes

• Measure before tuning. Take timeouts from the gateway's observed p99 and p99.9 over several weeks, not from vendor defaults.
• Ask in the review: "What happens to the order if the response is lost after the charge succeeded?" If nobody can answer, the design is missing a state.
• Count total attempts across layers. Client, ingress, service and SDK retries multiply. Write the product on the design page.
• Idempotency needs an owner. Key storage, expiry and the reconciliation job are real operational work; name the team that gets paged.
• Redundancy is not free. A second provider doubles integration and compliance surface. Buy it only when outage cost is measured and exceeds it.
• Defer multi-provider failover until the cheaper fixes are in and you have the downtime numbers.

Common mistakes

• Adding servers but keeping one load balancer or database. The single remaining box is still a SPOF. Re-draw the diagram and look for the thin neck.
• Calling dependencies without timeouts. One slow service then stalls every caller. Set a connect and a read timeout on every outbound call.
• Retrying forever or at every layer. This multiplies load exactly when the backend is weakest. Cap attempts, use backoff with jitter, and retry in one place.
• Retrying non-idempotent operations blindly. Customers get charged twice. Use idempotency keys or make the operation naturally idempotent.
• Treating replication as a backup. Replicas copy mistakes within moments. Keep separate, versioned backups in another failure domain.
• Never testing restores or failover. Untested recovery fails when it matters. Run scheduled drills and measure the time.
• Health checks that test too much. A check that depends on every downstream service can take down the whole fleet at once. Keep it shallow and separate liveness from readiness.

Key takeaways

• Failure is normal. Design so a fault in one component does not become a failure for users.
• Hunt single points of failure. Ask of every box in the diagram what happens if it disappears, including hidden ones like DNS and certificates.
• Redundancy needs independence. Copies in the same rack, zone or codebase tend to fail together.
• Series multiplies, parallel improves. A chain is weaker than its weakest link; independent copies make a tier far more available.
• Timeouts on every call. Waiting forever turns one slow service into a system-wide stall.
• Retry politely. Bounded attempts, exponential backoff and jitter, and only for transient errors.
• Idempotency makes retries safe. Use idempotency keys for operations that must not run twice.
• Backups are for mistakes, and only tested restores count. Replication does not protect you from deletions or corruption.

Practice exercises

Exercise 1 — Starter (30 min): Draw a simple architecture for a blog: browser, one web server, one database, one DNS provider. List every single point of failure you can find, including at least two that are not hardware. Then redraw the diagram with the cheapest changes that remove the biggest risks.

Exercise 2 — Intermediate (2 hours): For illustration, assume a request passes through a CDN (99.95%), a load balancer pair (each 99.9%), three app servers (each 99.5%, any one is enough), a primary and standby database (each 99.5%), and a payments provider (99.9%). Calculate the availability of each tier and of the whole path, then convert the result into downtime per year. Identify the weakest tier and propose one change, with its cost, that improves the total the most.

Exercise 3 — Advanced (half day): Write a small simulation (any language) of a service that handles 100 requests per second and fails for 30 seconds. Compare three client behaviours: no retries, immediate retries up to 3 times, and exponential backoff with full jitter. Plot or tabulate the load the server sees over time and the time it takes to recover after the fault ends. Write half a page explaining the differences you observe.

Related reading

• System Design: Requirements — how availability targets and the nines table are chosen in the first place.
• System Design: Load Balancing — health checks, draining and keeping the load balancer itself highly available.
• System Design: Replication — how data copies stay in sync, and what happens during failover.
• Resilience patterns for AI systems — timeouts, fallbacks and retries applied to calls to unreliable model APIs.
• System Design: The Approach — the next Foundations lesson, a repeatable framework for any design problem.