Failure Handling End to End: From Browser to Database

What you'll learn: By the end of this lesson you will be able to walk a request from the browser to the database and say, for each hop, how a failure is detected, contained, retried and made harmless to repeat, how finished work is undone, and what each audience is told. You will be able to explain why a fix at one layer can make another layer worse, name the "unknown outcome" state most designs forget, and review a design with a six-question checklist.

A parcel travels through a chain of depots. Any depot can lose it, hold it for days, or hand it on twice. The sender needs a tracking number so a re-sent parcel is recognised as the same one, a rule for when to re-send, a way to stop sending to a closed depot, a refund when the parcel is truly lost, and a clear message to the customer. Remove any one and the rest stop working: re-sending without a tracking number delivers two parcels.

Software has the same chain. This lesson is about the chain, not one link; the links have their own lessons (resilience patterns, idempotent operations, backpressure, distributed transactions). Here you get a working definition of each and how they fit together and collide.

Verified against the sources in "Official documentation" on 2026-11-08.

The core idea

Every failure passes through six questions, in order, and each needs an owner. If no layer owns a question, the failure falls through to the user.

1. Detect: how do we know something failed, and how long do we wait?
2. Contain: how do we stop one failure from using up shared capacity?
3. Retry: is trying again worthwhile, and which single layer does it?
4. Make repeats harmless: when a retry or duplicate arrives, can it do damage?
5. Compensate: when forward progress is impossible, how is finished work undone?
6. Communicate: what do the caller, the user and the on-call engineer each learn?

The questions depend on each other: a retry without an idempotency key is a duplicate waiting to happen, and a compensation without recorded progress cannot resume.

Three examples run through the lesson: a minimal example (retry arithmetic you can run in Node), a realistic example (ShopStream checkout when the payment provider stops answering) and a failure example (a lost reply that leaves a payment in an unknown state).

One request, six hops, six owners

Follow one click on "Place order" through ShopStream.

Figure 1. The path of one checkout request and the decision each hop owns. How to read this: left to right is the request. The sublabels name what each hop is responsible for. The provider sits outside the boundary, so the system can bound its behaviour but not fix it.

Each hop makes a different decision. The browser retries with the same key or shows a state. The edge sheds load or passes the remaining time down. Checkout degrades, retries or fails the order. Payments stops calling or calls with the key. The database platform fails over, and the service retries only what is idempotent. At the provider, a charge can succeed while the reply is lost: nobody can decide at that moment, and only reconciliation, later, settles it. That last case is the hard one and returns in the failure example.

Detect: a timeout is the only detector you always have

A caller cannot tell a dead server from a slow one or a lost reply from a lost request, the fact under the CAP proof (CAP theorem and PACELC, fallacies of distributed computing). Three signals cover the path:

• Timeouts and deadlines on every call. The gRPC documentation defines a deadline as a point in time past which a client is unwilling to wait, says gRPC sets none by default, and says propagating a deadline to outgoing calls is supported by some implementations (on by default in Java and Go, explicit in C++). The SRE chapter on cascading failures likewise advises deadline propagation, so inner calls stop working for a caller who has left.
• Error responses: HTTP 503 or 429, gRPC UNAVAILABLE or RESOURCE_EXHAUSTED.
• Health and saturation signals that fire before callers fail. The SRE book's four golden signals are latency, traffic, errors and saturation, and it says "a slow error is even worse than a fast error", so failed requests belong in latency measurements (monitoring chapter).

A timeout does not say whether the work happened: the gRPC status page says of DEADLINE_EXCEEDED that for state-changing operations the operation may still have succeeded. Step 5 returns to this.

Contain: stop one failure from becoming everyone's

Containment keeps a slow dependency from using every thread, connection or queue slot.

• A circuit breaker counts recent failures and, past a threshold, fails calls at once; after a pause it lets a few trial calls through. The Azure pattern page names the states closed, open and half-open and notes that retry expects eventual success while the breaker avoids calls likely to fail.
• A bulkhead gives each dependency its own pool, so one stuck dependency cannot take the shared one.
• Load shedding refuses some work early; graceful degradation serves a simpler answer on purpose, such as a static list in place of recommendations.

Containment decides what the caller sees instead of the answer, so it must be agreed with the layers above: an open breaker that throws into a caller that retries three times turns containment back into load.

Retry: one layer, one budget

Retries fix brief, random failures and worsen overload. The Amazon Builders' Library calls retries "selfish" and, for a five-deep stack with three retries per layer, says database load can grow 243 times. Its best practice is to retry at a single point in the stack.

Minimal example (runs in Node). Assume 3 attempts per layer and ShopStream's illustrative peak of 58 checkouts per second:

const attempts = 3;
const peakCheckoutsPerSecond = 58;

for (let layers = 1; layers <= 4; layers++) {
  const callsPerClick = attempts ** layers;
  console.log(`${layers} retrying layer(s): ${callsPerClick} provider calls per click, ${peakCheckoutsPerSecond * callsPerClick} per second at peak`);
}

Output, from a real run on Node 26.12.1:

1 retrying layer(s): 3 provider calls per click, 174 per second at peak
2 retrying layer(s): 9 provider calls per click, 522 per second at peak
3 retrying layer(s): 27 provider calls per click, 1566 per second at peak
4 retrying layer(s): 81 provider calls per click, 4698 per second at peak

A struggling provider sees 27 times its normal load when three layers retry, and 3 times with one.

Figure 2. The retry gate, one question per rule. How to read this: a retry happens only when every gate passes. The first gate is about ownership: it prevents the multiplication above.

The rules behind the gates:

• Safe to repeat. RFC 9110 section 9.2.2 says PUT, DELETE and the safe methods are idempotent, that a client should not automatically retry a non-idempotent request unless it knows the request is actually idempotent or can detect that the original was never applied, and that a proxy must not automatically retry non-idempotent requests.
• Transient only. The SRE chapter says not to retry permanent errors or malformed requests; gRPC says FAILED_PRECONDITION should not be retried until the state is fixed.
• Backoff, jitter and a budget. The SRE chapter recommends randomized exponential backoff, a cap per request and a server-wide retry budget. gRPC's retry throttling is a token bucket: failures remove a token, successes add a fraction, and retries pause below half.
• Obey the server. Retry-After is seconds or an HTTP date (RFC 9110 section 10.2.3); a 503 (section 15.6.4) and a 429 (RFC 6585) may carry it.

Make repeats harmless: the idempotency key

Retries create duplicates, and so do double submits and at-least-once brokers. The defence is on the receiving side: recognise the repeat and return the first result.

Stripe's API reference shows the shape. The client generates a unique key per logical operation; for API v1 the server saves the status code and body of the first request made with a key, returns the saved response (even a 500) on retries, may prune keys after at least 24 hours, and errors if a retry's parameters differ. An IETF Internet-Draft for an Idempotency-Key header (draft 08 at the time of writing, expiring 25 April 2027, so not a standard) proposes 422 for a key reused with a different payload and 409 while the original is still running.

The hard part is the receiver: the key check and the work must be one atomic step, the key must be stored durably, and a request still running needs its own answer. The payments stage builds the durable version.

Compensate: when forward progress is impossible

Retries and keys push a step towards success. When it cannot succeed, earlier finished steps must be undone, and a database rollback is unavailable because each service committed to its own store. The Azure compensating transaction and saga pages give the rules:

• A saga is a sequence of local transactions; if one fails, compensating transactions reverse the effects of earlier ones.
• A compensation is application-specific, need not restore the exact earlier state, and can fail, so the system records progress to resume from and makes each step an idempotent command.
• A pivot transaction marks the point of no return: irreversible steps run last.
• Retry transient failures first; compensate only when retries are exhausted or the failure is not transient. Sometimes a person must decide.

Orchestration and choreography are in Distributed Transactions and ShopStream Stage 5. One point belongs here: a timeout is not a failed step, it is an unknown outcome, and the saga needs a state for it.

Figure 3. The unknown-outcome state. How to read this: a timeout does not lead to Declined. It leads to a state resolved by asking the provider, never by guessing. Voided is allowed only after the provider says there is nothing to undo.

Communicate: three audiences, three messages

The caller needs an answer a program can act on. HTTP has status codes plus Retry-After, and RFC 9457 defines a JSON problem-details format (application/problem+json) with type and detail members. gRPC carries grpc-status in trailers, and its protocol document notes that the HTTP status of a gRPC error response is 200. GraphQL responses can carry both data and errors. Each style reports failure differently; the API styles lesson compares them.

The user needs an honest state. "Payment pending, we will email you" is true in the unknown-outcome state; "Payment failed" may be false and invites a second, real payment.

The operator needs few alerts that name symptoms: the SRE book says every page should be actionable and favours paging on symptoms over causes. A correlation id on every hop (observability and SLOs) lets one failed click be followed end to end.

The other side of this topic. The browser is the first hop and the first place a failure is shown. Client-side retries, error boundaries and what the screen shows while an outcome is unknown belong to the Frontend System Design track's failure-handling lessons, which are planned and not yet published.

How the layers fit, and where they collide

In practice: ShopStream checkout when the payment provider stops answering

This is the realistic example, an educational scenario on ShopStream. Every figure is illustrative.

This is a realistic scenario, not a case history.

The situation

ShopStream's checkout calls the payment provider to authorise a card. During a provider incident, calls take longer than the timeout and some succeed at the provider without the reply arriving. At the shared peak of 58 checkouts per second, shoppers tap "Place order" again and three layers each retry.

Starting architecture

Figure 4. The starting design: three layers retry and the payments hop has no protection. How to read this: amber boxes retry independently, which gives the 27 calls per click from the arithmetic above. The red box has neither a breaker nor an idempotency key, so a retried call can charge twice.

Options considered

The decision

Option C. The payments client owns retries with a budget and jitter; the app, BFF and checkout pass errors up unchanged. Every authorise call carries an idempotency key derived from the order and attempt. A breaker opens when the provider's error and slow-call rate crosses a threshold, and checkout then accepts the order as "payment pending". The unknown-outcome state is resolved by a status query, a webhook or the daily reconciliation, and the order is cancelled and its stock hold released only when the provider has no record.

Rollout and verification

A rollout would measure provider calls per checkout, then run the breaker log-only. Verification: injected provider latency and lost replies in staging, a check that no payment has two captures, and the count and age of orders in "payment pending".

The 2 a.m. failure

The page reads "orders pending payment over 15 minutes is rising" (illustrative). The breaker is open and checkout accepts orders as designed, but the status-query job fails because it shares the provider's saturated connection pool with live traffic. First checks: breaker state, the age of the oldest unknown-outcome order, and whether status queries have their own pool. Mitigation: a separate bulkhead for status queries at a low rate. Lasting fix: treat the resolver as production traffic with its own capacity.

Architect's notes

• Name the owner of each question per hop in the design document. A blank cell is the finding.
• Retry in one place, closest to the failure; every other layer passes errors through.
• Design the unknown-outcome state first for anything that moves money or stock.
• Test the failure path with injected latency and lost replies, not only errors.

Common mistakes

• Retrying at every layer. See the arithmetic above.
• Treating a timeout as a failure instead of an unknown outcome.
• Retrying a non-idempotent call with no key and no way to check the first attempt.
• A breaker with no agreed fallback. The user sees the same error, faster.
• Telling users "failed" when the state is unknown.

Key takeaways

• Failure handling is six questions in order: detect, contain, retry, make repeats harmless, compensate, communicate.
• Each question needs an owner per hop; unowned questions reach the user.
• A timeout shows the answer was late, not whether the work happened.
• Retry at one layer with backoff, jitter and a budget; each extra retrying layer multiplies load.
• Safe repetition comes from idempotent methods or keys, checked atomically on the receiver.
• Compensation is application-specific, can fail, and needs recorded progress.

Practice exercises

Exercise 1 (20 min). Change the script to 4 attempts per layer and 2 retrying layers. Predict the calls per second at 58 checkouts per second, then run it.

Exercise 2 (45 min). Draw the six-question table for ShopStream's "add to cart" path. Where does "make repeats harmless" differ from checkout?

Exercise 3 (1 hour). Extend Figure 3 for a refund that times out. Which states are new, and which step is irreversible?

Exercise 4 (2 hours). Pick a system you know. For its slowest dependency, list who retries, who has a deadline, what the user sees while the outcome is unknown, and who is paged.

Related reading

• Resilience Patterns and ShopStream Stage 9.
• Idempotent and Commutative Operations and Backpressure.
• Distributed Transactions and ShopStream Stage 5.
• Observability and SLOs.

Official documentation

• IETF, RFC 9110, HTTP Semantics (sections 9.2.2, 10.2.3, 15.6.4), RFC 6585 (429) and RFC 9457, Problem Details.
• Google, SRE book: Addressing Cascading Failures and Monitoring Distributed Systems.
• Amazon Builders' Library, Timeouts, retries and backoff with jitter.
• Azure Architecture Center, Compensating Transaction, Saga, Circuit Breaker.
• gRPC, Deadlines, Status codes, Retry.
• Stripe, Idempotent requests.