QuickCart Stage 9: Surviving Failure Without Losing Orders

Checkout degrades only at its edges: loyalty points and receipts move to the asynchronous pipeline.

Figure 4. What to shed, and when

How to read this: each step sheds less valuable work first, and the last step reserves what remains for checkout. Each step is a switch an engineer or an automatic rule can flip.

The SRE book notes degraded modes rarely run, so exercise them on purpose.

Knowing what is healthy

Health checks: liveness, readiness and startup

A health check is a small request asking an instance whether it is fit for a purpose. The Kubernetes documentation separates three questions (Stage 10 covers Kubernetes itself).

The classic mistake is a liveness probe that depends on other services. The documentation warns that a wrongly built liveness probe can cause cascading failures. If Payments' liveness check called the database and the database slowed, every Payments pod would restart at once. Keep liveness about the process. Put a dependency in readiness only if the pod cannot serve without it, and remember that if every pod shares one database, a blip removes them all together.

startupProbe:
  httpGet: { path: /healthz/started, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30      # allows 5 x 30 = 150 s to start (illustrative)
livenessProbe:
  httpGet: { path: /healthz/live, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /healthz/ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

The paths are QuickCart's own names. Load balancers do the same job at the edge by probing targets (load balancing).

When a machine, a zone or a region goes away

Redundancy and failover

Redundancy means more than one of everything that can fail; failover is the switch to the replacement. Reason in failure domains: process, machine, AZ, region.

State needs more care than servers. With synchronous replication a commit is acknowledged only after the standby has it, so losing the primary's zone loses no committed order, at the price of write latency; asynchronous replication can lag (replication). Failover can also be wrong: the AWS disaster-recovery paper warns that automatic region failover "should be used with caution", because a false alarm itself causes loss, and many teams automate every step but keep a human pressing the button. Inside a region, automatic promotion is normal, but it needs agreement on who the new primary is, which is consensus.

Disaster recovery: RPO and RTO

Disaster recovery (DR) is surviving the loss of a whole site. RPO (recovery point objective) is how much recent data you can lose, as time. RTO (recovery time objective) is how long you can be down. Tighter targets cost more. The AWS whitepaper lists four patterns:

In active-passive, region B waits; in active-active, both serve, so writes from both sides need a rule: route each shopper's writes to one home region (Orders is already sharded by shopper, Stage 4) or accept last-writer-wins and its loss risk. Replication also copies corruption, so keep point-in-time backups.

QuickCart's targets, all illustrative:

Set targets per data type, and test the plan.

Agreeing in a cluster

Consensus and leader election (Raft)

Consensus is a group of nodes agreeing on a value or an order of values despite crashes and delays; leader election is agreeing who is in charge. QuickCart needs it for a small metadata store: the shard map from Stage 4, configuration, the flags that switch degradation on, and the lease naming which scheduler instance assigns couriers.

Raft, from Ongaro and Ousterhout's 2014 paper "In Search of an Understandable Consensus Algorithm", works like this at concept level.

1. Roles and terms. Nodes are followers, candidates or a leader. Time is split into numbered terms, each with at most one leader. Messages carry the sender's term, and a node that sees a higher term steps down, which is how a stale leader is recognised.
2. Election. A follower that hears no heartbeat for a randomised timeout becomes a candidate, raises the term, votes for itself and asks for votes. Each node gives one vote per term, only to a candidate whose log is at least as up to date as its own. A majority wins.
3. Log replication. The leader appends each command to its log and sends it out. Once a majority has stored it, it is committed.

Take QuickCart's three-node metadata store, one node per AZ. A majority is 2 of 3, so it survives one loss; five nodes survive two. If the leader's AZ is cut off, the other two elect a leader in a higher term and keep committing. The isolated node cannot reach a majority, so it cannot commit writes: Raft chooses consistency over availability in a partition (CAP).

The cost is that every write needs a majority round trip, so the store is small and slow compared with a database. Use it for coordination, never for orders, and run a proven system (etcd, ZooKeeper, a managed service) rather than your own. See Consensus and Leader Election.

Distributed locks and fencing tokens

A distributed lock lets one process at a time do something, such as assign courier 17 to an order. The tempting recipe is a Redis lock: SET resource_name my_random_value NX PX 30000 (set if absent, expire after 30 seconds), released only if the stored value is still yours. The Redis documentation names the limits:

• Expiry. If the holder pauses (garbage collection, a stalled VM) past the TTL, a second client takes the lock and the first wakes still believing it holds it.
• Failover. Redis replication is asynchronous, so a promoted replica may not have the key and another client can take the lock.
• Clocks. Redis does not use a monotonic clock for expiry, so a wall-clock jump can shorten a lock.

Martin Kleppmann's analysis adds the central point: a lock for efficiency (avoiding duplicate work) can tolerate these gaps; a lock for correctness needs a fencing token, a number that rises with every grant. Each write carries it, and the resource keeps the highest token seen and rejects lower ones.

Figure 5. A paused scheduler is stopped by the fencing token

How to read this: A did nothing wrong in its own view; it slept past its lease. The protection lives in the table A writes to, not in A.

In SQL the check is a conditional update (fence starts at 0), and zero rows changed means rejected:

UPDATE courier_assignments
SET courier_id = $1, fence = $2
WHERE order_id = $3 AND fence < $2;

Prefer avoiding locks. Most "exclusive" actions are a conditional write. Reserving stock needs no lock if the database decides atomically:

-- Stage 3 inventory table, with a reserved column added
UPDATE inventory
SET reserved = reserved + $3
WHERE store_id = $1 AND product_id = $2 AND stock - reserved >= $3;

One row changed means reserved; none means out of stock. Reach for a lock service only when no atomic check exists.

Clocks and ordering

Machines do not share a clock. Each physical clock drifts, and NTP corrects it only to limited accuracy, so machines disagree by some skew. Suppose Inventory (clock fast) writes "mango stock = 0" stamped 10:00:05, and a store's app (clock slow) later writes "mango stock = 40" stamped 10:00:03. A last-write-wins store keeps 0 and silently loses the newer write. Use monotonic clocks for durations, and never compare timestamps from two machines to decide what came first.

What orders events instead is causality: A is before B if A could have influenced B.

• A Lamport clock is a counter per process: increment per event, attach it to messages, and on receipt take the larger of local and received, plus one. If A happened before B, A's stamp is smaller; the reverse is not guaranteed.
• A vector clock keeps one counter per process. Comparing two vectors shows "before" or concurrent (neither knew of the other), so a store can keep two conflicting versions for merging instead of dropping one. Vectors grow with the number of writers.
• A hybrid logical clock (HLC) pairs a physical time with a small counter and never moves backwards. If B's wall clock reads 90 and it receives a message stamped (100, 0), B's clock becomes (100, 1) and its next event is (100, 2), though its wall clock still says 90. It stays near real time and respects causality, which suits QuickCart's order-event timestamps (real systems add drift bounds).

So QuickCart updates carry versions or a status rank, not bare timestamps. See IDs, Clocks and Ordering.

Gossip

Gossip shares state across a large cluster without a coordinator. Each interval, a node exchanges what it knows with a random peer, which passes it on, so news spreads like a rumour in roughly logarithmic rounds of cluster size (a classic result from the epidemic-algorithms literature).

Apache Cassandra's documentation shows it: each node gossips every second with a random peer (and sometimes a seed node), tagging state with a (generation, version) pair so old information is ignored. Failure detection is local: each node runs a variant of the Phi Accrual Failure Detector on heartbeats, and "down" is that node's own judgement, not a broadcast. The SWIM paper (Das, Gupta and Motivala, 2002) describes a related design that pings a member directly and, if that fails, asks a few others to ping it before declaring it failed.

Gossip suits membership and failure detection in big clusters, such as a courier-location store. It is eventually consistent, so do not use it where one answer is needed now, such as who the leader is.

Making repeated and reordered messages harmless

Idempotent and commutative operations

Retries, redelivered events and replicas merging in different orders all deliver messages twice or out of order. Two properties neutralise that. An operation is idempotent if doing it twice equals doing it once, and commutative if order does not matter.

Give order statuses a rank (placed, paid, picked, delivered) and apply "take the higher rank". A duplicate or late "paid" event then cannot move a delivered order backwards. Cancellation needs its own explicit rule.

A G-counter (grow-only counter) keeps one count per node and merges by taking the maximum per node. If node A holds {A: 3, B: 0} and node B holds {A: 2, B: 4}, the merge is {A: 3, B: 4}, value 7, however often or in whatever order merged. CRDTs (conflict-free replicated data types; Shapiro and others, 2011) generalise this. Where commutativity is impossible, use an idempotency key and a deduplication table (Stage 6).

Proving it works

Chaos engineering

Settings are guesses until tested. Principlesofchaos.org defines chaos engineering as experimenting on a system to build confidence that it withstands turbulent conditions. An experiment has five parts.

1. Steady state: a measurable output, such as checkout success and p99 latency.
2. Hypothesis: "with recommendations delayed 5 seconds, checkout success stays above 99.9 percent."
3. Injected event: added latency, a killed pod, a blocked path.
4. Disproof attempt: compare against a control.
5. Blast radius: start with one pod or 1 percent of traffic, with a kill switch and abort conditions.

A game day is a scheduled rehearsal that includes people: detection, paging, communication, rollback. Often the runbook or alert is what breaks. QuickCart starts in staging, lowest risk first.

The failure-mode table

Decisions and trade-offs

In practice: QuickCart at this stage

The situation

On a Saturday at 19:00 the Recommender's feature store slows after a data refresh. The cart page keeps calling it, Cart and Checkout runs out of threads, the gateway retries, and checkout success falls to about 80 percent for 25 minutes. That is 20 percent failures for 25 minutes, about 5 of the month's 43 error-budget minutes. Two weeks later a payment-provider brownout makes retries stack across three layers. The brief: survive both, plus an AZ loss, within a quarter.

Starting architecture

Figure 6. Starting architecture: shared pools, retries in three places, single points in one AZ

How to read this: amber marks a risk the incident exposed: retries at several layers, a shared pool, and single instances in one zone.

Options considered

The decision

Rollout and verification

1. Run breakers in shadow mode (log only) for a week.
2. Canary at 5 percent behind a flag, then widen.
3. Move retries last: enable the Payments budget before disabling gateway retries, or there is a moment with none.
4. Run the experiments above in staging, then a production game day for AZ-1.

Dashboards: checkout success and p99, breaker state and pool use per dependency, retry ratio (target under 10 percent), shed requests by priority, metadata-store leader changes, and replica lag.

The 2 a.m. failure

At 02:00 AZ-1 loses its network. The page reads "Orders primary unreachable, checkout success 40 percent". The scheduler in AZ-1 is partitioned, not dead.

Figure 7. AZ-1 is lost and the database fails over

How to read this: the standby already held every committed order because commits waited for it. The old scheduler wakes later with a stale token and does no damage.

First three checks: the checkout and breaker dashboard (which dependency tripped); promotion status and replica lag (did the standby take over, and was it in sync); the metadata store's leader and term (a new leader in AZ-2 or AZ-3?). Mitigation: let promotion finish, confirm the new primary accepts writes, and shed low-priority traffic while two AZs carry the load. Follow-ups: alert on stale-token rejects and test that two AZs can carry peak load. This is a realistic scenario, not a case history.

Architecture after this stage

Figure 8. QuickCart after Stage 9: what this stage added

How to read this: accent boxes are new or changed in this stage. Normal traffic follows solid arrows through Region A; the red dashed arrows are fallback paths that carry traffic only when something has failed.

What QuickCart looks like now

Architect's notes

• Draw the dependency graph first. List every call on the critical path, mark it critical or optional and record its real p99. Optional calls often sit on the critical path.
• Set RPO and RTO per data type with the business. One number for everything is too costly or too weak.
• Unknown outcome is its own state. A timed-out charge may have succeeded. Model "pending" and reconcile; never map a timeout to "failed".
• Defer active-active. Add it when a measured requirement demands it; containment and tested failover come first.

Common mistakes

• No timeout, or a huge default. Threads pile up behind a slow dependency.
• Retrying at every layer. Three layers of three retries is 64 attempts.
• Retrying non-idempotent calls. Double charges.
• A breaker with no safe fallback. It only fails faster.
• A liveness probe that checks the database. A blip restarts every pod.
• A lock without a fencing token. A paused holder corrupts data.
• Trusting timestamps across machines. Skew silently drops writes.
• Untested failover. A plan that never ran is a guess.

Key takeaways

• Slow is worse than down. A slow dependency drains callers' threads and spreads failure.
• Bound every wait. Timeouts nest inside a deadline and come from measured percentiles.
• Retries are load. Use idempotency, jitter, one retry layer and a budget.
• Contain, then degrade. Breakers and bulkheads limit spread; a ranked ladder decides what to drop, and checkout is never on it.
• Three probes, three meanings. Liveness restarts, readiness removes from traffic, startup protects slow starts.
• RPO and RTO are priced business choices. Set them per data type and test the plan.
• Consensus is for small coordination data. Majorities make leaders safe; partitions favour consistency.
• Fence your locks, or avoid them. Prefer atomic conditional writes.
• Do not trust wall clocks for order. Use versions, logical or hybrid clocks.
• Make messages idempotent or commutative. Then retries and merges are safe, and chaos experiments show it.

Practice exercises

Exercise 1: Retry arithmetic (30 min). The gateway, Checkout and Payments each allow 2 retries (3 attempts). How many provider calls does one checkout cause when the provider is down, and how many per second at 58 checkouts per second? Redesign so the worst case is 3 attempts and say which layer keeps its retries.

Exercise 2: Size a bulkhead (45 min). Use Little's law. The cart service takes 2,300 requests per second and 1 in 10 calls the Recommender at 80 ms. How many calls are in flight normally, and at 10 s? Choose a pool size, a timeout and a shopper-visible fallback.

Exercise 3: Targets (1 hour). Write an RPO, RTO and DR pattern for six QuickCart data types, including the notification queue. Justify each number from the business.

Exercise 4: Game day (half day). Plan a game day that takes AZ-1 offline: steady-state metric, hypothesis, abort conditions, who is on call, and a review template with expected, observed and action columns.

Related reading

• Stage 8: Split the Monolith, the services this stage protects.
• Stage 10: Secure and Operate, where observability and canary releases make this operable.
• Resilience Patterns, more on load shedding and hedged requests.
• Multi-Region and Disaster Recovery, deployment models and failover testing.
• Consensus and Leader Election, Raft and leases in depth.
• IDs, Clocks and Ordering, logical and hybrid clocks.
• Observability and SLOs, the signals behind breakers and alerts.

Official documentation

• In Search of an Understandable Consensus Algorithm (the Raft paper)
• Kubernetes: liveness, readiness and startup probes
• Redis: distributed locks with Redis
• Amazon Builders' Library: timeouts, retries and backoff with jitter
• Google SRE book: addressing cascading failures
• AWS whitepaper: disaster recovery options in the cloud