System Design: Multi-Region and Disaster Recovery

Read it as a cost ladder: stop at the first rung that meets the targets. Most systems end at a warm or hot active-passive design; active-active is chosen for latency and recovery speed only when the team can afford the conflict handling.

Cross-region replication

Replication across regions is mostly asynchronous. A round trip between continents costs on the order of tens to a couple of hundred milliseconds (approximately), and waiting for that on every write would make an application feel slow, so most systems accept that the secondary lags the primary.

That lag is your realistic RPO. If the primary region disappears with 3 seconds of unreplicated writes, those 3 seconds are gone unless recorded elsewhere. Teams therefore monitor replication lag as a first-class metric and alert when it approaches the RPO.

Options for tightening the gap:

• Synchronous replication to the other region: RPO near zero, but write latency includes the cross-region trip, and if the link breaks you must choose between blocking writes and dropping to asynchronous mode (which silently raises your RPO).
• Quorum or consensus databases spanning regions (see consensus and leader election): a write succeeds when a majority acknowledges. With three regions, losing one still leaves a majority: no data loss and no manual promotion, at the cost of a cross-region acknowledgement per write.
• Event logs replicated separately: replicate a stream of changes and rebuild state from it in the second region.

Do not forget everything that is not the main database: object storage buckets, search indexes, queues, secrets, configuration, DNS records and container images. A region with a perfect database copy but no secrets or images cannot start.

Steering traffic between regions

Failover only works if clients actually move. Three mechanisms, often combined:

GeoDNS and latency-based DNS. DNS answers with the closest healthy region; health checks remove a failed one. The weakness is caching: resolvers cache answers for the TTL, and some ignore low TTLs, so some users keep going to the dead region for minutes.

Anycast. Several regions advertise the same IP address through BGP and the network delivers each packet to the nearest advertiser. If a region stops advertising, traffic reroutes without clients re-resolving DNS. Global load balancers and CDNs commonly use this.

Application-level routing. A global front door, or the client app, retries against a secondary endpoint when the first fails. Precise control (for example "send this user to their home region") at the cost of client logic.

Health checks need care. Check what matters: a region whose load balancer answers but which cannot reach its database is not healthy. Beware flapping: use hysteresis (several consecutive failures to mark down, several successes to mark up) and consider making full failover a deliberate, human-approved action where false positives are costly.

The hard part: writes in two places

In active-passive there is one writer, so no conflicts. In active-active two regions can accept writes to the same data simultaneously, and on replication they discover both changed it. Four honest options:

1. Avoid conflicts by partitioning ownership. Give each record a home region; a user's data is written only there and replicated elsewhere for reads and failover. When the home region fails, ownership moves.
2. Last write wins (LWW). Keep the version with the latest timestamp. Simple and always converges, but it silently discards one write and depends on clocks that skew (see IDs, clocks and ordering). Acceptable where losing one update is harmless, such as "last seen" markers.
3. Merge with application logic or versions. Keep both versions, detect the conflict with version vectors, and let the application merge or ask the user. A shopping cart can merge by taking the union of items.
4. CRDTs. A conflict-free replicated data type is designed so concurrent updates always merge automatically, in any order, to the same result. Examples: a grow-only counter (each region increments its own slot, the total is the sum) and sets with add and remove rules. CRDTs only fit data whose semantics you can express as a merge. They do not suit invariants such as "balance never goes below zero" or "only one person gets the last seat".

For invariants, you need a single decider: route those operations to one region (or one consensus group) and accept the latency, keeping everything else local.

Backup tiers: the last line of defence

Replication is not backup. If someone deletes a table or a bug corrupts data, replication copies the damage to every region within seconds. You also need point-in-time recovery and independent backups.

A widely used guideline is the 3-2-1 rule: at least three copies, on two kinds of storage, with one offsite. In the cloud that means frequent same-region snapshots for fast restores, copies in a second region for disaster recovery, and long-term archives in cheaper tiers with immutability (write-once retention) so ransomware or a compromised account cannot erase them, stored under separate credentials from production. Tiers differ in how quickly you can use them: hot (replicas and recent snapshots, minutes), warm (daily backups in standard object storage, minutes to hours), cold or archive (long retention, hours, cheapest).

Testing failover

Real disasters are chaotic, so rehearse in layers: tabletop exercises (who decides to fail over, who talks to customers?), component drills (promote a replica in staging and time it), a full regional failover game day (move real traffic on a planned date and, importantly, fail back, which is often harder because the original region holds stale data), and continuous verification of replication lag, standby capacity and configuration drift.

Measure against your targets: how long did detection take, how long did promotion take, how much data was lost, which steps were manual? Each manual step in a runbook is a risk at 3 a.m. Two hidden dependencies recur. Capacity: if region A serves 60 percent of traffic and B 40 percent, B must carry 100 percent or you plan degradation, such as shedding low-priority work as in resilience patterns. Shared control planes: if the failover tooling lives in the failing region, you cannot press the button.

In practice: a SaaS with contractual RPO and RTO and a regional failover drill

The situation

A B2B project-management SaaS has about 400 business customers, mostly in North America and Europe, and a platform team of seven. Enterprise contracts promise an RPO of 5 minutes and an RTO of 60 minutes for a regional outage, with service credits if missed. The stack: Kubernetes in one cloud region, PostgreSQL with an asynchronous replica in a second region, S3-compatible object storage with cross-region replication, Redis for sessions, Kafka for events, Terraform for infrastructure, and Prometheus and Grafana for monitoring. The last failover drill took 3 hours 10 minutes, mostly manual steps; replica lag has a p99 of about 40 seconds but reached 11 minutes during a bulk customer import. The auditors want evidence that the contract is achievable. Budget allows roughly 40 percent extra infrastructure spend (an assumption) and no active-active rewrite this year.

Starting architecture

Options considered

The decision

Rollout and verification

Make lag and drift visible first: alert when replication lag exceeds 2 minutes (40 percent of the RPO), and run a daily check comparing required secrets, images and configuration between regions. Bring region B to Terraform parity, then script promotion and time it in staging. In production, drain a small group of tenants, promote, run real traffic there, and fail back. Success criteria written before the drill: measured RTO under 45 minutes, data loss equal to the lag recorded at the cut, error rate under 1 percent after the DNS TTL passes, zero manual steps outside the runbook. Dashboards: replication lag, standby capacity versus production load, drift check result, backup restore time.

The 2 a.m. failure

During the full drill, promotion completes in 12 minutes but the page reads "Region B readiness probes failing, 0 of 6 app pods ready". The cause is a payment-webhook signing secret that exists in region A only, which the app's startup readiness check requires; the secret was added after the replication job was configured, and the daily check had not yet covered that namespace.

First three checks: kubectl get pods, then kubectl describe pod and kubectl logs for one unready pod in region B (the probe failure shows in events, the cause in the logs); the external-secrets sync status to see what did not replicate; and a diff of the secret inventory between regions. Mitigation: restore the missing secret from the break-glass vault, then shift traffic. Follow-up fix: make the secret inventory a required, automatically compared list in CI, extend the daily drift check to every namespace, and add "read events and logs of the first unready pod" as step one of the promotion runbook. Record the actual RTO from the drill, not the planned one.

Architect's notes

• Measure lag distribution, not the average. p99 and the worst case during batch work are your real RPO; throttle the workload that causes spikes.
• Ask who can declare a disaster. A named role with authority and a decision time limit saves more minutes than any script.
• The cost trap is idle capacity and cross-region traffic. Replication egress and a standby sized for full load are the recurring line items; state them before the drill, not after the invoice.
• Fail-back is a project. Plan the data reconciliation and a second drill; a one-way failover leaves you running without a standby.
• Keep the control plane outside the blast radius. Runbooks, pipelines, secrets and the DNS account must be reachable when region A is not.
• Defer active-active. Take it only when a contract or latency target forces it, and pair it with an explicit per-entity conflict policy.

Common mistakes

• Treating zones as regions. Multiple availability zones do not protect against a regional failure. Decide explicitly which failures you design for.
• Choosing RPO and RTO without the business. Targets set by engineers alone are too expensive or too weak. Agree them per system.
• Assuming replication is backup. Replication copies deletes and corruption. Keep point-in-time recovery and immutable backups.
• Active-active without a conflict strategy. Concurrent writes will collide. Decide ownership or merge rules up front.
• Never testing failover or restore. Untested plans fail on details such as expired credentials or missing images.
• Forgetting non-database state. Secrets, queues, object storage, search indexes and DNS must exist in the second region too.
• Underprovisioning the standby. Size it, or plan explicit degradation.

Key takeaways

• RPO is data you can lose, RTO is time you can be down. Both come from business impact, per system.
• Regions, not zones, protect against regional loss. Pick the failure you are designing for.
• Cost rises as targets fall. Backup and restore, warm standby, active-passive and active-active form a ladder.
• Asynchronous replication means a non-zero RPO. Monitor lag and treat it as your real recovery point.
• DNS failover is slowed by caching; anycast and global load balancers react faster. Combine mechanisms and use sensible health checks.
• Active-active needs a write strategy. Home regions, merges, CRDTs or a single decider; last write wins loses data silently.
• Replication is not backup. Keep point-in-time recovery, the 3-2-1 idea and immutable copies.
• Test, including fail-back. Drills turn a document into a capability.

Practice exercises

Exercise 1 — Starter (30 min): A service has an RPO of 1 minute and an RTO of 15 minutes. For each deployment model in this lesson (backup and restore, warm standby, active-passive, active-active), say whether it can plausibly meet both targets and why. State one assumption you made about replication lag or startup time for each.

Exercise 2 — Intermediate (2 hours): Design a two-region deployment for a note-taking app with 5 million users in Europe and North America. Draw the architecture including database, object storage, queue and DNS. Choose active-passive or active-active and justify it. Decide how conflicts are avoided or resolved when a user edits the same note from two devices in two regions, and write the failover steps as a runbook of at most ten numbered steps.

Exercise 3 — Advanced (half day): Plan a regional failover game day for a system you know or an invented e-commerce platform. Define success criteria (measured RTO, RPO, error rate), list all state that must exist in the second region, specify the traffic-shift procedure and how health-check flapping is prevented, prepare the fail-back plan including data reconciliation, and write a review template for what you would measure and what actions you would expect to come out of it.

Related reading

• System Design: Replication — leaders, followers, lag and failover within and across regions.
• System Design: CAP and Consistency — the trade-off you face when regions cannot talk to each other.
• System Design: Resilience Patterns — containing failures before they need a regional failover.
• System Design: IDs, Clocks and Ordering — why timestamp-based conflict resolution is risky.
• System Design: Capacity and Cost — the price of running a second region and how to size it.