Learn what the CAP theorem really says, how PACELC extends it, and how to choose the right consistency model, quorum settings and session guarantees for each feature.
What you'll learn: By the end of this lesson you will be able to state the CAP theorem precisely and spot common misstatements, explain PACELC and why it matters even when nothing is broken, tell linearizable, sequential, causal and eventual consistency apart with everyday examples, and pick a consistency level for each feature of a product instead of for the whole system.
Imagine a bank with two branches in different cities. Each branch keeps its own paper ledger, and normally they phone each other after every transaction. One day the phone line is cut. A customer walks into branch A and asks to withdraw cash. The clerk can refuse until the line is back, which keeps the books correct but turns the customer away. Or she can hand over the cash and sort out any conflict later, which keeps the customer happy but risks the two ledgers disagreeing.
That small story is the whole CAP theorem. A distributed system is a set of computers that must sometimes lose contact with each other. When that happens you cannot have both "always answer" and "always answer correctly". The theorem is famous and often misquoted, so this lesson states it carefully, looks at what happens when nothing is broken (most of the time), and builds a vocabulary of consistency models you can apply feature by feature. If replication is new to you, read the replication lesson first, because everything here is about what replicas are allowed to show you.
The core idea
Consistency is a promise about what a reader may see after a write. Strong promises are easy to reason about but cost latency and availability. Weak promises are fast and resilient but push complexity into the application. Good design chooses the weakest promise that still keeps each feature correct.
CAP tells you what you give up during a failure, PACELC tells you what you give up the rest of the time, and the consistency models are the menu of promises.
CAP stated precisely
CAP is about three properties of a replicated data store:
Consistency (C) means linearizability: every read sees the most recent completed write, as if there were a single copy. This is much stronger than the "C" in ACID, which is about keeping database invariants true.
Availability (A) means every request sent to a non-failing node gets a non-error response, without a guarantee that it contains the latest write. It does not mean "the service is mostly up".
Partition tolerance (P) means the system keeps working when the network drops or delays messages between nodes, leaving nodes in groups that cannot talk to each other.
The precise statement: when a network partition happens, a system must choose between consistency and availability. "Pick any two of three" is misleading, because partitions are not optional. Networks drop packets, switches fail, and a long garbage-collection pause can look like a partition to everyone else. So the real question is never "CA, CP or AP?" but: given that a partition will happen, what should each operation do while it lasts?
flowchart TD
P["Network partition occurs"] --> Q{"Node receives a request"}
Q -->|"refuse or wait"| CP["CP behavior: stay correct, some requests fail"]
Q -->|"answer from local data"| AP["AP behavior: stay available, may be stale"]
CP --> H1["After healing: minority side catches up, nothing to merge"]
AP --> H2["After healing: replicas reconcile conflicting writes"]
Read it top to bottom: the partition is the trigger, the request is where you choose, and the bottom row is the bill that arrives when the network heals. In a three-node store with one node cut off, a CP-leaning store keeps serving on the two-node majority and makes the isolated node refuse requests. An AP-leaning store lets every node keep accepting reads and writes, so the isolated node may return old data and accept conflicting writes.
NOTE
CAP says nothing about normal operation, latency or durability. A product is not "an AP database" in all situations; many systems let you choose per operation through read and write consistency settings.
PACELC: the rest of the story
Partitions are rare, yet you still face a trade-off when healthy, because keeping replicas in step costs time. PACELC: if there is a Partition, choose between Availability and Consistency; Else, choose between Latency and Consistency.
The "else" half shapes everyday performance. If a write must be acknowledged by a replica in another region before it returns, every write pays a cross-region round trip (roughly tens to over a hundred milliseconds, depending on distance). If it returns after the local node alone has stored it, it is fast, but a reader elsewhere might not see it yet.
Style
During a partition
When healthy
Typical shape
PC/EC
Consistent, may refuse requests
Pays latency for consistency
Consensus or synchronous replication
PA/EL
Available, may diverge
Low latency, eventual convergence
Leaderless or asynchronously replicated stores
PC/EL
Consistent in a partition
Favors latency otherwise
Single leader, local reads in normal times
Treat these rows as tendencies, not labels for products. Many databases are configurable per query.
The consistency models: a menu of promises
Ordered from strongest to weakest, each model forbids more surprising outcomes than the next.
Linearizable (strong). Every operation appears to take effect at one instant between its start and end, and all clients agree on that order. Once a write completes, every later read from any client sees it. Think of booking the last seat: when Alice's booking of 12A returns success, Bob must not be able to book 12A a millisecond later. Use it for unique usernames, balances, inventory that must not oversell, and leader election. The cost is coordination on the critical path: higher latency, and refusal to serve on the minority side of a partition.
Sequential. All clients see operations in the same order and each client's own operations keep their issue order, but that order need not match real time. It is mostly a concept; few product decisions turn on the difference from linearizability.
Causal. If one operation could have influenced another, everyone sees them in that order; unrelated operations may appear in different orders to different people. A reply must never appear before the post it answers, but two strangers' posts may show in either order. Causal consistency can be provided without blocking during a partition, at the cost of tracking dependencies (see the IDs, clocks and ordering lesson).
Eventual. If writes stop, all replicas converge. There is no promise about when, and meanwhile readers can see old values. A like counter that updates two seconds late harms nobody, so this is the right default for data where small staleness is invisible.
Model
Promise
Everyday example
Cost
Linearizable
Reads see the latest completed write, globally
Booking the last seat
Latency; unavailable on the minority side
Sequential
One agreed order, per-client order kept
Shared log viewed by all
Still needs coordination
Causal
Cause is seen before effect
Reply never before the post
Dependency tracking
Eventual
Replicas converge if writes stop
Like and view counts
Stale reads; conflicts to resolve
Quorums: tuning consistency with numbers
Leaderless stores let you tune consistency per request with quorums. Data lives on N replicas; a write succeeds when W acknowledge; a read asks R replicas and takes the newest version. If R + W > N, every read quorum overlaps every write quorum in at least one replica. For illustration, with N = 3: W=1,R=3 gives fast writes and slow reads; W=2,R=2 is balanced and tolerates one failed replica; W=3,R=1 makes one down replica block all writes; W=1,R=1 has no overlap guarantee, so reads can be stale.
sequenceDiagram
participant C as Client
participant A as Replica A
participant B as Replica B
participant D as Replica C
C->>A: write v2
C->>B: write v2
A-->>C: ack
B-->>C: ack
Note over C: W=2 reached, write succeeds
C->>B: read
C->>D: read
B-->>C: v2
D-->>C: v1
Note over C: R=2, newest version v2 wins
The stale replica (C) answered, but because the read also touched a replica that saw the write, the client could pick the newest version.
WARNING
R + W > N is necessary for this intuition, not a guarantee of linearizability. Concurrent writes, sloppy quorums (writes landing on stand-in nodes during failures), clock-based conflict resolution and partially failed writes can still produce stale reads. Systems that need linearizability typically use a consensus protocol (consensus lesson).
Session guarantees: the practical middle ground
Between "strong everywhere" and "eventual everywhere" sit promises made to one user's session:
Read-your-writes: after I update my profile, I see my update even if others lag.
Monotonic reads: once I have seen version 5, I never see version 4 afterwards.
Monotonic writes: my writes apply in the order I made them.
Writes-follow-reads: my reply is ordered after the post I read.
They are cheap compared to global strong consistency and fix the anomalies users actually notice. Typical implementations: route a user's reads to the leader for a short window after they write, pin a session to one replica, or attach a version token to the session and refuse to read from a replica that is behind it.
Choosing consistency per feature
The mistake is picking one global level. For each feature ask: what is the worst thing that happens if a user sees stale or conflicting data here, and for how long?
flowchart TD
Start["Feature needs a consistency level"] --> Q1{"Does staleness cost money or break a uniqueness rule?"}
Q1 -->|"yes"| Lin["Linearizable on one authoritative key or leader"]
Q1 -->|"no"| Q2{"Would a user notice their own write missing?"}
Q2 -->|"yes"| RYW["Eventual plus read-your-writes"]
Q2 -->|"no"| Q3{"Would out-of-order cause and effect confuse people?"}
Q3 -->|"yes"| Causal["Causal, scoped per conversation or entity"]
Q3 -->|"no"| Ev["Eventual with a short cache TTL"]
Start at the top and stop at the first yes. Balances and unique usernames land on linearizable; a profile edit lands on read-your-writes; chat threads on causal per conversation; likes and catalogue prices on eventual.
TIP
Start strong for the small amount of data that guards money, uniqueness and safety, and relax everything else. It is far easier to keep a small strongly consistent core than to bolt strong guarantees onto an eventually consistent system later.
Conflicts: what AP systems must do after healing
If you allow divergent writes you must merge them. Last write wins keeps the latest timestamp: simple, but it silently discards one of two concurrent writes, and clock skew makes "latest" unreliable. Siblings keep both versions for the application or user to merge. Merge by construction uses CRDTs, data types that merge without conflict (the multi-region lesson introduces them). Avoid conflicts by routing all writes for a key to one place. "Whichever write arrives last" is a decision too, just an unexamined one.
In practice: seat reservation across two regions
NOTE
A realistic scenario, not a case history. This is a composite example built from common production patterns. It does not describe any specific company, and every number in it is an illustrative assumption.
The situation
A live-events company sells reserved seats for concerts and sports in two regions, eu and us. The platform team is eight engineers. Contracts require 99.9 percent availability for browsing and selling, and the business rule is absolute: a seat must never be sold twice. The stack is PostgreSQL, Redis, Kafka, Nginx and Kubernetes in both regions, provisioned with Terraform and watched with Prometheus and Grafana.
The pain: to cut latency for European buyers, an earlier design wrote seat state to a local PostgreSQL in each region and replicated asynchronously to the other. On a big on-sale, assume about 0.4 percent of seats (around 120 in a 30,000-seat venue) were sold in both regions within the same replication window. Support had to refund and apologise. Browsing latency was fine; correctness was not.
Starting architecture
Both sides accept writes to the same seats, so two buyers can both "win" the seat before either write arrives on the other side. This is the PA/EL end of PACELC, applied to a feature that needed the PC end.
Options considered
Option
What it fixes
What it costs
Risk
A. Keep active-active, add last-write-wins plus nightly reconcile
Nothing in real time; refunds become routine
Cheap
Customer-visible double sales continue
B. One home region per event, writes forwarded there; seat map reads served locally
No double sale by construction; cheap reads
Remote buyers pay a cross-region write; home region outage blocks its events
Failover for home regions needs a runbook
C. Globally consistent distributed SQL across both regions
Linearizable seats everywhere
Every write pays consensus across regions; new database to operate
Team has no operating experience; migration cost
The decision
ADR field
Content
Context
Double sales come from two writable copies of the same seat. Browsing tolerates seconds of staleness; selling does not.
Decision
Option B. Each event has a home region. All holds and sales for it go to its PostgreSQL primary, using a conditional update on the seat row. The seat map is served from local replicas and Redis.
Consequences
Good: a seat has one authority, so linearizable for sale. Bad: a European buyer of a US event pays a cross-region round trip on hold and purchase; if the link or home region is down, those events cannot sell.
Revisit when
Cross-region hold latency p95 exceeds the checkout budget for more than 5 percent of sessions, or a single event needs sales from both regions at a rate one primary cannot absorb.
stateDiagram-v2
[*] --> Available
Available --> Held: hold request wins the row
Held --> Available: hold expires
Held --> Sold: payment confirmed
Sold --> [*]
A seat's life has only three states, and the transitions that matter (hold and sell) all execute at the home region.
Rollout and verification
The team would ship in four steps. First, add an event_home_region column and route all new events by it behind a feature flag. Second, run a dual-read period: the new path serves traffic while the old replicated state is compared for disagreement. Third, canary one low-demand event per week, then a mid-size one. Fourth, load test a full on-sale at 2x forecast against the home-region primary, including cross-region callers.
sequenceDiagram
participant B as EU buyer
participant E as EU API
participant U as US API
participant P as US primary
B->>E: hold seat 12A for US event
E->>U: forward hold with idempotency key
U->>P: UPDATE seat SET state held WHERE state available
P-->>U: 1 row updated
U-->>E: hold granted for 5 minutes
E-->>B: seat held
Proof it worked: a daily query counting seats with more than one confirmed order must return zero; dashboards track hold success rate, cross-region forward latency p95 and p99, replication lag per replica, and refunds for oversold seats as a business metric with a target of zero.
The 2 a.m. failure
The page reads: "Booking forward errors above 5 percent for us-homed events". The inter-region link is degraded, so EU buyers see hold failures, while US buyers are fine. Detection comes from the forward_error_total and forward_latency_seconds metrics plus link probes. First three checks: the link probe dashboard, kubectl get pods and logs for the forwarding client's timeouts, and pg_stat_replication on the us primary to confirm replicas are lagging, not diverging.
sequenceDiagram
participant B as EU buyer
participant E as EU API
participant U as US API
B->>E: hold seat for US event
E-xU: forward times out
E-->>B: seat map still readable, hold refused, try again or join waitlist
The mitigation is deliberate: keep the seat map readable from local replicas, return a clear "try again" for holds, and never fall back to writing locally, because that would resurrect the double sale. Follow-up fixes: put a circuit breaker and a short timeout on the forward path so threads are not exhausted, and consider homing events by audience geography.
Architect's notes
Ask per operation, not per system. In design review, make the team write down, for each write and each read, what it does during a partition. A table with one row per feature beats the sentence "we are AP".
Measure the cross-region round trip before you promise anything. The PACELC "else" cost is a number you can read from a ping and a staging write; do it before choosing synchronous cross-region commits.
Give each piece of data one home. Most "we need global consistency" requirements dissolve into "this key needs one authoritative writer".
Know who gets paged. A home-region design means the on-call for region us owns events sold to Europeans. Decide that ownership before the incident.
Defer distributed SQL. It is a reasonable later move if one primary per event stops scaling, but adopting it first buys operational risk for a problem a routing rule already solves.
Watch the invariant, not just the infrastructure. An automated check for "seat sold twice" is the real health metric; replication lag is only a hint.
Common mistakes
Saying "we chose CA". Partitions are a fact of networks. Decide what happens during one for each operation and write it down.
Treating CAP's "C" as ACID's "C". CAP consistency is linearizability across replicas; ACID consistency is about database invariants.
Labelling a whole product "AP" or "CP". Behavior depends on configuration and per-query settings; read the docs for the modes you use.
Assuming R + W > N means strong consistency. Sloppy quorums and concurrent writes can still return stale reads.
Ignoring the "else" in PACELC. Strong consistency costs latency even with no failures; measure it first.
Using eventual consistency where users notice. Without read-your-writes, a user who saves and sees the old value saves again.
Last-write-wins by default. It quietly loses data under concurrent updates; choose merge behavior intentionally.
Key takeaways
Partitions are inevitable. CAP's real question is what to do during one: stay consistent and refuse some requests, or stay available and risk staleness and conflicts.
CAP is narrow. It covers only partitions and uses strict definitions of consistency (linearizability) and availability.
PACELC covers normal times. Without a partition you still trade latency against consistency.
Consistency is a menu. Linearizable, sequential, causal and eventual forbid progressively fewer anomalies in exchange for speed and availability.
Quorums are tunable, not magical. R + W > N guarantees overlap, not linearizability.
Session guarantees fix what users notice. Read-your-writes and monotonic reads are cheap and prevent confusing anomalies.
Choose per feature. Keep a small strongly consistent core for money and uniqueness, and relax elsewhere.
Conflicts need a policy. Decide how divergent writes merge, or route writes so they cannot diverge.
Practice exercises
Exercise 1 — Starter (30 min): Take a food-delivery app and list ten features (place order, driver location, menu, ratings, wallet balance, promo code redemption, and so on). For each, write the worst consequence of a stale read and assign one of: linearizable, causal, eventual, or eventual with read-your-writes.
Exercise 2 — Intermediate (2 hours): For a store with N = 5 replicas, build a table of every pair (W, R) from 1 to 5. Mark which pairs satisfy R + W > N, and for each state how many replica failures reads and writes can tolerate. Then identify two scenarios where a satisfying pair can still return a stale value (hint: concurrent writes and sloppy quorums).
Exercise 3 — Advanced (half day): Redesign the seat-reservation scenario for a third region and for events that sell equally in all regions. Decide the home-region rule, what the seat map shows during a partition, how holds expire safely if the home region is lost, and what you would test in a failover drill. Draw the write path and the reconcile path with Mermaid or on paper.
Related reading
System Design: Replication - the mechanics of leaders, followers and lag that these consistency models describe.
Designing Reliable AI Systems - practical failure handling that complements the availability trade-offs in this lesson.
Enroll to keep reading
Lesson 4 of 14 · System Design Building Blocks: Data, Messaging and Scale
Learn what the CAP theorem really says, how PACELC extends it, and how to choose the right consistency model, quorum settings and session guarantees for each feature.
In the rest of this lesson
Quorums: tuning consistency with numbers
Session guarantees: the practical middle ground
Choosing consistency per feature
Conflicts: what AP systems must do after healing
In practice: seat reservation across two regions
Architect's notes
Enrolling is free. It unlocks every lesson in System Design Building Blocks: Data, Messaging and Scale, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.