What you'll learn: By the end of this lesson you will be able to say when accepting writes in more than one place is worth its cost, explain how a conflict between two writes is detected and the five common ways to resolve it (and what last-write-wins loses), compare leader topologies, describe how a leaderless store such as Dynamo or Cassandra converges after a failure, and choose a design for ShopStream data that must be edited from several places.
Two branch offices each keep a paper ledger and phone their entries to each other every evening. Usually that works, because the offices record different customers. Then both amend the same customer's address on the same afternoon. Someone has to decide which is right, and "whoever phoned last" is a rule that quietly throws away one clerk's work.
That is multi-leader replication in one picture. The replication lesson introduced it briefly. This lesson goes deeper on when to accept writes in several places, how conflicts are found and resolved, and how a leaderless store repairs itself. The quorum arithmetic, hinted handoff and read repair are taught in quorum reads and writes, so they appear here only as far as the conflict story needs.
Verified on 2026-11-19 against the Dynamo paper, the Apache Cassandra, Riak KV and Apache CouchDB documentation, the Amazon DynamoDB global tables page and the PostgreSQL 18 manual. No code on this page was run.
The core idea
A single leader gives every write one place to be ordered. Multi-leader and leaderless designs give that up so that writes can be accepted in more than one place, and the bill is concurrent writes to the same item. Everything below is a way of paying that bill: detect the concurrency, decide a winner or a merge, and make every copy converge on the result.
Three illustrations run through the lesson, each labelled:
- Minimal example: two regional leaders both change one product price; the later timestamp wins and one edit disappears.
- Realistic example: ShopStream store tablets editing the catalogue while offline.
- Failure example: a tablet with a wrong clock wins every conflict.
Multi-leader: where it earns its cost
In multi-leader replication, several nodes accept writes, and each forwards its changes to the others. It is a poor way to "scale writes" for one record, because every leader still applies every write. It is justified by three situations:
- Multiple datacentres or regions. A leader per region lets users write to a nearby node, and a regional outage does not stop the other regions. Amazon's DynamoDB global tables are a managed form: the documentation says that when an application writes to a replica in one Region, DynamoDB replicates the write to all the others.
- Offline clients. A phone or a store tablet that must keep working without a connection is, in effect, a leader with one user. CouchDB's conflict documentation is written for this peer-to-peer style, and describes how every peer picks the same winner.
- Collaborative editing. Several people edit one document and each sees their own edit at once; merging is the territory of CRDTs and conflict resolution.
Figure 1. Multi-leader across two regions. How to read this: each region writes to its own leader, so writes stay fast and survive the loss of the other region. The dashed line between the leaders is the price: it carries changes in both directions, and nothing stops the same row changing on both sides before the changes arrive.
Write conflicts: detection and resolution
A write conflict is two writes to the same item that neither saw the other. As logical and vector clocks explains, two writes are concurrent when neither happened before the other, and a version vector lets a store tell "this version replaces that one" from "these two are independent".
Minimal example (illustrative). A product's price is 3.20. Region A sets it to 3.50 and region B sets it to 3.40, a few hundred milliseconds apart, before either change has crossed the network.
Figure 2. A conflict resolved by last write wins. How to read this: both writers got "ok" from their own leader. Both leaders end at 3.40. The 3.50 edit was accepted, acknowledged and then silently dropped.
Detecting a conflict
A store can only call something a conflict if it can compare versions. Four mechanisms appear in the documentation of real systems:
- Version vectors or causal context. Riak tags each stored value with "a piece of metadata called a causal context", which it uses to decide whether values are related; when it cannot, it keeps both as siblings. The Dynamo paper describes the same idea with vector clocks and calls the automatic case syntactic reconciliation: a version that causally follows another simply replaces it. A conflict is what is left over.
- Revision histories. CouchDB's conflict page says that a normal read hides conflicts and that asking for them returns "the revs of the other, conflicting revision(s)". The losing revisions are kept, "hidden away as a conflicting revision".
- Timestamps only. Cassandra and DynamoDB global tables skip detection: each compares timestamps and keeps the larger. There is no "conflict" to report, only a winner.
- Constraint violations. In PostgreSQL logical replication, a change that violates a unique constraint on the receiving side is detected as a conflict, and the page says that such a conflict "will stop the replication; it must be resolved manually". A missing row to update or delete is skipped, not an error.
Resolving it: five common policies
| Policy | How it works | Cost | Example in the documentation |
|---|---|---|---|
| Last write wins (LWW) | Keep the version with the larger timestamp | Drops a concurrent write; depends on clocks | DynamoDB global tables use "the latest internal timestamp" per item; Cassandra: "Updates resolve according to the conflict resolution rule of last write wins" |
| Keep both, let the application merge | Return all versions (siblings); the application writes back one | The application must do it correctly | Riak siblings; the Dynamo paper's shopping cart merge |
| Deterministic winner plus visible losers | Every peer picks the same winner; losers stay readable | Someone must notice and merge | CouchDB |
| Merge by construction (CRDT) | Data types whose merge is defined, so no conflict arises | Restricted data types, metadata growth | See CRDTs |
| Avoid the conflict | Route all writes for an item to one home region | Cross-region writes for that item pay a round trip | Common design practice; no single source |
Two quotes explain why LWW needs care. Riak's documentation says it "will necessarily drop some writes in the case of concurrent updates". Cassandra's warns that its "correctness does depend on these clocks". The Dynamo paper adds a design view: if the data store resolves conflicts, "its choices are rather limited", while the application, knowing the schema, can choose a merge that suits the user. Dynamo's session service used timestamp reconciliation; its shopping cart merged versions.
to one home region?"} Q1 -->|yes| R1["Avoid the conflict:
route to the home region"] Q1 -->|no| Q2{"Is losing one of two
concurrent edits acceptable?"} Q2 -->|yes| R2["Last write wins,
with trusted clocks"] Q2 -->|no| Q3{"Is there a defined way
to combine two values?"} Q3 -->|yes| R3["Merge or CRDT"] Q3 -->|no| R4["Keep both versions,
application or person decides"]
Figure 3. A first-pass conflict policy. How to read this: ask the questions in order. Most data should exit at the first one; the lower exits cost more code and more operations.
Topologies: who forwards to whom
With more than two leaders you must choose how changes travel: all-to-all (every leader sends to every other), circular (each forwards to the next) or star (through a hub). A ring or star has fewer links, but a failed node in the path stops changes passing through it; all-to-all has no single path to lose, but a change can reach one leader before an earlier change it depends on, so causal ordering needs care.
Figure 4. A ring of leaders. How to read this: a change made at A travels A to B to C. A common way to stop it going round forever is for each change to carry the identity of the node where it began, so that a node never forwards it back to its origin.
PostgreSQL shows the loop concern in practice: in CREATE SUBSCRIPTION, origin = none asks the publisher "to only send changes that don't have an origin", while the default any sends changes regardless of origin. The page warns that with copy_data = true and origin = none the true origin of the initial rows cannot be known.
Leaderless replication
In a leaderless design no node is special. A client or coordinator sends each write to several replicas and reads from several. The Dynamo paper describes its goal as an "always writeable" store: it deliberately moves the work of conflict resolution to the read path so that writes are never rejected.
The N, R and W arithmetic, sloppy quorums, hinted handoff and read repair are in the quorum lesson. What matters here is how a leaderless store repairs divergence, as the documentation describes it:
- On the write path: Cassandra's documentation says writes are "always sent to all replicas, regardless of consistency level", and lists hinted handoff as a best-effort convergence technique. In Dynamo, a node holding a hinted replica keeps it separately and delivers it when the intended node recovers; the paper notes hinted handoff "works best if the system membership churn is low and node failures are transient".
- On the read path: the coordinator may gather several versions; Dynamo returns "all the versions it deems to be causally unrelated" and the reconciled version is written back. Cassandra lists replica read repair as best-effort.
- In the background: anti-entropy repair compares hash trees. Dynamo's Merkle trees let two replicas compare roots and descend only into differing branches, which "reduce the number of disk reads performed during the anti-entropy process".
- Bounding version metadata: Dynamo truncates a vector clock past a threshold (the paper says "say 10"), dropping the oldest pair, which can blur which version descends from which.
Across datacentres, Cassandra uses NetworkTopologyStrategy with a replication factor per datacentre, and offers LOCAL_QUORUM (a majority in the coordinator's datacentre) and EACH_QUORUM (a majority in every datacentre). This is the leaderless answer to the multi-region question: a client writes at local latency and the other datacentre catches up.
Figure 5. A write at a local-quorum consistency level. How to read this: the coordinator sends the write to every replica but waits only for a majority in its own datacentre, so a read in the remote one may briefly return the older value.
Trade-offs
| Design | Gain | Cost | Wrong when |
|---|---|---|---|
| Single leader with followers | One order of writes, no conflicts | Remote writers pay a round trip; failover | Writers must work offline or in several regions |
| Multi-leader | Local writes, survives a region loss, offline work | Conflicts, weak constraints, harder debugging | One home region per item is possible |
| Leaderless | No failover step, always-writeable | Stale reads, conflict handling, clock dependence | You need a single agreed order |
| Last write wins | No application code | Silently drops writes; trusts clocks | Every edit matters |
| Siblings or merge | No lost edits | The application must resolve correctly | Nobody owns the merge code |
| Home-region routing | No conflicts by design | A round trip for non-local writes | Users move between regions mid-session |
Failure example: the clock that always wins
Under LWW, timestamps decide. If one tablet's clock is two days ahead, its edits beat everyone else's for two days. The damage is silent: no error, only the wrong price. A second failure is the PostgreSQL case above: a unique-constraint conflict stops replication until a person resolves it.
In practice: offline catalogue edits across 5,000 stores
This is an educational scenario on ShopStream with illustrative numbers.
This is a realistic scenario, not a case history.
The situation
This is the realistic example. ShopStream has 5,000 stores (the shared figure, illustrative). Each keeps a local catalogue on a tablet so staff can mark stock or set a shelf price override even when the connection drops. Each tablet is a leader for its store's overrides and a regional service is the other leader; orders and payments stay out, because they have one writer. Staff report that stock marks made during a drop vanish after reconnecting, and head office wants to push promotional prices that stores can still override.
Starting architecture
Every edit carries the tablet's wall-clock time, and the sync service keeps the larger timestamp per product.
Options considered
| Option | Gain | Cost | Wrong when |
|---|---|---|---|
| A. Online-only edits | No conflicts | Tablets stop working in a drop | Connections are unreliable |
| B. Keep LWW, add clock checks | Small change | Still drops concurrent edits | Edits must not be lost |
| C. Operations for stock, versioned price with a review queue | No lost stock marks; price conflicts become visible | Sync service and a review screen to build | The team cannot own the merge code |
The decision
Option C. A price override is an absolute value, so concurrent edits are a true conflict; a stock movement ("received 20 units") is better recorded as an operation that is added, as in idempotent and commutative operations. Stock movements sync as operations. Price overrides carry a version vector per product; concurrent price edits are both kept and shown to a store manager. Head office promotional prices live in a separate field so they never conflict with local overrides. Timestamps from tablets are never used to order edits.
Rollout and verification
Run the new sync beside the old one for two weeks and compare outcomes per product. Sample the flagged conflicts by hand. Check that a reconnecting tablet applies its edits exactly once.
The 2 a.m. failure
The page reads: "Checkout price differs from shelf price in 40 stores". A tablet's clock was reset and one store's overrides now win.
First checks: list changes whose timestamp is later than the server's receive time, and compare the clock offset reported at sync. Mitigation: reject client timestamps ahead of the server and re-apply the head office price. Follow-up: the server assigns the order; the tablet clock is no longer an input.
Architect's notes
- Ask first whether writes in two places are needed. A read replica near users solves most locality needs without conflicts.
- Name the policy per data type, not per system.
- Never let a client clock order writes. Use server-assigned versions or version vectors.
- Rehearse a partition: write the same item on both sides, heal, and check the outcome matches the policy.
- Keep constraints single-writer: unique values and non-negative counters do not survive multiple leaders without extra design.
Common mistakes
- Adopting multi-leader to scale writes. It adds conflicts without adding capacity for one item.
- Last write wins by default, with no review of what is lost.
- Trusting wall clocks from devices you do not control.
- Forgetting loops in a ring or all-to-all setup.
- Skipping anti-entropy and assuming hints and read repair converge everything.
Key takeaways
- Multi-leader and leaderless designs accept writes in several places and pay with concurrent writes.
- Conflicts are detected with version metadata, revision histories or constraint checks; timestamp-only systems never see one, they only pick a winner.
- Last write wins is cheap, drops concurrent writes and trusts clocks.
- The alternatives are siblings and application merge, a deterministic winner with visible losers, CRDTs, or one home region per item.
- Leaderless stores repair divergence with hints, read repair and hash-tree comparison.
- Choose a policy per data type; keep orders, payments and unique constraints single-writer.
Practice exercises
Exercise 1 (30 min). Two leaders accept edits to one record. Draw the timeline, apply last write wins, then a version vector, and state which edit survives in each case.
Exercise 2 (1 hour). For ShopStream, list five fields a store tablet can edit. For each, choose a policy from the table and justify it in one sentence.
Related reading
- Replication: leader-follower, lag and failover, the lesson this one builds on.
- Quorum reads and writes: R + W > N, sloppy quorums, hinted handoff and read repair in depth.
- Logical and vector clocks: how concurrency is detected.
- CRDTs and conflict resolution: merging without a coordinator.
Official documentation
- DeCandia et al., Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007): sections 2.3, 4.4, 4.6, 4.7 and 6.3.
- Apache Cassandra documentation: Dynamo architecture.
- Riak KV documentation: conflict resolution.
- Apache CouchDB documentation: conflicts.
- Amazon DynamoDB: how global tables work.
- PostgreSQL 18 manual: logical replication conflicts and CREATE SUBSCRIPTION.