Skip to content

    System Design

    Multi-Leader and Leaderless Replication: Conflicts and Repair

    Learn when multi-leader and leaderless replication earn their cost, how write conflicts are found and resolved, and what hints, read repair and Merkle trees do.

    By Published 13 min read

    Share

    Standard lessonadvancedBackend System Design

    Before this: Replication, Quorum Reads and Writes (R + W > N)

    What you'll learn: By the end of this lesson you will be able to say when accepting writes in more than one place is worth its cost, explain how a conflict between two writes is detected and the five common ways to resolve it (and what last-write-wins loses), compare leader topologies, describe how a leaderless store such as Dynamo or Cassandra converges after a failure, and choose a design for ShopStream data that must be edited from several places.

    Two branch offices each keep a paper ledger and phone their entries to each other every evening. Usually that works, because the offices record different customers. Then both amend the same customer's address on the same afternoon. Someone has to decide which is right, and "whoever phoned last" is a rule that quietly throws away one clerk's work.

    That is multi-leader replication in one picture. The replication lesson introduced it briefly. This lesson goes deeper on when to accept writes in several places, how conflicts are found and resolved, and how a leaderless store repairs itself. The quorum arithmetic, hinted handoff and read repair are taught in quorum reads and writes, so they appear here only as far as the conflict story needs.

    Verified on 2026-11-19 against the Dynamo paper, the Apache Cassandra, Riak KV and Apache CouchDB documentation, the Amazon DynamoDB global tables page and the PostgreSQL 18 manual. No code on this page was run.

    The core idea

    A single leader gives every write one place to be ordered. Multi-leader and leaderless designs give that up so that writes can be accepted in more than one place, and the bill is concurrent writes to the same item. Everything below is a way of paying that bill: detect the concurrency, decide a winner or a merge, and make every copy converge on the result.

    Three illustrations run through the lesson, each labelled:

    • Minimal example: two regional leaders both change one product price; the later timestamp wins and one edit disappears.
    • Realistic example: ShopStream store tablets editing the catalogue while offline.
    • Failure example: a tablet with a wrong clock wins every conflict.

    Multi-leader: where it earns its cost

    In multi-leader replication, several nodes accept writes, and each forwards its changes to the others. It is a poor way to "scale writes" for one record, because every leader still applies every write. It is justified by three situations:

    • Multiple datacentres or regions. A leader per region lets users write to a nearby node, and a regional outage does not stop the other regions. Amazon's DynamoDB global tables are a managed form: the documentation says that when an application writes to a replica in one Region, DynamoDB replicates the write to all the others.
    • Offline clients. A phone or a store tablet that must keep working without a connection is, in effect, a leader with one user. CouchDB's conflict documentation is written for this peer-to-peer style, and describes how every peer picks the same winner.
    • Collaborative editing. Several people edit one document and each sees their own edit at once; merging is the territory of CRDTs and conflict resolution.
    Two regions, each with a leader that accepts writesShoppers, region A to Leader A (local writes); Leader A to Follower A; Shoppers, region B to Leader B (local writes); Leader B to Follower B; Leader A to Leader B (async, both ways)REGION AREGION BShoppers, region AShoppers,region ALeader A: accepts writesLeader Aaccepts writesFollower AFollower AShoppers, region BShoppers,region BLeader B: accepts writesLeader Baccepts writesFollower BFollower Blocal writeslocal writesasync, both ways

    Figure 1. Multi-leader across two regions. How to read this: each region writes to its own leader, so writes stay fast and survive the loss of the other region. The dashed line between the leaders is the price: it carries changes in both directions, and nothing stops the same row changing on both sides before the changes arrive.

    Write conflicts: detection and resolution

    A write conflict is two writes to the same item that neither saw the other. As logical and vector clocks explains, two writes are concurrent when neither happened before the other, and a version vector lets a store tell "this version replaces that one" from "these two are independent".

    Minimal example (illustrative). A product's price is 3.20. Region A sets it to 3.50 and region B sets it to 3.40, a few hundred milliseconds apart, before either change has crossed the network.

    sequenceDiagram accTitle: Two leaders accept concurrent writes to one price accDescr: Region A and region B each set the same product price while the other change is still in flight. Each leader accepts its own write. When the changes cross, a last-write-wins rule keeps the later timestamp and discards the other edit, with no error reported to either writer. participant A as Leader A participant B as Leader B Note over A,B: price starts at 3.20 A->>A: set 3.50 at 10:00:00.200 B->>B: set 3.40 at 10:00:00.500 A->>B: replicate 3.50 B->>A: replicate 3.40 Note over A,B: last write wins keeps 3.40 on both, the 3.50 edit is gone

    Figure 2. A conflict resolved by last write wins. How to read this: both writers got "ok" from their own leader. Both leaders end at 3.40. The 3.50 edit was accepted, acknowledged and then silently dropped.

    Detecting a conflict

    A store can only call something a conflict if it can compare versions. Four mechanisms appear in the documentation of real systems:

    • Version vectors or causal context. Riak tags each stored value with "a piece of metadata called a causal context", which it uses to decide whether values are related; when it cannot, it keeps both as siblings. The Dynamo paper describes the same idea with vector clocks and calls the automatic case syntactic reconciliation: a version that causally follows another simply replaces it. A conflict is what is left over.
    • Revision histories. CouchDB's conflict page says that a normal read hides conflicts and that asking for them returns "the revs of the other, conflicting revision(s)". The losing revisions are kept, "hidden away as a conflicting revision".
    • Timestamps only. Cassandra and DynamoDB global tables skip detection: each compares timestamps and keeps the larger. There is no "conflict" to report, only a winner.
    • Constraint violations. In PostgreSQL logical replication, a change that violates a unique constraint on the receiving side is detected as a conflict, and the page says that such a conflict "will stop the replication; it must be resolved manually". A missing row to update or delete is skipped, not an error.

    Resolving it: five common policies

    Policy How it works Cost Example in the documentation
    Last write wins (LWW) Keep the version with the larger timestamp Drops a concurrent write; depends on clocks DynamoDB global tables use "the latest internal timestamp" per item; Cassandra: "Updates resolve according to the conflict resolution rule of last write wins"
    Keep both, let the application merge Return all versions (siblings); the application writes back one The application must do it correctly Riak siblings; the Dynamo paper's shopping cart merge
    Deterministic winner plus visible losers Every peer picks the same winner; losers stay readable Someone must notice and merge CouchDB
    Merge by construction (CRDT) Data types whose merge is defined, so no conflict arises Restricted data types, metadata growth See CRDTs
    Avoid the conflict Route all writes for an item to one home region Cross-region writes for that item pay a round trip Common design practice; no single source

    Two quotes explain why LWW needs care. Riak's documentation says it "will necessarily drop some writes in the case of concurrent updates". Cassandra's warns that its "correctness does depend on these clocks". The Dynamo paper adds a design view: if the data store resolves conflicts, "its choices are rather limited", while the application, knowing the schema, can choose a merge that suits the user. Dynamo's session service used timestamp reconciliation; its shopping cart merged versions.

    flowchart TD accTitle: Choosing a conflict policy accDescr: A decision flow. If an item can be routed to one home region, avoid conflicts. If a lost edit is acceptable, last write wins is enough. If values can be combined by a defined rule, use a merge or a CRDT. Otherwise keep both versions and let the application or a person decide. S["Item can be written in several places"] --> Q1{"Can all writes for it go
    to one home region?"} Q1 -->|yes| R1["Avoid the conflict:
    route to the home region"] Q1 -->|no| Q2{"Is losing one of two
    concurrent edits acceptable?"} Q2 -->|yes| R2["Last write wins,
    with trusted clocks"] Q2 -->|no| Q3{"Is there a defined way
    to combine two values?"} Q3 -->|yes| R3["Merge or CRDT"] Q3 -->|no| R4["Keep both versions,
    application or person decides"]

    Figure 3. A first-pass conflict policy. How to read this: ask the questions in order. Most data should exit at the first one; the lower exits cost more code and more operations.

    Topologies: who forwards to whom

    With more than two leaders you must choose how changes travel: all-to-all (every leader sends to every other), circular (each forwards to the next) or star (through a hub). A ring or star has fewer links, but a failed node in the path stops changes passing through it; all-to-all has no single path to lose, but a change can reach one leader before an earlier change it depends on, so causal ordering needs care.

    Circular topology: each leader forwards to the nextLeader A to Leader B (forward); Leader B to Leader C (forward); Leader C to Leader A (forward)Leader A: tags its own changesLeader Atags its ownchangesLeader BLeader BLeader CLeader Cforwardforwardforward

    Figure 4. A ring of leaders. How to read this: a change made at A travels A to B to C. A common way to stop it going round forever is for each change to carry the identity of the node where it began, so that a node never forwards it back to its origin.

    PostgreSQL shows the loop concern in practice: in CREATE SUBSCRIPTION, origin = none asks the publisher "to only send changes that don't have an origin", while the default any sends changes regardless of origin. The page warns that with copy_data = true and origin = none the true origin of the initial rows cannot be known.

    Leaderless replication

    In a leaderless design no node is special. A client or coordinator sends each write to several replicas and reads from several. The Dynamo paper describes its goal as an "always writeable" store: it deliberately moves the work of conflict resolution to the read path so that writes are never rejected.

    The N, R and W arithmetic, sloppy quorums, hinted handoff and read repair are in the quorum lesson. What matters here is how a leaderless store repairs divergence, as the documentation describes it:

    • On the write path: Cassandra's documentation says writes are "always sent to all replicas, regardless of consistency level", and lists hinted handoff as a best-effort convergence technique. In Dynamo, a node holding a hinted replica keeps it separately and delivers it when the intended node recovers; the paper notes hinted handoff "works best if the system membership churn is low and node failures are transient".
    • On the read path: the coordinator may gather several versions; Dynamo returns "all the versions it deems to be causally unrelated" and the reconciled version is written back. Cassandra lists replica read repair as best-effort.
    • In the background: anti-entropy repair compares hash trees. Dynamo's Merkle trees let two replicas compare roots and descend only into differing branches, which "reduce the number of disk reads performed during the anti-entropy process".
    • Bounding version metadata: Dynamo truncates a vector clock past a threshold (the paper says "say 10"), dropping the oldest pair, which can blur which version descends from which.

    Across datacentres, Cassandra uses NetworkTopologyStrategy with a replication factor per datacentre, and offers LOCAL_QUORUM (a majority in the coordinator's datacentre) and EACH_QUORUM (a majority in every datacentre). This is the leaderless answer to the multi-region question: a client writes at local latency and the other datacentre catches up.

    Leaderless write across two datacentresClient to Coordinator (write); Coordinator to Replica 1 (must ack); Coordinator to Replica 2 (must ack); Coordinator to Replica 3 (sent, not awaited)LOCAL DATACENTREREMOTE DATACENTREClientClientCoordinator: any nodeCoordinatorany nodeReplica 1: local DCReplica 1local DCReplica 2: local DCReplica 2local DCReplica 3: remote DCReplica 3remote DCwritemust ackmust acksent, not awaited

    Figure 5. A write at a local-quorum consistency level. How to read this: the coordinator sends the write to every replica but waits only for a majority in its own datacentre, so a read in the remote one may briefly return the older value.

    Trade-offs

    Design Gain Cost Wrong when
    Single leader with followers One order of writes, no conflicts Remote writers pay a round trip; failover Writers must work offline or in several regions
    Multi-leader Local writes, survives a region loss, offline work Conflicts, weak constraints, harder debugging One home region per item is possible
    Leaderless No failover step, always-writeable Stale reads, conflict handling, clock dependence You need a single agreed order
    Last write wins No application code Silently drops writes; trusts clocks Every edit matters
    Siblings or merge No lost edits The application must resolve correctly Nobody owns the merge code
    Home-region routing No conflicts by design A round trip for non-local writes Users move between regions mid-session

    Failure example: the clock that always wins

    Under LWW, timestamps decide. If one tablet's clock is two days ahead, its edits beat everyone else's for two days. The damage is silent: no error, only the wrong price. A second failure is the PostgreSQL case above: a unique-constraint conflict stops replication until a person resolves it.

    In practice: offline catalogue edits across 5,000 stores

    This is an educational scenario on ShopStream with illustrative numbers.

    This is a realistic scenario, not a case history.

    The situation

    This is the realistic example. ShopStream has 5,000 stores (the shared figure, illustrative). Each keeps a local catalogue on a tablet so staff can mark stock or set a shelf price override even when the connection drops. Each tablet is a leader for its store's overrides and a regional service is the other leader; orders and payments stay out, because they have one writer. Staff report that stock marks made during a drop vanish after reconnecting, and head office wants to push promotional prices that stores can still override.

    Starting architecture

    Store tablets and the regional catalogue, starting architectureStore tablets to Sync service (edits); Sync service to Regional catalogue; Head office tool to Regional catalogue (writes)Store tablets: local copyStore tabletslocal copySync service: last write winsSync servicelast write winsRegional catalogue: second leaderRegionalcataloguesecond leaderHead office tool: promo pricesHead officetoolpromo priceseditswrites

    Every edit carries the tablet's wall-clock time, and the sync service keeps the larger timestamp per product.

    Options considered

    Option Gain Cost Wrong when
    A. Online-only edits No conflicts Tablets stop working in a drop Connections are unreliable
    B. Keep LWW, add clock checks Small change Still drops concurrent edits Edits must not be lost
    C. Operations for stock, versioned price with a review queue No lost stock marks; price conflicts become visible Sync service and a review screen to build The team cannot own the merge code

    The decision

    Option C. A price override is an absolute value, so concurrent edits are a true conflict; a stock movement ("received 20 units") is better recorded as an operation that is added, as in idempotent and commutative operations. Stock movements sync as operations. Price overrides carry a version vector per product; concurrent price edits are both kept and shown to a store manager. Head office promotional prices live in a separate field so they never conflict with local overrides. Timestamps from tablets are never used to order edits.

    Rollout and verification

    Run the new sync beside the old one for two weeks and compare outcomes per product. Sample the flagged conflicts by hand. Check that a reconnecting tablet applies its edits exactly once.

    The 2 a.m. failure

    The page reads: "Checkout price differs from shelf price in 40 stores". A tablet's clock was reset and one store's overrides now win.

    sequenceDiagram accTitle: A tablet with a wrong clock overwrites a newer edit accDescr: Head office sets a price. A store tablet whose clock is two days ahead sets an older override and stamps it with a future time. When the two changes meet, the future timestamp wins and the head office price is overwritten. participant H as Head office participant R as Regional leader participant T as Tablet, clock 2 days ahead H->>R: set price 2.90 at day 1 T->>R: set override 3.60, stamped day 3 R->>R: larger timestamp wins, keeps 3.60 Note over R: the newer head office edit is lost

    First checks: list changes whose timestamp is later than the server's receive time, and compare the clock offset reported at sync. Mitigation: reject client timestamps ahead of the server and re-apply the head office price. Follow-up: the server assigns the order; the tablet clock is no longer an input.

    Architect's notes

    • Ask first whether writes in two places are needed. A read replica near users solves most locality needs without conflicts.
    • Name the policy per data type, not per system.
    • Never let a client clock order writes. Use server-assigned versions or version vectors.
    • Rehearse a partition: write the same item on both sides, heal, and check the outcome matches the policy.
    • Keep constraints single-writer: unique values and non-negative counters do not survive multiple leaders without extra design.

    Common mistakes

    • Adopting multi-leader to scale writes. It adds conflicts without adding capacity for one item.
    • Last write wins by default, with no review of what is lost.
    • Trusting wall clocks from devices you do not control.
    • Forgetting loops in a ring or all-to-all setup.
    • Skipping anti-entropy and assuming hints and read repair converge everything.

    Key takeaways

    • Multi-leader and leaderless designs accept writes in several places and pay with concurrent writes.
    • Conflicts are detected with version metadata, revision histories or constraint checks; timestamp-only systems never see one, they only pick a winner.
    • Last write wins is cheap, drops concurrent writes and trusts clocks.
    • The alternatives are siblings and application merge, a deterministic winner with visible losers, CRDTs, or one home region per item.
    • Leaderless stores repair divergence with hints, read repair and hash-tree comparison.
    • Choose a policy per data type; keep orders, payments and unique constraints single-writer.

    Practice exercises

    Exercise 1 (30 min). Two leaders accept edits to one record. Draw the timeline, apply last write wins, then a version vector, and state which edit survives in each case.

    Exercise 2 (1 hour). For ShopStream, list five fields a store tablet can edit. For each, choose a policy from the table and justify it in one sentence.

    Official documentation

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments