What you'll learn: By the end of this lesson you will be able to say what a lock is for (efficiency or correctness), explain how a lease lock works and why its expiry is both its strength and its weakness, show with a runnable Java lab how a paused client overwrites newer data, add fencing tokens so the storage layer rejects the stale write, describe what the Redis lock recipe and Redlock claim and where Kleppmann and antirez disagree, and choose between a lock with fencing, a conditional write and no lock at all.
A meeting-room booking system hands out the room for an hour. You book it and start writing on the whiteboard, then fall asleep at the desk. The hour ends, the system gives the room to the next team, and they wipe the board. You wake up, still sure the room is yours, and carry on writing over their work. Nobody broke the booking system's rules; it cannot see you, and you cannot see the clock while you sleep.
A distributed lock has exactly this shape. The lesson on consensus and leader election introduces leases and fencing tokens in a few paragraphs. This lesson is the dedicated treatment: the mechanism, a lab that runs the failure, and a careful reading of what the Redis documentation and its critics actually say.
Verified with Java 25 and the Redis, etcd and ZooKeeper documentation pages listed under "Official documentation", read on 2026-11-08. The lab is plain Java with a fake clock; it uses no Redis.
The core idea
A lock with an expiry (a lease) keeps a crashed client from holding the lock forever, but it cannot stop a paused client from acting after its lease has run out. The only place that can stop that client is the resource being protected, and it can do so if every lock grant carries a number that only grows and the resource refuses any request with a smaller number. That number is a fencing token.
Three examples run through the lesson, each labelled:
- Minimal example: two clients and one balance in plain Java, the lab in "Build it".
- Realistic example: ShopStream's per-store catalogue import, in "In practice".
- Failure example: the paused holder that overwrites newer data, shown step by step with and without fencing.
What a lock is for: efficiency or correctness
Martin Kleppmann's article "How to do distributed locking" separates two reasons to take a lock, and the distinction decides everything that follows.
| Purpose | What a lock failure costs | Example | What is enough |
|---|---|---|---|
| Efficiency | Some work done twice: a minor cost | Two workers both build the same thumbnail | A simple lock with an expiry; it is fine if it occasionally fails |
| Correctness | Corrupted data, a lost update, a double charge | Two writers both update one account balance | A lock plus a check at the resource, because the lock alone can be wrong |
If a duplicate is merely wasteful, a lease lock is a good tool. If a duplicate is harmful, the lock is at best a way to make conflicts rare, and the safety has to come from somewhere else. The rest of this lesson is about where.
A lease lock, step by step
A lease is a lock that expires. The lock service grants the lock to one client for a fixed time. When the time runs out the service treats the lock as free and may grant it to the next caller, whether or not the first client has finished or even noticed.
The expiry exists for one reason: clients crash. A lock with no expiry that is held by a dead client is held forever. The lease turns "the holder died" into "the lock comes back after the time to live".
Figure 1. Two places, two jobs. How to read this: the lock service decides who may start. It cannot see what a client does afterwards. The storage sees every write, so it is the only component that can refuse the stale write from client A (red dashed arrow).
A lease keeps two holders from being granted the lock at the same moment only if one authority decides and clocks run at roughly the same rate; the Redis page says so for its own algorithm. That makes the service's grants consistent. It says nothing about what a client does between being granted the lock and using it.
The failure: a paused holder
A client can stop for a long time between any two instructions: a stop-the-world garbage-collection pause, a virtual machine frozen by its host, a starved process, a delayed packet. The client does not experience the pause; when it resumes, its next instruction runs as if no time had passed.
Figure 2. The lost update. How to read this: both clients followed the protocol. The lock service granted the lock to one holder at a time. Storage accepted two writes, and the second silently replaced the first.
The tempting fix is for client A to check its lease just before writing. Kleppmann answers directly: you cannot fix this problem "by inserting a check on the lock expiry just before writing back to storage", because the pause can happen after the check and before the write. No amount of care inside the client closes that gap. This is not a flaw in a particular lock product; it applies to any lock whose holder can outlive its grant.
Fencing tokens: make the resource say no
A fencing token is a number that increases every time the lock is granted. Client A gets token 1; when its lease expires and B takes the lock, B gets token 2. Every request that A or B sends to the storage carries its token. The storage remembers the highest token it has seen and rejects a request with a lower one.
Figure 3. The same pause with fencing. How to read this: the lock service is unchanged. The difference is that the storage keeps one number and compares it on every request, so the late write is refused instead of applied.
Kleppmann puts it in one sentence: the approach "requires the storage server to take an active role in checking tokens, and rejecting any writes on which the token has gone backwards". Three consequences follow. The resource must cooperate: a system that cannot check a token (a legacy file share, a third-party API) needs a conditional write or an idempotent operation instead. Every resource under the lock must check, because one unchecked side effect such as an email is still exposed. And the token must come from the authority that grants the lock, atomically with the grant.
Where tokens come from
| System | What can serve as the token | Source |
|---|---|---|
| A lock table in a database | A counter column incremented in the same statement that grants the lock | This lesson's design sketch (not from a vendor page) |
| ZooKeeper | The zxid or the znode version number (Kleppmann's suggestion). The recipes page builds locks from sequential ephemeral nodes and does not discuss fencing |
Kleppmann; ZooKeeper recipes |
| etcd | Lock returns a key that exists while the lock is held; the lock is tied to a lease, which, if it expires, releases the lock. The page does not describe the key's revision as a token |
etcd API reference |
| Redis (single instance or Redlock) | Nothing built in. Kleppmann: Redlock "does not have any facility for generating fencing tokens" | Kleppmann |
A fencing token is a requirement you place on your lock service, not a feature every product ships.
A second way to say no: a conditional write
Fencing orders the grants. A conditional write (compare and set) instead checks that the data is still what you read. antirez's reply describes this: give each lock a random token, "implement Check and Set" by setting the resource's state to the token, and write only if it is unchanged. It detects interference without a monotonic counter, and needs the resource to take part just as fencing does. The lab's stretch exercise builds it.
Build it: the paused client, with and without fencing
Goal. See a lost update with a lease lock and no fencing, see the stale write rejected with fencing, and see which fencing designs still leave a hole.
Setup. From the repository's examples/backend folder:
cd examples/backend && ./gradlew :labs:fenced-lease-lock:test
./gradlew -q :labs:fenced-lease-lock:runThe first command runs the 18 tests; the second prints the five scenarios below. The code is in examples/backend/labs/fenced-lease-lock/src/main/java/buildingai/labs/fencedlease/.
How the model works. LeaseLockService grants one lock for a lease of 10,000 ms and gives every grant the next token (1, 2, 3, ...). A fake clock moves only when a scenario says so; "a pause" advances it by 15,000 ms. ProtectedAccount is the storage layer, and its whole mechanism is one method:
private void checkAndRemember(long token) {
if (token < highestToken) {
rejectedRequests++;
throw new StaleTokenException(token, highestToken);
}
highestToken = token;
}Fencing.NONE never calls it. WRITES_ONLY calls it on writes. READS_AND_WRITES calls it on reads and writes. AddJob is "add an amount to the balance" in three separate steps (acquire, read, write), so a scenario can pause a client between any two of them.
Steps. Run the tests, then the demo, and compare with the output below, using the guide after it.
Expected output, captured from a real run (Java 25.0.3, 2026-11-08):
Lease lock with fencing tokens. Lease: 10000 ms. Balance starts at 100; A adds 50, B adds 10; right answer: 160.
1. Lock without fencing: the storage accepts every token
t(ms) who step result balance
----------------------------------------------------------------------------------------------------------------
1000 A acquire the lock granted, token 1, lease until 11000 100
1000 A read the balance read 100 (token 1) 100
16000 A pauses (GC, VM stall, slow network) clock +15000 ms; its lease has expired 100
16000 B acquire the lock granted, token 2, lease until 26000 100
16000 B read the balance read 100 (token 2) 100
16000 B write read+10 accepted: balance is now 110 (token 2) 110
16000 B release the lock released 110
16000 A write read+50 (still holds token) accepted: balance is now 150 (token 1) 150
final balance: 150 (right answer 160): WRONG, an update was lost; storage accepted 2 writes and rejected 0 requests
2. Same pause, storage checks tokens on reads and writes
t(ms) who step result balance
----------------------------------------------------------------------------------------------------------------
1000 A acquire the lock granted, token 1, lease until 11000 100
1000 A read the balance read 100 (token 1) 100
16000 A pauses (GC, VM stall, slow network) clock +15000 ms; its lease has expired 100
16000 B acquire the lock granted, token 2, lease until 26000 100
16000 B read the balance read 100 (token 2) 100
16000 B write read+10 accepted: balance is now 110 (token 2) 110
16000 B release the lock released 110
16000 A write read+50 (still holds token) REJECTED by storage: token 1 < highest seen 2 110
16000 A start over: acquire granted, token 3, lease until 26000 110
16000 A read the balance read 110 (token 3) 110
16000 A write read+50 accepted: balance is now 160 (token 3) 160
16000 A release the lock released 160
final balance: 160 (right answer 160): correct; storage accepted 2 writes and rejected 1 requests
3. Storage checks tokens on writes only; A wakes between B's read and B's write
t(ms) who step result balance
----------------------------------------------------------------------------------------------------------------
1000 A acquire the lock granted, token 1, lease until 11000 100
1000 A read the balance read 100 (token 1) 100
16000 A pauses (GC, VM stall, slow network) clock +15000 ms; its lease has expired 100
16000 B acquire the lock granted, token 2, lease until 26000 100
16000 B read the balance read 100 (token 2) 100
16000 A write read+50 (still holds token) accepted: balance is now 150 (token 1) 150
16000 B write read+10 accepted: balance is now 110 (token 2) 110
16000 B release the lock released 110
final balance: 110 (right answer 160): WRONG, an update was lost; storage accepted 2 writes and rejected 0 requests
4. Same timing, storage checks tokens on reads and writes
t(ms) who step result balance
----------------------------------------------------------------------------------------------------------------
1000 A acquire the lock granted, token 1, lease until 11000 100
1000 A read the balance read 100 (token 1) 100
16000 A pauses (GC, VM stall, slow network) clock +15000 ms; its lease has expired 100
16000 B acquire the lock granted, token 2, lease until 26000 100
16000 B read the balance read 100 (token 2) 100
16000 A write read+50 (still holds token) REJECTED by storage: token 1 < highest seen 2 100
16000 B write read+10 accepted: balance is now 110 (token 2) 110
16000 B release the lock released 110
16000 A start over: acquire granted, token 3, lease until 26000 110
16000 A read the balance read 110 (token 3) 110
16000 A write read+50 accepted: balance is now 160 (token 3) 160
16000 A release the lock released 160
final balance: 160 (right answer 160): correct; storage accepted 2 writes and rejected 1 requests
5. Every ordering of two 3-step clients and two long pauses (560 orderings, same clients as above)
storage orderings lost updates first losing ordering (A/B = that client's next step, P = pause)
NONE 560 108 AAPBBABP
WRITES_ONLY 560 54 AAPBBABP
READS_AND_WRITES 560 0 -| Scenario | What it shows |
|---|---|
| 1 | No fencing. A reads 100 and pauses, B adds 10, A wakes and writes 100 + 50: final balance 150, B's update lost. Storage cannot know A's lease expired. |
| 2 | Storage checks tokens. A's late write (token 1) is refused because token 2 was seen; A takes token 3, reads 110, writes 160, the right answer. |
| 3 | Writes-only checking, and A wakes between B's read and B's write. The highest write token so far is 0, so A's token 1 passes; B then writes from its stale read. Final balance 110: fencing writes alone left a hole. |
| 4 | The same timing with the token on reads too. B's read raised the highest token to 2, so A's write is refused and the retry gives 160. |
| 5 | All 560 orderings of two three-step clients and two long pauses: 108 lose an update with no fencing, 54 with writes-only fencing, none when reads and writes are checked. Exhaustive for this small model only. |
Break it. In ProtectedAccount.read, change the condition fencing == Fencing.READS_AND_WRITES to false. The "reads and writes" storage now behaves like writes-only, and three tests fail: checkingTokensOnReadsAsWellClosesThatWindow, aReadWithAStaleTokenIsRejectedOnlyWhenReadsAreFenced and withFencingOnReadsAndWritesNoOrderingLosesAnUpdate.
Stretch. Replace the token check with a version column and a conditional write, and decide which of the four scenarios it protects. Add a second resource and show why it must check too. Add a renewal thread and pause that.
What the model leaves out. A real network, real clocks, a fault-tolerant lock service (this one is a single in-memory object), several locks and a real database. The WRITES_ONLY hole and the reads-and-writes remedy are observations of this model, not claims from a published source.
Redis locks and Redlock: what the primary sources say
Redis is where many engineers first meet a distributed lock, so here is exactly what the Redis documentation, Kleppmann and antirez each claim.
The single-instance recipe. The Redis page gives SET resource_name my_random_value NX PX 30000: set the key only if it does not exist, with a 30,000 ms expiry, to a value unique across clients. The random value makes release safe: delete the key only if it still holds your value (a Lua script, or DELEX key IFEQ my_random_value, which the page says arrived in Redis 8.4). A plain DEL could remove the next holder's lock. The page calls this approach viable "in applications where a race condition from time to time is acceptable".
Why not just add a replica. Redis replication is asynchronous: if the master crashes before the key reaches the replica and the replica is promoted, a second client can acquire a lock the first still holds.
Redlock. With N independent masters (the page's example uses 5), a client records the time, tries to set the same key and value on all of them with a short per-node timeout, and counts the lock as acquired only if a majority accepted it and the elapsed time is less than the lock validity time. The remaining validity is the initial validity minus the elapsed time. The page states the guarantee as holding only while the client finishes within the validity time, minus a small margin for clock drift.
What the Redis page says about its own limits. Its "Disclaimer about consistency" tells readers who care about correctness to implement fencing tokens, "especially important for processes that can take significant time and applies to any distributed locking system", and notes that Redis does not use a monotonic clock for expiry, so a wall-clock shift "may result in a lock being acquired by more than one process".
The two published positions.
| Question | Kleppmann (2016) | antirez's reply |
|---|---|---|
| Is Redlock suitable when correctness matters? | No: "please don't use Redlock"; use a consensus system such as ZooKeeper, with fencing on every access | Replies to the criticisms point by point |
| Fencing tokens | Redlock has no facility to generate them; the storage must check them | Argues that a random token plus check-and-set on the resource can do the job, and calls the monotonic-token approach unrealistic |
| Clocks | Redis uses gettimeofday, subject to jumps; safety depends on timing assumptions |
Agrees "Redis and Redlock implementations should switch to the monotonic time API"; says clock jumps can be avoided by not changing the clock or using slewing NTP |
| Pause between the check and the use | The lock cannot prevent it | Says it affects every expiring lock, and that the client should check remaining validity before using the resource |
Both agree that a pause after the last check can outlive the lease; they disagree on how much it matters and on whether Redlock is the right tool, and this lesson does not settle that. The practical reading is to match the lock to its purpose: for efficiency the single-instance recipe is cheap and reasonable; for correctness put a check at the resource whichever lock you use, and take the lock from a system whose grants are decided by consensus.
Choosing: lock with fencing, conditional write, or no lock
| Option | Gain | Cost | When it is the wrong choice |
|---|---|---|---|
| Lease lock, no fencing | Simple; fewer duplicates | A pause can still cause a conflict | Whenever a conflict corrupts data |
| Lease lock plus fencing token | A stale holder's writes are refused | A token source, and every resource must check | A resource that cannot check tokens |
| Conditional write (version or compare and set) on the data | No lock service at all for single-row changes | A retry loop; contention is visible as failures | Work that spans several resources |
| Lock from a consensus system (ZooKeeper, etcd) plus fencing | Grants decided by a majority; tokens available | An extra system to run; latency of a quorum | A cheap, efficiency-only lock |
| Avoid the lock: one writer per key through a queue, or idempotent operations | No mutual exclusion to get wrong | A routing layer; idempotency design | Work that cannot be partitioned or repeated safely |
In practice: one importer per store, even when a pod freezes
This is the realistic example. It is an educational scenario on the course's running example, ShopStream, and nothing in it describes a real company. Every figure is illustrative or taken from the shared ShopStream numbers.
This is a realistic scenario, not a case history.
The situation
ShopStream has 5,000 stores (the shared planning number). Each store uploads a price file and a background importer applies it to that store's catalogue rows. A lease lock per store keeps two importer pods from interleaving. One night two uploads for the same store arrive close together and its prices briefly show the older file's values.
Starting architecture
Figure 4. The design under discussion. How to read this: the lock table decides which pod may import a store. The catalogue accepts every write from anyone, which is the weak point (amber).
Options considered
| Option | Behaviour when a pod freezes | Cost | Risk |
|---|---|---|---|
| A. Longer lease | The window shrinks | One number | A long pause still wins; crashes take longer to recover |
| B. Check the lease before writing | The window shrinks | A line of code | The check can be followed by a pause; Kleppmann's point |
| C. Token from the lock table, checked by the catalogue write | A stale pod is refused | A token column and a changed update | Needs every writer to send the token |
| D. One queue partition per store, one consumer | No lock needed | Routing by store id | Rebalancing can still briefly give two consumers |
The decision
Option C. The lock table grants the lock and increments a token in one statement, and the catalogue write refuses an older token. The statements are a sketch, not run here, written for PostgreSQL:
-- Acquire: grant only if the previous lease has expired; the token grows with every grant.
UPDATE import_lock
SET holder = $1, token = token + 1, expires_at = now() + interval '30 seconds'
WHERE store_id = $2 AND expires_at < now()
RETURNING token;
-- Apply: the write succeeds only if no newer grant has already written.
UPDATE store_catalogue
SET prices = $3, fence = $4
WHERE store_id = $2 AND fence <= $4;If the second statement updates zero rows, the importer was fenced out and must stop and re-queue its work. The acquire statement uses the database's own clock for expiry, so one clock decides, not one per pod. A blind overwrite of one row, like this, needs only the check on the write; a read-modify-write needs the lab's lesson: carry the token on the read as well, or do both in one transaction with the check.
Rollout and verification
Add the fence column with a default of 0, ship importers that send the token, then enable the check last. Verify with a fault test: freeze one importer between acquiring and writing (a process stop signal works), let its lease expire, let a second importer finish, resume the first, and assert its write is refused. Watch fenced_write_rejected_total (an increase proves the guard works) and lease acquisition latency.
The 2 a.m. failure
The page reads: a store's prices flipped back to an older file. First checks: importer logs for two pods on one store, the lock table's token and expires_at, and the catalogue row's fence. If fence is lower than token, a writer skipped the check (a one-off script, an old importer version). Mitigation: re-apply the newer file under a new grant. Lasting fix: make the catalogue refuse any price update without a token, so a forgotten code path fails loudly instead of overwriting quietly.
Architect's notes
- Name the purpose. Write "efficiency" or "correctness" next to every lock in a design review; the answer changes the architecture.
- Put the guard where the damage happens. The storage layer is the last component that sees the write.
- Test a frozen process, not only a killed one. A pause is the failure leases do not cover.
- Consider not locking. Idempotent operations and one writer per key remove the problem.
Common mistakes
- Treating a lease as proof of exclusivity. It proves the service granted the lock, not that the holder is still running.
- Checking the lease just before the write. A pause can follow the check.
- Releasing with a plain delete. A late release removes the next holder's lock; compare your own value first.
- Fencing only the writes of a read-modify-write (the lab's scenario 3), or generating the token on the client.
Key takeaways
- A lock serves efficiency or correctness; only correctness needs more than a lease.
- A lease cannot stop a paused holder; the pause can occur after any check.
- A fencing token grows with every grant and is checked by the resource, which refuses lower ones; conditional writes also need the resource to take part.
- The Redis page itself tells readers who care about correctness to implement fencing tokens.
- The cheapest safe design is often no lock at all.
Practice exercises
Exercise 1 (30 min). Run the lab and, for each of scenarios 1 to 4, write the final balance and which request the storage refused, if any.
Exercise 2 (1 hour). Run "Break it", then explain in two sentences why scenario 3 loses an update even though a token check is on. Then do the stretch: add a version and a conditional write and decide which scenarios it protects.
Exercise 3 (1 hour). For three jobs in a delivery app (assign a courier, send a receipt email, close a day's payouts), state whether the lock is for efficiency or correctness and what the resource-side check would be.
Exercise 4 (2 hours). Write the fault test for ShopStream's importer: freeze a pod, expire its lease, let another pod finish, resume the first, and state the assertion and the metric that must increase.
Related reading
- Consensus and Leader Election: leases and fencing in the context of leaders, and where majorities come from.
- IDs, Clocks and Ordering: why clocks on different machines cannot be compared directly.
- Distributed Transactions: coordinating changes across more than one resource.
- CAP Theorem and PACELC: why a majority-based lock service refuses requests on the minority side of a partition.
- Resilience Patterns: timeouts and retries that make pauses visible.
- ShopStream Stage 4: scale writes: the write path the importer example sits beside.
Official documentation
- Kleppmann, How to do distributed locking (2016): efficiency versus correctness, the paused client, fencing tokens and the critique of Redlock.
- antirez, Is Redlock safe?: the author's reply.
- Redis documentation, Distributed Locks with Redis: the single-instance recipe, Redlock and the page's own disclaimer.
- etcd documentation, Concurrency API reference (v3.5): the Lock service and its relation to leases.
- Apache ZooKeeper, Recipes and solutions: locks from sequential ephemeral nodes.