Go beyond cache-aside: write policies, invalidation, stampedes, hot keys, distributed caches, negative caching and multi-level caches, with the failure modes of each.
What you'll learn: By the end of this lesson you will be able to choose between write-through, write-back and write-around policies, prevent cache stampedes and hot-key overload, explain how a distributed cache spreads keys with consistent hashing, and design a multi-level cache whose contents stay acceptably consistent with the database.
Picture a busy coffee shop. The barista keeps the five most popular drinks pre-made on the counter. That is a cache, and it works beautifully until something goes wrong. A new menu arrives and the counter still has last week's recipes. Ten customers ask for a drink that is not on the counter, and all ten baristas start brewing the same thing at once. One drink becomes a viral hit and a single machine is overwhelmed. Each of these has a name in system design, and each has a standard fix.
Caching basics covered why caches work and the cache-aside pattern. This lesson is about the difficult part: what happens when caches are written to, when they expire, when they fail and when they are spread over many machines. Caches are the cheapest way to cut load and latency, and one of the most common sources of subtle production incidents.
The core idea
A cache is a second copy of data, and every second copy can disagree with the first. All advanced caching techniques answer one of three questions: how do writes reach the cache and the database (write policies), how does a stale entry get replaced (invalidation), and what happens when many requests need the same missing entry at once (stampedes and hot keys).
Before choosing a technique, decide how much staleness each piece of data can tolerate. A product description can be minutes old. An account balance usually cannot be cached at all. That tolerance sets everything else.
Write policies: how data gets into the cache and the database
Solid arrows are on the write path; dotted ones happen later or on reads. Write-through writes to the cache, which writes to the database before acknowledging: the cache is fresh for written keys, at the cost of write latency and caching data that may never be read. Write-back acknowledges from the cache and flushes later in batches: very fast writes, and repeated updates to a key coalesce, but if the node dies before flushing, acknowledged writes are lost, so use it only for loss-tolerant data (counters, metrics) or a replicated, persistent cache. Write-around writes only to the database and fills the cache on reads: it avoids polluting the cache with write-once data, but the first read after a write is a miss and an old cached value lives until invalidated or expired.
Policy
Write latency
Staleness risk
Data-loss risk
Use when
Write-through
Higher (two writes)
Low
Low
Read-after-write must be fresh; data read often
Write-back
Lowest
Database lags
Real: unflushed writes can be lost
High write rate, loss tolerable
Write-around
Normal
Cached copy can be stale
Low
Write-heavy, rarely re-read data
Many web applications simply use cache-aside: read the cache, fall back to the database on a miss and populate the cache; on a write, update the database then delete the cache key. That keeps the cache out of the write path and is the default we compare to below.
Invalidation: replacing stale entries
The difficulty is real, because the cache usually does not know when the source of truth changed. The main strategies:
Time-based expiry (TTL). Simple and robust, with a hard upper bound on staleness. Choose the TTL from the business tolerance, not a guess.
Explicit invalidation on write. The code that changes data also deletes the key. Fresh, but every writer must know every dependent key.
Event-driven invalidation. The database publishes changes (change data capture, see search indexing) and a consumer deletes affected keys. This decouples writers from caches.
Versioned keys. Put a version in the key, such as product:42:v17. Updating bumps the version; old entries are never read again and expire by TTL.
TIP
Prefer delete over update on writes. If two concurrent writers update the cache value, the older one can win. Deleting forces the next reader to load current database state.
Even delete-on-write has a gap. A reader misses and reads the old row; a writer updates the row and deletes the key; the slow reader then writes its old value into the cache.
sequenceDiagram
participant R as Reader
participant W as Writer
participant C as Cache
participant D as Database
R->>C: get key
C-->>R: miss
R->>D: read row
D-->>R: old value
W->>D: update row
W->>C: delete key
R->>C: set key to old value
Note over C: stale entry until TTL
The race is rare, which is why a TTL remains a safety net even with explicit invalidation. Short TTLs for hot data, delayed second deletes and versioned keys all tighten it.
Cache stampede and its fixes
A cache stampede (thundering herd, dogpile) happens when a popular key expires and many requests miss at once, all rebuilding the same value from the database. For illustration, if a key serves 5,000 requests per second and takes 200 ms to rebuild, about 1,000 requests hit the database for the identical query during that window. If the database is near capacity, rebuilds slow down, more requests pile up, and the system spirals.
Only one request pays for the rebuild; the rest wait or take the stale value. The fixes, alone or together:
Request coalescing (single-flight). Within one process the first miss rebuilds and others wait for its result.
Distributed lock or lease. Across servers, the first to miss takes a short lock (a key set only if absent, with an expiry). Others wait and re-read, or serve stale. Always time-limit the lock so a crashed rebuilder cannot block everyone.
Jittered TTLs. Add random variation (for example plus or minus ten percent) so keys loaded together do not expire together.
Early (probabilistic) refresh. Refresh shortly before expiry; each reader near expiry refreshes with a probability that rises as time runs out, so usually one request does it.
Stale-while-revalidate. Serve the expired value while one background request fetches the new one.
Pre-warming. Load the hottest keys before a known event such as a sale or a cache-flushing deploy.
Hot keys
A hot key receives a disproportionate share of traffic: a celebrity profile, a viral item, a global config blob. In a distributed cache a key lives on one node (plus maybe a replica), so that node can saturate while the cluster is idle. Remedies: local in-process caching of the hottest keys with a seconds-long TTL; key replication under derived keys (profile:42#1 to #8, readers pick randomly, writers invalidate all); reading from cache replicas; and detect and promote, tracking per-key request counts and moving the top few into a faster tier.
NOTE
Spot hot keys with per-node load, not the cluster average. A cluster at 30 percent average CPU can still have one node at 100 percent.
Distributed caches and consistent hashing
One cache server cannot hold all your data or serve all your traffic. The naive placement node = hash(key) mod N breaks when N changes: nearly every key maps elsewhere, the cache effectively empties, and the database absorbs the load, a self-inflicted stampede. Consistent hashing places nodes (many virtual points each) on a ring of hash values; a key belongs to the first node clockwise from its hash. Adding or removing a node moves only the keys next to it, roughly 1/N of the total. The sharding lesson covers the ring in depth.
flowchart TD
K["Key hash"] --> Ring["Hash ring with virtual nodes"]
Ring -->|"first node clockwise"| N1["Cache node 1"]
Ring --> N2["Cache node 2"]
Ring --> N3["Cache node 3"]
N3 -.->|"node fails: only its keys move"| N1
Make sure every client uses the same topology and hashing method, or one key lands on different nodes. Cache replication is for availability, not durability: the cache must always be rebuildable from the database. And plan for the cache cluster being completely empty, with rate limiting or a warm-up procedure so the database survives it.
Negative caching
A lookup that finds nothing can be cached too. Without it, requests for non-existent ids hit the database every time, which is wasteful and dangerous when someone requests millions of random ids to bypass the cache (cache penetration). Cache a "not found" marker with a short TTL (seconds to a minute), delete it when the item is created, and for large id spaces put a Bloom filter in front, which can say "definitely not present" (false positives possible, no false negatives). Cache errors cautiously: a short negative TTL protects a struggling dependency, but a long one turns a brief outage into a long one.
Multi-level caches
Level
Typical content
Latency (rough)
Staleness control
Main risk
Browser
Static assets, cacheable API responses
Near zero
Cache headers, versioned filenames
Cannot be invalidated remotely
CDN
Images, scripts, public pages
Tens of ms
TTL, purge, cache keys
Old content after deploys
In-process
Hot keys, configuration
Microseconds
Short TTL
Each server holds a different copy
Shared (Redis, Memcached style)
Query results, sessions
About a millisecond in a data centre
TTL, invalidation
Stampedes, hot keys, cold start
Database buffer cache
Hot pages and indexes
Memory speed
Managed by the database
Not under your control
More levels mean more hits and more places for stale data. A purge at one level does nothing for the others, and worst-case staleness is roughly the sum of the layers' TTLs, so give short TTLs to layers you cannot purge.
Keeping the cache and database consistent
Perfect consistency needs distributed transactions, which teams rarely pay for. Aim for bounded staleness:
Treat the database as the source of truth; the cache is disposable.
Set a TTL on every entry, even with explicit invalidation; it bounds how wrong the cache can be if an invalidation is lost.
Invalidate after the database commit, never before.
Make invalidation reliable: retry it, or publish it from the database change stream.
Provide read-your-writes where it matters, reading from the database for that user briefly after an edit.
Do not cache what must be exact: balances during checkout, inventory at the moment of purchase, authorisation that must reflect immediate revocation.
IMPORTANT
A cache that is the only holder of data is a database with no durability. If losing the cache would lose information you cannot rebuild, redesign before you scale.
In practice: a flash-sale product page
NOTE
A realistic scenario, not a case history. This is a composite example built from common production patterns. It does not describe any specific company, and every number in it is an illustrative assumption.
The situation
A mid-size online retailer runs hourly flash sales: one hero product, 2,000 units, announced by push notification and email. A team of 14 owns the storefront. The stack is Nginx in front of a Kubernetes deployment, a Redis cluster of 6 nodes for product data, a PostgreSQL primary with 2 read replicas, and a CDN for images and static files. Observability is Prometheus and Grafana.
At the last sale the page went from about 300 to a peak of about 40,000 requests per second within 90 seconds. The hero product page was cached with a 60 second TTL in Redis. When the key expired at the peak, thousands of requests missed together, the primary's connection pool saturated, p99 climbed from 120 ms to several seconds, and checkout (which shares the database) failed. The team also showed an exact "units left" counter on the page, which forced reads to bypass the cache.
Starting architecture
Every miss and every stock-count request lands on the primary, and the hot key lives on one Redis node.
Options considered
Option
What it fixes
What it costs
Risk
A. Do nothing, raise the TTL to 10 minutes
Fewer expiries
Price or description edits show late
Does not stop the cold-start or first-load stampede
B. Single-flight rebuild, stale-while-revalidate, jittered TTL, pre-warm, short local cache, bucketed stock
Removes the stampede and the hot-key node limit
Some code and a runbook
Slightly stale page; stock text is approximate
C. Full-page edge caching on the CDN
Absorbs most traffic before it reaches origin
Purge workflow, personalisation split
Stale price risk if purge fails
The decision
ADR field
Content
Context
One key at the moment of expiry decides the incident; exact stock on the page forces database reads.
Decision
Option B now. The page body is cached with a 5 minute TTL plus jitter and refreshed early by one worker via a Redis lock; stale-while-revalidate serves old content during the refresh. A 2 second in-process copy sits in each pod. Stock shows buckets (more than 50, a few left, sold out) from a counter updated asynchronously. Purchase still checks inventory in PostgreSQL.
Consequences
Good: the database sees about one query per rebuild, not thousands. Bad: the page can be a few seconds stale, and the stock label can lag reality, so the buy button must handle "sold out" gracefully.
Revisit when
full-page CDN caching becomes worthwhile: origin requests exceed what the storefront pods can serve at p99 under 300 ms during a sale, even with the key-level fixes in place.
Rollout and verification
The team would ship behind a flag per product. Step one: add the lock-protected early refresh and jitter in shadow mode, logging when it would have rebuilt. Step two: canary on a low-traffic product for a day. Step three: pre-warm the hero key five minutes before the announcement, and run a load test that ramps to 2x the last peak, including a forced key deletion during the ramp.
Dashboards: hit ratio per key family, origin requests per second, database connections in use, rebuilds per minute (should be near the number of keys, not the request rate), per-node Redis CPU and p99 for hits and misses separately. Target: database query rate flat during the ramp.
The 2 a.m. failure
A deploy at the start of a sale restarts the Redis cluster for a config change, so the cache is empty exactly when traffic arrives. The page reads: "DB connections above 90 percent, cache hit ratio below 20 percent". Detection: cache_hit_ratio, db_connections_in_use and rebuild lock wait time all fire.
sequenceDiagram
participant S as Shoppers
participant A as App pods
participant R as Redis
participant D as Database
S->>A: 40000 requests per second
A->>R: get hero key
R-->>A: miss after restart
A->>D: rebuild, lock holder only
A-->>S: stale in-process copy or queued response
First three checks: redis-cli info stats on each node for hit and miss counters; the Grafana panel for rebuilds per minute; SELECT application_name, state, count(*) FROM pg_stat_activity GROUP BY 1, 2 to see whether connections are held by rebuilds or by checkout. Mitigation: enable the load-shedding flag that returns the static fallback page for non-hero products, run the pre-warm job by hand, and raise the lock timeout only if the rebuilder is slow. Follow-up fix: make the pre-warm step part of the deploy pipeline and block cache restarts during announced sales.
Architect's notes
Ask what the empty-cache number is. In a design review, require the team to state database load with a cold cache. If the answer is "it falls over", the cache is a hidden single point of failure.
Measure per-key, not per-cluster. Hit ratio of 95 percent overall can hide one hot key and one saturated node. Tag metrics by key family.
Cap the lock and the stale window. Both are policies: how long may a rebuilder hold the lock, and how stale may a served value be. Write the numbers down.
Exactness belongs to the database. Cache descriptions and prices; do not let a cached stock count decide whether to accept an order.
Ownership trap. Cache invalidation spans teams: the pricing team writes, the storefront team caches. Name one owner for each key family and its TTL.
Defer full-page edge caching until the key-level fixes are measured; it introduces a purge pipeline you then have to operate.
Common mistakes
Identical TTLs for bulk-loaded data. Everything expires together. Add jitter.
Updating the cache from writers. Concurrent writers can leave the older value. Delete and let the next read repopulate.
Invalidating before committing. A reader can re-cache the old row. Delete after commit.
No plan for a cold cache. The database takes the full load after a flush. Rate limit, coalesce and pre-warm.
Caching without a TTL. A lost invalidation stays wrong forever. Always set an upper bound.
Using write-back for data you cannot lose. Unflushed writes vanish with the node. Use a durable queue or write-through.
Watching only the average. Hot keys hide behind a healthy cluster average. Monitor per node.
Key takeaways
A cache is a second copy. Every technique exists to control how far it can drift from the source of truth.
Pick the write policy by the cost of loss. Write-through favors freshness, write-back speed at the risk of loss, write-around write-heavy data.
Delete after commit, and keep a TTL. Together they give bounded staleness even when an invalidation fails.
Stampedes are prevented by coordination. Coalesce, lock rebuilds, jitter TTLs, refresh early or serve stale while revalidating.
Hot keys need a different tool. Local caches, key replication and read replicas spread load hashing cannot.
Consistent hashing limits damage. Changing the node count moves a small share of keys.
Cache the absence too. Short negative entries and Bloom filters protect the database from missing-data lookups.
Design for the empty cache. If the system cannot survive a cold start, the cache is a hidden single point of failure.
Practice exercises
Exercise 1 — Starter (30 min): For five kinds of data in an online shop (product description, price, stock count, user cart, order history), choose a write policy, a TTL and an invalidation approach, and justify each in one sentence.
Exercise 2 — Intermediate (2 hours): Assume a key receives 2,000 requests per second, rebuilds in 150 ms, and expires every 60 seconds. Calculate how many requests pile onto the database at each expiry with no protection. Then show how much each of the following reduces that number: in-process coalescing across 10 servers, a distributed lock, and early refresh. State your assumptions.
Exercise 3 — Advanced (half day): Design the caching layer for a flash-sale page with a CDN, an in-process cache and a shared cache, plus a database. Specify TTLs for each level, how a price correction propagates and how long shoppers may see the old price, how you handle a product that suddenly gets 100 times normal traffic, and what happens when the shared cache cluster is lost. Draw the request path and the invalidation path.
Lesson 5 of 14 · System Design Building Blocks: Data, Messaging and Scale
Go beyond cache-aside: write policies, invalidation, stampedes, hot keys, distributed caches, negative caching and multi-level caches, with the failure modes of each.
In the rest of this lesson
Cache stampede and its fixes
Hot keys
Distributed caches and consistent hashing
Negative caching
Multi-level caches
Keeping the cache and database consistent
Enrolling is free. It unlocks every lesson in System Design Building Blocks: Data, Messaging and Scale, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.