System Design: Caching Basics

Reading: ask the cache for key product:123; on a hit, return it; on a miss, read the row from the database, store it in the cache with a TTL (time to live, an expiry time), and return it.

Writing: update the database (the source of truth), then delete the cached entry so the next read refetches fresh data.

The first request pays for the database read and fills the cache; every request after it, until the TTL passes or the entry is deleted, is a hit. Here is a minimal Redis version, deliberately short.

import json

TTL_SECONDS = 600  # assumption: ten minutes of staleness is acceptable for descriptions

def get_product(product_id):
    key = f"product:{product_id}"
    cached = redis.get(key)
    if cached is not None:
        return json.loads(cached)          # cache hit

    row = db.query("SELECT * FROM products WHERE id = %s", product_id)
    if row is not None:
        redis.set(key, json.dumps(row), ex=TTL_SECONDS)
    return row                             # cache miss path

def update_title(product_id, title):
    db.execute("UPDATE products SET title = %s WHERE id = %s", title, product_id)
    redis.delete(f"product:{product_id}")  # invalidate; next read repopulates

Why delete instead of update on a write? If two writers update the same row at nearly the same time, their cache updates can land in a different order from their database updates, leaving the older value in the cache until the TTL expires. Deleting is simpler and tends to be safer, though races remain (a reader that fetched the old row just before the write can store it just after the delete), which is one reason every entry also gets a TTL. We explore these races and other write strategies in advanced caching.

TTL and eviction

Two separate mechanisms control what stays in a cache.

TTL (time to live)

A TTL is an expiry time attached to an entry. When it passes, the entry is removed or treated as a miss. TTL is your safety net for staleness: even if you forget to invalidate, wrong data lives at most as long as the TTL. Choosing a TTL is a business question, not a technical one: how stale can this data be before someone is harmed or annoyed?

Eviction policies

A cache has finite memory. When it fills, it must evict something.

• LRU (least recently used): discard the item not used for the longest time. A good general default because of temporal locality.
• LFU (least frequently used): discard the item used least often overall. Good when a stable set is consistently popular; slow to forget items that were once hot.
• FIFO discards the oldest entry, ignoring usage; random is cheap and sometimes adequate.

Many systems, including Redis, offer several configurable policies, and some use approximate versions of LRU or LFU to save memory. Check your cache's documentation for exact behaviour.

An entry is only ever safe to serve while Fresh. Every arrow out of Fresh is either a cost you chose (TTL, invalidation) or one you did not (eviction), and a healthy design copes with all of them.

What to cache and what not to

Walk any candidate through these three questions. Good candidates are read far more than written, expensive to compute, and shared by many users. Poor candidates include data that must always be exact (inventory at checkout, permissions that must apply immediately), data rarely re-read, very large objects that evict many small hot ones, and anything where an error response could be cached by mistake and served for the whole TTL.

Staleness and invalidation basics

Cache invalidation has a reputation as one of the hard problems in computer science, and the reason is simple: the cache and the source of truth are two copies of the same fact, and keeping two copies in agreement is hard when either can change at any time. You have three basic tools, usually combined.

1. Expire by TTL. Easiest, guaranteed to converge, but allows staleness up to the TTL.
2. Invalidate on write. When the data changes, delete or update the cached copy. Fresher, but you must find every place that writes and every key affected (a product appears on its page, in category lists and in search results).
3. Version the key. Include a version or content hash in the key or URL, such as image_v7.jpg. When data changes you use a new key and the old one simply expires. This avoids invalidation entirely and suits static files.

Caching exchanges freshness for speed. Decide the freshness you can afford first, then choose the simplest tool that provides it.

Measuring a cache

A cache you do not measure may be hurting you. Track at least these:

Also set a timeout on cache calls. A hung cache that blocks every request is worse than no cache.

In practice: caching product pages for an online marketplace

The situation

An online marketplace lists about five million products from independent sellers. A Java application and PostgreSQL (one primary, one replica) serve the web and mobile apps; Prometheus and Grafana monitor them. Six engineers own the storefront. A product page needs the product row, the seller summary, the rating aggregate, and the current price and stock, which is four queries and a join. At a peak of about 600 page views per second (illustrative), database CPU sits near 80 percent, p95 page time is about 700 ms, and a seller campaign can tip the primary over. A small share of products (the top few percent, illustratively) draws most views.

Starting architecture

Every page view repeats the same four queries, even for a product that thousands of people have just viewed.

Options considered

The decision

The target is a hit ratio of at least 90 percent on product keys and at least 70 percent on price keys, which you would verify rather than assume.

Rollout and verification

Ship it in three steps behind a flag. First run in shadow mode: the app still serves from the database but also reads the cache and logs any difference, so you can measure the mismatch rate before users see anything. Then serve from the cache for 5 percent of requests, then 25, 50 and 100, watching the signals. Finally, force a failure in staging: stop Redis and confirm pages still load from the database.

Shoppers read through steps 1 to 3; a seller edit goes through the same app, which updates the database first (A) and only then deletes the cached key (B).

Dashboards to build: hit ratio per key family, database queries per second (should fall visibly), page p95 and p99, Redis memory and eviction rate, and the shadow-mode mismatch count.

The 2 a.m. failure

At 02:00 the managed Redis node restarts for maintenance and comes back empty. Every request misses at once.

The page says "database CPU above 90 percent and product page p99 above 3 s". Detection: hit ratio drops to near zero while database queries per second jump. The first three checks: INFO stats (keyspace hits and misses), INFO keyspace (key count near zero) and INFO server (uptime reset) on Redis, the database pg_stat_activity view (many identical product queries), and the app's cache error counters. Mitigation: enable the feature flag that sheds low-priority traffic or serves a simplified page, scale the app only if the database has headroom, and run a warm-up script that preloads the most viewed products. Follow-up fixes: add jitter to TTLs so keys do not expire together, coalesce concurrent misses for the same key into one database read (see advanced caching), and run Redis with a replica and automatic failover.

Architect's notes

• Measure the skew first. Pull a day of page views and compute how many products cover 80 percent of traffic. That number sizes the cache and predicts the hit ratio before you build anything.
• Decide freshness per field, not per page. Description, rating and price have different business tolerance. One TTL for the whole page is a smell.
• Ask what happens on a cold cache. If the answer is "the database falls over", you have a hidden dependency on the cache, not an optimisation. Test it.
• Cache keys are an API. Include everything the response depends on (language, currency, seller region). Review keys like you would review a public interface.
• Name an owner for invalidation. Whoever writes the data owns deleting the key. In a design review, ask for the list of write paths that touch cached fields.
• Defer clever write strategies. Cache-aside with TTL and delete-on-write covers most needs. Add write-through or refresh-ahead only when a measured miss cost justifies it.

Common mistakes

• Caching without a TTL. Entries live until evicted, so stale or wrong data can persist for days. Set a TTL on practically every entry, even if it is long.
• Assuming the cache is always right. It is a copy. Treat the database as the truth and design for the cache being empty, stale or down.
• Shared keys for personal data. Leaks information between users. Build keys that include everything the response depends on.
• Caching errors or empty results blindly. A failed lookup stored for ten minutes turns a blip into a long outage. Cache only successes, or give negatives a much shorter TTL.
• No plan for a cold cache. After a restart or deploy, every request misses at once. Warm the cache, stagger expiries and be ready to shed load.
• Optimising before measuring. Adding a cache to a query that is not a bottleneck adds complexity for no gain.
• Never checking hit ratio. A cache with a 10% hit ratio is mostly overhead. Monitor and tune it.

Key takeaways

• A cache trades freshness for speed. Decide how stale is acceptable before you design it.
• Locality makes caching work. Repeated and skewed access means a small cache can serve most traffic.
• Cache near the user when safe. Browser, CDN, application and database layers each help in different ways.
• Cache-aside is the default pattern. Read the cache, fall back to the database on a miss, then populate; on writes, update the database and delete the cached copy.
• TTL is your safety net. Even with invalidation, set an expiry so mistakes heal themselves.
• Eviction manages memory. LRU is a sensible default; LFU suits stable popularity patterns.
• Do not cache everything. Skip data that must be exact, rarely re-read, or personal under shared keys.
• Measure hit ratio and database load. If the database load did not fall, the cache is not paying its way.

Practice exercises

Exercise 1 — Starter (30 min): A cache hit takes 2 ms and a miss takes 52 ms (assumptions). Compute the average latency for hit ratios of 50%, 80%, 90% and 99%. Tabulate the results and write two sentences about what you notice about the benefit of going from 90% to 99%.

Exercise 2 — Intermediate (2 hours): For a marketplace, make a table of ten data items (product description, image files, price, stock count, seller profile, review count, shopping cart, search results, shipping estimate, order status). For each, decide whether to cache it, at which layer, with what TTL and what invalidation approach, and justify any item you refuse to cache.

Exercise 3 — Advanced (half day): Run Redis locally (Docker works) and write a small service with a slow function that sleeps 100 ms to simulate a database query. Implement cache-aside, then measure latency and hit ratio under repeated requests drawn from a skewed distribution (for example 80% of requests for 20% of keys). Try three TTLs and two cache sizes with an LRU eviction setting, and report how hit ratio and average latency change. Note any surprising result.

Related reading

• System Design: Scaling Basics — where caching fits among the first scaling moves.
• System Design: CDN and Object Storage — caching at the edge for images and video.
• System Design: Estimation — using latency numbers to reason about cache benefits.
• System Design: Advanced Caching — write strategies, stampedes and hot keys.
• Redis documentation — official reference for data types, TTLs and eviction configuration.