QuickCart Stage 2: Scaling Reads With Balancers, Caches and a CDN

Three more features matter. Connection draining (AWS: deregistration delay, 300 seconds by default) lets in-flight requests finish on a target being removed. Slow start (AWS: 30 to 900 seconds, off by default) ramps up a new target's share. Fail-open: AWS can send traffic to all targets, healthy or not, when too few are healthy, because a degraded answer often beats none.

Figure 4. QuickCart's fleet behind a balancer

How to read this: solid arrows are requests; the dashed arrow is the balancer probing each server on a timer. A server that fails its probes leaves rotation without any shopper noticing.

Growing and shrinking the fleet

Auto-scaling

Auto-scaling changes the number of servers automatically from a measured signal, under a policy, between a minimum and maximum. CPU suits CPU-bound rendering but misleads when the database is the bottleneck; requests per server suits web tiers once you have load-tested a target; queue length suits workers. QuickCart starts with requests per server.

The arithmetic. The Kubernetes Horizontal Pod Autoscaler documents a proportional rule: desired replicas = ceil[current replicas x (current metric / desired metric)]. With 6 servers at 90 percent CPU against a 60 percent target, 6 x 90 / 60 = 9, so scale to 9. The same document skips scaling when the ratio is within a tolerance of 1.0 (0.1 by default), so 64 percent against 60 percent (ratio about 1.07) changes nothing.

Cooldowns and stabilisation. New servers take time to start, so a scaler that reacts at once overshoots. A cooldown pauses further scaling after an action: Amazon EC2 Auto Scaling's simple scaling policies use a 300-second default, while target tracking and step scaling use instance warm-up instead. A stabilisation window uses the highest recent recommendation before scaling down; the Kubernetes documentation gives a default downscale window of 5 minutes. The principle: scale out fast, scale in slowly.

The dangers of scaling in. Removing servers is where outages start. Without connection draining a terminated server drops its in-flight requests. Anything kept only in the server vanishes (the second reason for statelessness). Scaling out, in and out again within minutes (flapping) wastes money and chills caches, so use wider thresholds for scale-in.

There is also a danger on the way out. More servers means more database connections. At an illustrative 20 connections per server, 12 servers hold 240 and 40 servers hold 800. A database healthy at 12 servers can fall over when a scaler reacts to a slow database by adding servers. Always set a maximum the database can survive. That loop is the star of the 2 a.m. failure below.

Caching in layers

Cache layers

A cache keeps a copy of a result so the next request can skip the expensive work (Caching Basics explains why: locality and skew). A request can be answered at four levels, and the closer to the shopper, the cheaper.

Figure 5. The path of a product page request through the layers

How to read this: each arrow is a miss. A hit at any layer ends the journey there, so everything to its right is never touched. The leftmost hit is the cheapest.

Sizing the read path. Take the 2,300 requests per second peak and assume (illustrative) 3 lookups per page request: product details, the store's price and stock, a rating summary. That is 2,300 x 3 = 6,900 lookups per second. The hit ratio is hits divided by lookups. With the caching lesson's illustrative latencies (1 ms for a hit, 21 ms for a miss):

Going from 90 to 99 percent cuts database reads tenfold. The fleet follows the same arithmetic: if one app server handles 200 requests per second (illustrative), 2,300 / 200 = 11.5 means 12 servers, and 14 leaves room to lose two.

Hit ratio varies by key: a product read 0.5 times per second with a 30-second TTL gets 15 reads per window and at most one miss (about 93 percent), while a product read once a minute gets nearly none.

Cache strategies

A cache strategy decides who loads the cache and how writes reach it.

QuickCart uses cache-aside almost everywhere: it degrades safely when Redis is down. Pseudocode in TypeScript (cache is any client with get, set(key, value, ttlSeconds) and del; db is a placeholder):

// Pseudocode: cache-aside for one product
async function getProduct(id: string): Promise<Product | null> {
  const key = `product:v1:${id}`;
  const cached = await cache.get(key);
  if (cached !== null) return JSON.parse(cached) as Product; // hit

  const row = await db.getProduct(id);                        // miss
  if (row !== null) await cache.set(key, JSON.stringify(row), jitteredTtl(600));
  return row;
}

async function updateProductTitle(id: string, title: string) {
  await db.updateTitle(id, title);       // source of truth first
  await cache.del(`product:v1:${id}`);   // then drop the stale copy
}

The v1 in the key lets you invalidate a whole kind of entry by changing the version, and only non-null results are stored. jitteredTtl is defined below. Deeper patterns: Advanced Caching.

Eviction policies

A cache has finite memory, so a new entry can force an old one out. The eviction policy picks the victim; a TTL separately removes entries by age (Redis removes expired keys when a client reads them and also by sampling in the background).

• LRU (least recently used): evict the entry used longest ago. The bet is recency.
• LFU (least frequently used): evict the entry with the fewest uses. The bet is stable popularity.

A cache holds 3 entries. Requests in order: milk, eggs, milk, bread, apples, rice, milk. Milk is the staple; the rest are a scan of one-time products.

LRU gets 1 hit in 7, LFU gets 2: the scan pushed milk out of the LRU cache, while its count protected it under LFU. (Ties on one use evict the older entry, a common tie-break.) LFU's weakness is the reverse: an item hot last month keeps a high count, so real implementations decay counts over time.

Redis sets the choice with maxmemory-policy. Per its documentation, allkeys-lru is a good default when a subset of keys is read far more than the rest; allkeys-lfu exists from Redis 4.0; volatile-ttl evicts the shortest remaining TTL among keys that have one; noeviction errors on writes that need memory. Redis approximates LRU and LFU by sampling. On 64-bit systems maxmemory is 0 by default (no limit), so set one before expecting eviction.

Working set (illustrative): 100,000 products at 2 KB is 200 MB, which fits. Prices and stock vary by store: 5,000 stores x 2,000 stocked products (Stage 1) is 10 million entries, about 1 GB at 100 bytes each before overhead.

Cache invalidation and stampede

Invalidation keeps the cached copy and the source of truth in agreement. Three tools, usually combined: expire by TTL; delete on write; and version the key (product:v2:..., or a version in an image URL). Races remain: a reader fetches the old row, a writer updates the database and deletes the key, then the reader stores its old row. That stale entry lives until its TTL, which is why every entry keeps one.

A cache stampede (thundering herd) happens when a hot key expires and many requests miss at once, all rebuilding the same value. If the "fresh produce" list is read 400 times per second (illustrative) and takes 250 ms to rebuild, about 400 x 0.25 = 100 requests miss together and send the same query to the database, which slows the rebuild and invites more misses.

Figure 6. A stampede, and request coalescing stopping it

How to read this: all 100 requests miss, but only one query leaves the app server. Without coalescing each would send its own.

The defences:

• Request coalescing (single flight). The first miss rebuilds; others wait for its result.

const inFlight = new Map<string, Promise<unknown>>();

async function singleFlight<T>(key: string, load: () => Promise<T>): Promise<T> {
  const running = inFlight.get(key) as Promise<T> | undefined;
  if (running) return running;            // join the rebuild in progress
  const p = load().finally(() => inFlight.delete(key));
  inFlight.set(key, p);
  return p;
}

This works per process, so 12 servers can still run 12 rebuilds, down from 100 but not 1. For one rebuild across servers, use a short Redis lock: SET lock:key token NX EX 10 returns OK to the one caller that got it and a null reply to the rest, and the expiry stops a crashed rebuilder blocking everyone. Release only if the stored token is yours. Redis's documentation prefers its Redlock algorithm for stronger guarantees; for a rebuild lock, where the worst case is one extra rebuild, the simple form usually suffices.

• Jittered TTLs. Randomise expiry so keys loaded together do not expire together:

function jitteredTtl(baseSeconds: number, spread = 0.1): number {
  const factor = 1 - spread + Math.random() * 2 * spread; // between 0.9 and 1.1
  return Math.round(baseSeconds * factor);
}
// jitteredTtl(600) returns a whole number of seconds from 540 to 660

• Stale-while-revalidate. Serve the expired value while one background request fetches the new one (RFC 5861 defines the HTTP form; in your own cache, store a "soft" expiry beside the value).
• Pre-warming. Load the hottest keys before a known event or after a restart.

Distributed caches: Redis and Memcached

An in-process cache has a flaw at fleet scale: each server holds its own copy, copies disagree, and each warms up separately. A distributed cache is one service all servers share.

QuickCart picks Redis because one service serves sessions (per-key expiry), caches and, later, counters; for a plain cache Memcached would also do. Either way the cache is a dependency, so set a timeout on every call.

Spreading keys over nodes. The naive rule hash(key) mod N has a nasty property: going from 3 nodes to 4 keeps only keys where h mod 3 equals h mod 4, which is 1 in 4, so about 75 percent of keys move and the cache goes cold at once. At 6,900 lookups per second that sends most of them to the database at once. Consistent hashing places nodes on a ring of hash values; a key belongs to the first node clockwise from its hash. Adding a fourth node takes over only the arc beside it, so about 1 in 4 keys move. See Sharding; Stage 4 applies the idea to the orders database.

CDN and edge for product images

A CDN (content delivery network) is a set of cache servers in many edge locations. A shopper's request goes to a nearby edge. On a hit the edge returns the file; on a miss it fetches from the origin, keeps a copy and returns it, so the origin sees roughly one request per file per edge, not one per shopper. Images are large, identical for everyone and rarely change. With an illustrative 10 images per page view, 2,300 page requests per second become about 23,000 image requests per second; at an illustrative 95 percent CDN hit ratio about 1,150 per second reach the origin. Servers are stateless, so the origin cannot be one server's disk: here it is an image endpoint on the app tier that reads bytes from a database table, and Stage 3 moves them to object storage. See CDN and Object Storage.

Figure 7. Where images and API calls go

How to read this: shoppers only talk to the edge. Image hits never leave it; misses travel to the origin, and API calls go through the balancer.

The origin tells every cache how long it may keep a file in the Cache-Control response header. QuickCart uses two policies. For product images, with the version in the file name (/img/p/1042-v7.webp):

Cache-Control: public, max-age=31536000, immutable

max-age=31536000 is a year in seconds, public allows shared caches, and immutable says the file will not change while fresh. A new photo gets a new name (-v8), so nothing needs purging. For the public product list JSON:

Cache-Control: public, max-age=0, s-maxage=30, stale-while-revalidate=60

s-maxage=30 applies to shared caches only and overrides max-age; the 0 makes a browser's copy stale at once. After 30 seconds the CDN may serve the old copy up to 60 more seconds while it refreshes in the background (RFC 5861). Support for stale-while-revalidate varies by CDN and browser; check your provider's documentation.

Never send carts, orders, account pages or responses that set cookies to a shared edge cache: mark them private or no-store. A cached page can show a price up to 30 seconds old, so checkout always reads the database for the price it charges.

Read replicas: a first look

A read replica is a second database that receives the primary's changes and serves read-only queries. In PostgreSQL it is a standby using streaming replication, which is asynchronous by default: a committed transaction may not yet be visible on the standby (the documentation says typically under one second when the standby keeps up). That causes read-your-writes surprises: a store updates a price, the page reloads from a replica, and the old price appears.

QuickCart's first rule: browsing, category lists and store listings go to the replica; reads right after the same user's write, checkout and payment reads, and stock checks go to the primary. Leaders, followers, lag and failover belong to Scale Writes and Replication.

Decisions and trade-offs

In practice: the Saturday promotion at 100,000 users

The situation

QuickCart is in Stage 2 with 100,000 monthly active users and a normal peak of about 23 requests per second. A Saturday push notification, "Free delivery today", is expected to raise the peak 10 times, to about 230 requests per second, within minutes. The single server tops out near 100 page requests per second when it also serves images (illustrative), and database CPU is already high at normal peaks.

Starting architecture

Figure 8. Where the load lands at the promotion peak, before the change

How to read this: 10 images per page add about 2,300 image requests per second and 3 lookups per page add about 690 database reads per second, on a server limited to about 100 page requests per second.

Options considered

The decision

The maximum of 4 is deliberate: at 20 connections per server (illustrative) the database carries at most 80.

Rollout and verification

1. Add the balancer and a second server, with sticky routing as a bridge; then move sessions to Redis and turn stickiness off. Sign in and refresh 20 times: the shopper stays signed in whichever server answers.
2. Run the cache in shadow mode (read it, still serve from the database, log mismatches), then serve from it for 5, 25, 50 and 100 percent of requests, watching database reads per second and p95 and p99 latency.
3. Put the CDN in front of images and check that a repeat request shows a cache-hit indicator.
4. Rehearse failure. In staging stop Redis and confirm pages load from the database; drain one server and confirm no request fails.

The 2 a.m. failure

Redis is restarted for maintenance at 02:00 and returns empty while the nightly catalogue import is slowing the database. Every request misses.

Figure 9. The cold-cache failure loop

How to read this: the cold cache starts the loop and the auto-scaler feeds it. More servers open more connections to an already slow database.

The page reads "p99 above 3 seconds, cache hit ratio below 20 percent". Detection: Redis keyspace_hits and keyspace_misses (INFO stats) show the hit ratio near zero while database reads climb. First three checks: 1) INFO server on Redis (an uptime reset proves a restart); 2) database connections and slowest queries (identical product queries?); 3) the scaler's event log (did it reach its maximum of 4? The cap kept connections at 80).

Mitigation: pause the import, serve category pages from a stale copy behind a flag, and run the warm-up script for the most viewed products. Follow-ups: jittered TTLs, single-flight rebuilds, Redis with a replica and failover, a timeout on every cache call, and no maintenance during imports.

Architecture after this stage

Figure 10. QuickCart after Stage 2 (new components in the accent colour)

How to read this: accent boxes are new in this stage; the primary database is unchanged. Requests flow left to right and stop at the first layer that can answer. Dashed arrows are background work.

What QuickCart looks like now

Architect's notes

• Remove structural blockers before chasing throughput. At 23 requests per second QuickCart needed no capacity; it needed to stop being one machine with a memory.
• Measure the skew first: how many products cover 80 percent of requests predicts the hit ratio.
• Decide freshness per field, and ask what happens on a cold cache. If the answer is "the database falls over", test it.
• Cap auto-scaling at what the database survives, and watch hit ratio per layer (Observability and SLOs).

Common mistakes

• State left in a server while adding servers: shoppers are logged out at random.
• A deep health check that fails every server when one dependency blips and empties the fleet.
• Auto-scaling on a signal that is not the bottleneck, or with no maximum.
• Caching without a TTL, or caching errors. A blip becomes a ten-minute outage.
• Leaving Redis maxmemory unlimited, modulo hashing then resizing, and reading the checkout price from a cache.

Key takeaways

• Scale out the stateless parts; keep stateful parts few. Sessions go to a shared store.
• Layer 7 balancers read HTTP. Least connections for varied cost, weights for unequal servers, hashing for warm per-key caches.
• Keep health checks shallow, require consecutive results, drain on removal.
• Auto-scaling needs a good signal, cooldowns and limits: scale out fast, in slowly, and cap at what the database holds.
• The leftmost cache hit is cheapest; the last points of hit ratio protect the database most.
• Cache-aside with TTL, jitter and delete-on-write is the default; write-back only for data you can afford to lose.
• LRU bets on recency, LFU on frequency, TTL bounds staleness. Give Redis a maxmemory and a policy.
• Defend against stampedes with coalescing, jitter and stale-while-revalidate.
• CDN for large, shared, rarely changing files with versioned URLs; never for personal data.

Practice exercises

Exercise 1, starter (30 min): Redo the eviction trace with a 2-entry cache and the sequence milk, eggs, bread, milk, eggs, milk, counting LRU and LFU hits.

Exercise 2, intermediate (1 hour): At 1,150 requests per second (Stage 4, illustrative) with 3 lookups per request, compute database reads per second at 80, 90 and 95 percent hit ratios, and the servers needed at 200 requests per second each.

Exercise 3, intermediate (2 hours): For ten QuickCart items (description, image, price, stock, rating, cart, search results, delivery estimate, order status, session) choose layer, TTL, invalidation and strategy, and justify any item you refuse to cache.

Exercise 4, advanced (half day): Run Redis locally with a service whose database call sleeps 100 ms. Implement cache-aside, then fire 200 concurrent requests at one expired key with and without single flight, counting database calls. Compare allkeys-lru and allkeys-lfu under maxmemory 10mb on a skewed request pattern.

Related reading

• Stage 1: The Brief — the numbers and first architecture.
• Stage 3: Choose the Data Model — object storage, the catalogue and search.
• Stage 4: Scale Writes — replication, sharding and consistent hashing.
• Stage 9: Survive Failure — timeouts, circuit breakers and degradation.
• Level lessons: Scaling Basics, Load Balancing, Caching Basics, Advanced Caching, CDN and Object Storage, Replication and Estimation.

Official documentation

• Redis: key eviction — policies, approximated LRU and LFU.
• Memcached documentation — overview, design philosophy, client-side server selection.
• RFC 5861: HTTP Cache-Control extensions for stale content — stale-while-revalidate.
• AWS: target groups for Application Load Balancers — algorithms, slow start, deregistration delay.
• Kubernetes: Horizontal Pod Autoscaling — scaling formula, tolerance, stabilisation.
• PostgreSQL: log-shipping standby servers — streaming replication and hot standby.