System Design: Load Balancing

Health checks and connection draining

A load balancer is only useful if it avoids dead servers. It does so with health checks.

• Passive checks watch real traffic. If a backend returns errors or times out repeatedly, it is marked unhealthy.
• Active checks send a probe on a schedule, for example GET /health every few seconds. A backend that fails several probes in a row is removed. One that passes several in a row is added back.

Requiring several consecutive results avoids flapping, where a server is yanked in and out of rotation by a single blip.

What should the check test? A "shallow" check says "the process is up and can serve HTTP". A "deep" check also verifies it can reach its database. Deep checks catch more, but if the shared database has a brief problem and every server's deep check fails, the balancer may remove all servers at once and turn a partial problem into a total outage. Many teams use a shallow check for traffic routing and report deeper dependencies through monitoring.

Connection draining (also called deregistration delay) handles planned removal. When you deploy or scale in, you do not want to cut off a customer mid-payment. The balancer stops sending new requests to the server but lets existing ones finish, up to a time limit. Then the server can be stopped safely.

A server only receives traffic in the Healthy state. Starting and Draining are the two states that deployments live in, and most deployment errors are bugs in one of those transitions.

Sticky sessions, and why to avoid them

Sticky sessions (session affinity) make the balancer send a given user to the same server every time, usually through a cookie or an IP hash. Teams reach for them when the app stores login state in server memory: if the user hits a different server, they appear logged out.

The problem is what stickiness costs. A few heavy users pinned to one server can overload it while others idle. When that server dies, everything it held in memory is lost and its users lose work. New servers receive traffic only from new users, and draining takes longer.

The better fix is to make servers stateless: keep session data in a shared store such as Redis, or use signed tokens that the client carries. Stickiness is acceptable in narrow cases, such as a long-lived WebSocket whose state lives on the server holding it, or a backend cache you want to keep warm, but treat it as an optimisation, never as the place where the only copy of important data lives.

TLS termination and other jobs

Because it sits at the front, a layer 7 load balancer is a convenient place to do shared work.

• TLS termination. The balancer decrypts HTTPS, so backends handle plain HTTP inside the private network. This centralises certificates and saves backend CPU. If compliance requires encryption inside the network too, re-encrypt to the backends or use mutual TLS between services.
• Retries and timeouts. It can retry a failed idempotent request (one that is safe to repeat) on another server. Be careful retrying non-idempotent requests like "place order".
• Rate limiting and basic protection. It can reject abusive traffic early, a topic covered in rate limiting.
• Path-based routing. One domain can fan out to different pools.

Do not overload it. The more logic you pack into the balancer, the more it becomes a bottleneck and a place where one bad configuration breaks everything.

Making the load balancer highly available

If there is only one balancer, we have just moved the single point of failure. A highly available design needs redundancy at this layer too. Three approaches exist, and they are often combined.

1. An active-passive or active-active pair sharing a virtual IP address. In active-passive, the standby watches the primary with heartbeats and takes over the address if the primary stops responding.
2. DNS with several addresses, each pointing at a separate balancer. Unhealthy addresses can be removed by DNS health checks, but answers are cached for the record's TTL, so changes take time to reach everyone.
3. Anycast, where the same IP address is announced from several locations. Routing sends each client to a nearby one, and a failed location withdraws its announcement. Large global services and CDNs use this heavily.

Managed cloud load balancers hide most of this: the provider runs many balancer nodes across data centres and you see a single stable name. That is a big reason to prefer them unless you need routing logic they cannot express.

In practice: a retail site with bumpy deployments

The situation

A mid-size online retailer runs its storefront as one Spring Boot application on Kubernetes: 12 identical pods that serve browsing, search and checkout together. A cloud layer 4 balancer feeds an Nginx ingress controller, which spreads requests round robin across the pods. PostgreSQL holds orders, Redis holds sessions, and Prometheus with Grafana provides metrics. A platform team of three supports about 25 application engineers, who deploy several times a day.

The pain, with illustrative numbers: peak traffic is around 1,200 requests per second. Browse requests take about 60 ms, but checkout calls a payment provider and takes two to three seconds. During every rolling deploy the 5xx rate climbs to about 1.5 percent for two minutes, and p99 latency doubles. Some pods have most of their request threads tied up waiting on the payment provider, and show high latency, while neighbours sit nearly idle, because round robin keeps handing each pod its turn regardless of how many slow checkouts it is already holding.

Starting architecture

Every path shares one pool and one algorithm, so a burst of slow checkouts competes directly with fast browsing for the same pods.

Options considered

The decision

Rollout and verification

Ship the changes behind a staged rollout. First the shutdown behaviour: the pod handles termination by failing its readiness check, sleeping a few seconds so the ingress has time to remove it from rotation, then finishing in-flight requests within the grace period. Then the algorithm change on a canary ingress weighted to a small slice of traffic.

The preStop sleep exists because endpoint removal takes a moment to reach every ingress replica; without it, the ingress can keep sending requests to a pod that has already started shutting down.

Verify with a load test that deploys during steady traffic, then watch Grafana: 5xx rate per deploy (target near zero), per-pod load spread, meaning in-flight requests and CPU, between busiest and idlest pod (target a narrow band), p99 for browse and checkout separately, and in-flight requests per pod. Alert if 5xx exceeds a small threshold for two consecutive minutes during a rollout.

The 2 a.m. failure

The readiness endpoint was made "deep" during the work: it runs SELECT 1. At 02:10 PostgreSQL fails over to a replica and connections drop for about 30 seconds.

The page says "storefront 5xx above 50 percent". Detection: the ready-endpoint count for the service (from kube-state-metrics) drops to zero while pods are still running. The first three checks: kubectl get pods (all Running but not Ready), kubectl get endpoints storefront (empty, or look at its EndpointSlices), and kubectl describe pod (readiness probe failures). Mitigation: pods rejoin by themselves once SELECT 1 succeeds again, so confirm the database failover has completed first. If the database stays down for long, switch readiness to the shallow check (through a runtime setting if the app has one, otherwise a normal rollout) so pods can serve cached pages and clear errors for database-backed calls. Do not restart pods in bulk. Follow-up: split liveness, readiness and dependency health, keep readiness shallow, and alert on "ready endpoints below half of desired" so this is caught before it is total.

Architect's notes

• Measure the shape of traffic before choosing an algorithm. Request duration distribution (p50 versus p99 per path) tells you whether round robin is fine or least connections pays off. Do not guess.
• Deploy failures are usually shutdown bugs, not startup bugs. In a design review, ask: what happens to a request that arrives 100 ms after the pod receives SIGTERM, and how long is the grace period compared with the slowest request?
• Ask who owns the probe. Health endpoints written by application engineers and consumed by the platform team are a classic ownership gap. Decide who reviews changes to them.
• Warm-up matters. A fresh pod with a cold JVM and empty caches gets a full share of traffic at its worst moment. Ask whether your balancer supports a slow-start ramp.
• Defer the pool split until a number forces it. Two pools double the capacity planning and the alert surface. Set a measurable trigger, as in the ADR above, and wait for it.
• Retries multiply load. Confirm the ingress only retries idempotent requests, and that retries have a budget, or a slow backend gets hit harder precisely when it is struggling.

Common mistakes

• Treating the load balancer as magic. It spreads load, but if every server depends on one overloaded database, the database is still the bottleneck. Find the real bottleneck first.
• A single load balancer. It becomes the new single point of failure. Use a redundant pair, multiple nodes or a managed service.
• Relying on sticky sessions for state. A failed server loses sessions. Externalise state instead.
• Health checks that are too deep or too shallow. Too shallow and broken servers keep receiving traffic; too deep and one shared dependency hiccup removes the whole fleet.
• Blind retries on non-idempotent requests. Retrying "charge the card" can duplicate the effect. Retry only idempotent requests, or use idempotency keys.
• Ignoring draining. Killing servers instantly during deploys cuts live requests. Enable draining and test it with real traffic.
• Round robin for wildly uneven requests. Long and short requests need least connections or separate pools.

Key takeaways

• A load balancer is the front door. It gives clients one address while spreading work across many interchangeable servers.
• L4 is fast and simple, L7 is smart. Use layer 7 for path or header routing; layer 4 for raw speed or non-HTTP protocols.
• Start with round robin or least connections. Move to weighted or consistent hashing only with a reason such as mixed servers or per-key caches.
• Health checks decide who gets traffic. Require several consecutive results, and avoid checks that can fail the entire fleet at once.
• Drain before you remove. Draining lets in-flight requests finish during deploys and scale-in.
• Avoid sticky sessions. Make servers stateless with shared session storage.
• Protect the protector. Use redundant balancers, DNS with several addresses, anycast or a managed service.
• Terminate TLS centrally, but keep the balancer lean. Too much logic there creates a bottleneck.

Practice exercises

Exercise 1 — Starter (30 min): A pool has three servers handling these request durations in order: 5 s, 50 ms, 50 ms, 5 s, 50 ms, 50 ms. Simulate round robin by hand and write down how many seconds of work each server receives. Then simulate least connections, assuming each new request goes to the server with the least work outstanding, and compare the results.

Exercise 2 — Intermediate (2 hours): Draw the retail architecture with a layer 7 balancer routing /browse, /checkout and /static to three pools. Mark where TLS terminates, which health check each pool uses, and what happens to an in-flight checkout when you deploy a new version. Write a short note justifying the algorithm for each pool.

Exercise 3 — Advanced (half day): Install Nginx or HAProxy locally (or use Docker) with two tiny backends that report their name. Configure round robin, then least connections, then a health check, and finally kill one backend while sending steady requests with a load tool. Record what clients see, how quickly the failure is detected, and how the numbers change when you require two versus five failed probes before removal. Summarise your findings in one table.

Related reading

• System Design: Scaling Basics — the next lesson, on stateless tiers and how to decide what to scale first.
• System Design: The Request Lifecycle — shows where the load balancer sits between DNS, TLS and your application.
• System Design: Sharding — a deeper treatment of consistent hashing and virtual nodes.
• System Design: Rate Limiting — the protective job a gateway or balancer often performs at the edge.
• Microservices best practices — how routing and service boundaries affect what sits behind the balancer.