System Design: Capacity Planning and Cost

5. Validate, watch, re-plan

A plan is a hypothesis. Validate it with a load test, track actual utilisation and saturation against it in production, and re-run the plan on a schedule and before known events.

Load testing

A load test sends synthetic traffic and measures how latency, errors and resource usage change. Different tests answer different questions.

Good practice:

• Use realistic traffic mixes and data. A test that hits one cached URL measures the cache, not your system, and an index over 1,000 rows behaves nothing like one over 500 million.
• Watch the whole path, including databases, caches, queues and downstream services. The bottleneck is often not where you expect.
• Find the knee. Plot latency against throughput: latency stays flat, then bends sharply upward. Capacity is a bit to the left of that bend.
• Avoid the coordinated-omission trap. A load generator that waits for slow responses before sending the next request under-reports the delays real users would see. Use a fixed arrival rate.
• Test in an environment that matches production closely enough to trust, and be very careful when testing against production.

Autoscaling and its limits

Autoscaling adds or removes instances based on a metric such as CPU, request rate or queue depth. It is excellent for smoothing daily cycles and saving money. It is not a substitute for planning.

Read it as a chain of delays: detection, node provisioning, image pull, start-up and warm-up each add time, and a sharp spike can overwhelm the existing fleet before new capacity arrives. Other limits:

• It scales one tier, but the bottleneck may be elsewhere. More application servers make a saturated database worse by opening more connections.
• Quotas and capacity exist. Accounts have quotas, subnets have finite addresses, node groups have maximum sizes, and instance types can run out in a zone. Set maximums on purpose and request quota increases before an event.
• Metrics can mislead. CPU is a poor signal for I/O-bound services; queue depth or concurrency often works better. Use cooldowns so the policy does not flap.
• Cold starts. New instances start with empty caches, so pre-warm or ramp traffic gradually.
• Scaling down is risky too. Removing instances while connections or jobs are in flight needs graceful draining.

Combine reactive autoscaling for the unexpected with scheduled scaling for the predictable, such as a known sale at noon, and pair both with load shedding, as in resilience patterns.

Where the money goes

Cost is a system property just like latency. Before optimising, look at what drives the bill. The categories are common across providers, though pricing models and rates differ and change, so check your own provider's current pricing.

Two patterns recur. Data transfer is the surprise: moving bytes between regions, or out to the internet, can cost more than the compute that produced them. Idle waste is common: forgotten development environments, over-sized databases and unattached volumes quietly accumulate. Every resilience measure also has a price: a fully running second region roughly doubles infrastructure, as discussed in multi-region and disaster recovery, and the growth buffer plus zone redundancy create the 18-versus-8 gap above. Treat these as insurance premiums chosen per system.

When a bill surprises you, triage it in this order.

Build versus buy

For each component you can build it, run open-source yourself, or buy a managed service. Compare the total cost of ownership, not only the invoice.

Include people cost: an engineer spending a quarter of their time keeping a database cluster healthy is a real expense that never appears on the cloud invoice. Include the exit cost too. Many teams buy at small scale and revisit when the bill grows large enough to justify the engineering.

Efficiency levers

Before buying more capacity, make each unit do more. Roughly in order of payoff:

1. Remove work. Cache results, avoid repeated queries, batch small operations.
2. Fix the hot spots. One index can save a database tier.
3. Right-size to the measured bottleneck: memory-heavy, compute-heavy or I/O-heavy.
4. Use the right storage tier and expire data on a schedule.
5. Reduce bytes. Compression, smaller payloads, image resizing and a CDN cut bandwidth and egress.
6. Smooth demand. Move non-urgent work to a queue and run it off-peak, lowering the peak you provision for.
7. Buy commitment discounts only after usage is stable, for the baseline you are confident about.

In practice: an e-learning platform's surprise cloud bill and its term-start capacity plan

The situation

An online learning platform serves about 150,000 enrolled students with a platform team of eight. Lessons are mostly video; quizzes and discussion are database-backed. The stack: Kubernetes with a horizontal pod autoscaler, managed PostgreSQL with one read replica, Redis for sessions and caching, S3-compatible object storage behind a CDN for video, OpenSearch for course search, RabbitMQ for quiz-grading jobs, and Prometheus and Grafana. Two problems. First, last month's cloud bill came in at roughly twice the forecast, and nobody could say why in under a day. Second, term starts on Monday at 09:00 and last term the first 20 minutes brought slow logins and a database at its connection limit. Assume the next term's forecast is 30,000 concurrent active students in the first 15 minutes, each generating about 0.1 requests per second, so 3,000 requests per second at peak. Constraint: finance wants the monthly bill within 15 percent of forecast.

Starting architecture

Options considered

The decision

Rollout and verification

Triage first: in this scenario assume the surprise came from one video path serving files straight from object storage instead of through the CDN, cross-zone traffic between pods and the search cluster, and debug logging left on. Fix those and confirm with the next daily cost report. Then run a load test with a fixed arrival rate at 1.2 times the forecast, replaying a realistic mix of logins, lesson views and quiz submissions against production-sized data in a staging copy, and record where latency bends. Verify before the event: HPA and node group maximums above the planned pod count, account quota above the maximum nodes, database connections under the limit with margin, replica lag acceptable. Dashboards: API p99 and error rate, pod count versus desired, pending pods, database connections and queue depth, cache hit ratio, and daily cost with cost per active student.

The 2 a.m. failure

At 09:05 on term-start Monday the page reads "API p99 above 1.5 seconds, 20 pods Pending". The scheduled scale-up asked for 36 pods, but the node pool's maximum size was left at last term's value, so only 16 pods can be scheduled, about 2,400 requests per second of capacity against a 3,000 peak.

First three checks: list pods in the Pending phase; kubectl describe one to read the scheduling event (insufficient CPU, or node group max reached); and read the cluster autoscaler status for the node group limit. Mitigation: raise the node group maximum, and in the meantime turn off the low-priority flags (search typeahead and recommendations) to shed load until the pods land. Follow-up fix: add every limit (node group maximum, account quota, subnet addresses, database connections) to the pre-event checklist, and alert whenever desired capacity exceeds the configured maximum.

Architect's notes

• Measure unit cost before optimising. Cost per active student per day tells you whether the bill is growing because of success or regression.
• Ask in review: "what is the limit of every tier, and who knows it?" Node maximums, quotas, connections and IP ranges are where plans quietly fail.
• Egress and cross-zone traffic are the late surprises. Model bytes moved between components in the design, not after the first invoice.
• Give the bill an owner. Tag resources by team and review cost weekly with the same ritual as reliability; an unowned line item grows.
• Rehearse the peak, not the average. A load test at a fixed arrival rate before each term is cheaper than one incident.
• Defer commitments and rewrites. Buy discounts for the proven baseline only, and delay re-platforming until the efficiency levers are exhausted.

Common mistakes

• Planning for average load. Peaks cause the outages. Size for peak, then add headroom.
• Guessing per-instance capacity. Without a load test the number is fiction. Measure it against the SLO.
• Running at 90 to 100 percent utilisation to "save money". Queueing delay explodes near saturation. Plan to a target below the knee.
• Relying on autoscaling alone. It lags, hits quotas and cannot fix a saturated downstream. Add headroom, scheduled scaling and shedding.
• Ignoring data transfer and cross-zone traffic. The bill can be dominated by bytes moved, not compute. Model it early.
• Forgetting people cost in build-versus-buy. Self-hosting is not free. Include operations effort and risk.
• Never revisiting the plan. Review utilisation and forecasts on a schedule and before events.

Key takeaways

• Capacity means meeting the SLO at peak, with a failure in progress. Average load and "not crashed" are the wrong measures.
• Plan in five steps. Forecast, measure per-unit capacity, compute, add headroom, validate and re-plan.
• Headroom is a decision. Utilisation targets, N+1 redundancy and growth buffers each have a price you can state.
• Load tests find the knee. Use realistic traffic and data, a fixed arrival rate and the whole dependency path.
• Autoscaling is a smoother, not a safety net. It lags and can be blocked by quotas or a downstream bottleneck.
• Egress and idle resources drive many surprise bills. Measure data transfer and hunt waste.
• Build versus buy is total cost of ownership. Include engineer time, risk and exit cost.
• Remove work before adding hardware. Caching, indexes, compression and batching usually beat bigger machines.

Practice exercises

Exercise 1 — Starter (30 min): A service must handle a forecast peak of 5,000 requests per second. A load test shows one instance holds its latency target up to 400 requests per second. Using a 25 percent growth buffer and surviving the loss of one of three zones, calculate the number of instances needed and show each step. Then calculate how many in-flight requests exist at peak if the average latency is 80 ms.

Exercise 2 — Intermediate (2 hours): Design a load-test plan for an API that has a cache in front of a database. Specify the traffic mix (read-write ratio, cache hit assumption), the data volume, the ramp and hold profile for a load test, a stress test and a soak test, the metrics you will record at each tier, and how you will tell where the knee is. List three results that would make you change the architecture.

Exercise 3 — Advanced (half day): Take a design from this course (for example the photo-sharing app or the URL shortener) and build a monthly cost model. Assume traffic, storage growth and egress, and list every cost category from the table. Identify the top three cost drivers, propose one lever for each with an estimated percentage saving (clearly labelled as an assumption), and write a build-versus-buy recommendation for the database, the queue and the search index including people cost.

Related reading

• System Design: Estimation — the back-of-envelope arithmetic this lesson turns into a plan.
• System Design: Scaling Basics — vertical and horizontal scaling and finding the bottleneck.
• System Design: Observability and SLOs — the metrics and targets capacity is measured against.
• System Design: Resilience Patterns — load shedding and bulkheads for when capacity runs out.
• System Design: Multi-Region and Disaster Recovery — the cost of redundancy across regions.