System Design: Scaling Basics

A caveat on tokens: signed tokens remove the session store from the request path, but they are hard to revoke before they expire, so many systems still keep a server-side element for logout or blocking. There is no free option, only trade-offs.

Finding the bottleneck

Before scaling anything, ask which resource is saturated. A bottleneck is the component that reaches its limit first and so caps the whole system.

Two terms from the table: an N+1 query problem is fetching a list and then running one extra query for every item in it, and connection pooling means keeping a set of reusable database connections that requests borrow instead of opening a new one each time (see databases basics).

Start at the top and answer from measurements, not hunches. The last box is the honest fallback: if no resource looks saturated, follow a single slow request through a trace to see where its time goes.

How to find the bottleneck in practice:

• Look at utilisation and saturation together. The USE method (utilisation, saturation, errors) asks for each resource: how busy is it, is work waiting, and is it failing?
• Load test in a safe environment and watch which graph hits the ceiling first. Turn the load up gradually.
• Beware averages. Look at p95 and p99 latency (the slowest 5% and 1% of requests), since a healthy average can hide a painful tail.

Scaling reads versus scaling writes

Most applications are read-heavy, and reads and writes scale differently, so treat them separately.

Scaling reads is the easier problem, because reads do not change data and can be served from copies: cache results (caching basics), use a CDN for static content, add read replicas (copies of the database that serve reads; the main copy that accepts writes is the primary; replicas lag slightly, see replication), and denormalise (store some data redundantly so reads need fewer joins).

Scaling writes is harder, because every write must go to the authoritative copy and often must be ordered and consistent: optimise the write path (batching, fewer indexes on hot tables), scale the single primary vertically as far as sensible, buffer with a queue so bursts are absorbed (message queues), and only then partition (shard) the data across several primaries (sharding).

The order in each branch matters: the earlier steps are cheaper and simpler. Sharding is the last resort, not the first.

Auto scaling basics

Auto scaling adds or removes servers automatically based on load, so you do not pay for idle capacity at 3 a.m. and do not fall over at lunchtime. It works best with the stateless application tier, because new servers must start serving immediately.

• Reactive (metric-based). You choose a metric, such as average CPU at 60% or requests per server, and the system adds servers when it is above the target and removes them when below.
• Scheduled or predictive. You know traffic rises at month end or during a planned campaign, so you add capacity ahead of time.

Pitfalls to understand:

• Start-up time. A new server needs time to boot, warm caches and pass health checks. Reactive scaling is always a little late, so keep headroom for sudden spikes.
• Flapping. Scaling out and in rapidly wastes money. Use cooldown periods and scale in more slowly than out.
• Limits. Set a minimum (for availability) and a maximum (to cap cost and protect downstream systems). Ten new app servers each opening fifty database connections can overwhelm the database even though the app tier scaled fine.
• The wrong metric. CPU is a poor signal for a service that waits on I/O. Queue length or request latency may be better.

What to scale first

When something is slow, use this table as a starting point. It assumes you have measured and confirmed the symptom.

In practice: a payroll SaaS and its month-end spike

The situation

A payroll SaaS serves about 2,000 small and mid-size employers. A stateless Spring Boot API runs on Kubernetes behind an ingress, with the Horizontal Pod Autoscaler scaling it on CPU. A managed PostgreSQL instance (one primary, one read replica) stores employees, pay runs and tax rules. Redis holds sessions. Payslip PDFs are generated inside the API request and saved to S3-compatible storage. Prometheus and Grafana provide metrics. A team of seven engineers owns it, and a contractual rule says pay runs must complete before bank cut-off times, so slow means penalties.

On the last three days of each month, traffic is roughly eight times the monthly baseline (illustrative). The pain: the autoscaler grows the API from 6 to 40 pods, but pay-run completion time rises from about 4 minutes to about 25. Database CPU sits near 95 percent, and logs show "timeout waiting for connection from pool".

Starting architecture

Forty pods times a pool of 20 is 800 possible connections against a limit of 500, and every pod also renders PDFs while holding a database transaction open.

Options considered

The decision

Rollout and verification

Put PgBouncer in front of the primary in a staging environment first and run the application's test suite against it, because pooling modes change what is safe: in transaction pooling, session-level state such as SET commands or advisory locks does not carry over between statements. Then release behind a feature flag per employer, starting with internal test accounts. Load test with twice last month's peak before the next month end. Prove it with these signals: pool wait time, database connections versus the limit, primary CPU, queue depth and age of the oldest message, and pay-run duration percentiles.

The target design: the API tier stays stateless and capped, the pooler guards the database, and slow PDF work leaves the request path.

The 2 a.m. failure

On the 28th at 02:00 a scheduled bulk payroll import starts. The page reads "API error rate above 5 percent, pool wait high". Detection: the connection-wait metric and the database connection count both approach their limits while the queue depth is flat. The first three checks: SELECT state, count(*) FROM pg_stat_activity GROUP BY state; (a large number of sessions "idle in transaction"), SHOW POOLS; in the pooler (clients waiting), and kubectl get hpa (the autoscaler is at its cap).

Mitigation: pause the import job, terminate the idle-in-transaction sessions older than a few minutes, and let the pool drain. The follow-up fix is to call the slow tax service before opening the transaction, give the import its own small pool so it cannot starve interactive traffic, and set a database idle_in_transaction_session_timeout.

Architect's notes

• Measure the connection budget before you autoscale. Multiply maximum pods by pool size and compare it with the database limit. Make the autoscaler's maximum a number you computed, not a default.
• Ask what the business deadline is. A month-end spike with a bank cut-off is a scheduled event, so scale on a schedule and treat reactive autoscaling as a safety net.
• Separate interactive from batch capacity. Bulk imports and PDF generation should not share a connection pool or queue with a user waiting on a screen.
• The database is a shared resource with one owner. Decide who approves new heavy queries and who is paged when connections run out. Unowned databases get tragedy-of-the-commons failures.
• Count the bill of the peak. Running 40 pods for three days is cheap; permanently oversizing the database for those three days usually is not. Prefer elastic app capacity and a carefully sized database.
• Defer sharding. Revisit it only when the primary, after pooling, caching and queueing, is still saturated at the next doubling of customers.

Common mistakes

• Scaling before measuring. Adding servers to fix a database problem wastes money and hides the cause. Profile first.
• Keeping state on app servers. Local sessions or files prevent scaling out. Move state to shared stores.
• Jumping to microservices or sharding early. These add large operational cost. Try indexes, caching, replicas and a bigger machine first.
• Ignoring the tail. Average latency looks fine while p99 is terrible. Track percentiles.
• Autoscaling without limits or headroom. Without a maximum, a retry storm can scale costs and connections out of control; without headroom, spikes outrun boot time.
• Forgetting downstream capacity. More app servers means more database connections. Use pooling and cap concurrency.
• Treating scale as one-off. Traffic patterns change. Re-measure regularly and load test before big events.

Key takeaways

• Scalability is a loop. Measure, find the bottleneck, fix it with the simplest tool, and measure again.
• Vertical scaling is simple but capped. A great first step and often right for databases, but it leaves a single point of failure.
• Horizontal scaling needs stateless services. If any server can handle any request, you can add and remove them freely.
• Externalise state. Sessions, files and data belong in shared stores, not in server memory or local disk.
• Find the bottleneck before you spend. CPU, memory, disk, network and database each have distinct symptoms and fixes.
• Treat reads and writes differently. Reads scale with caches and replicas; writes need tuning, queues and eventually sharding.
• Autoscaling is for the stateless tier. Give it minimums, maximums, cooldowns and check downstream limits.
• Cheap fixes first. Indexes, caching and a bigger machine usually come before sharding or new architecture.

Practice exercises

Exercise 1 — Starter (30 min): List every piece of state in a simple payroll app (login, uploaded timesheets, pay runs, generated payslips, cached tax tables). For each, say where it should live so that the app servers are stateless, and name one risk of the wrong choice.

Exercise 2 — Intermediate (2 hours): For each symptom, write the one measurement you would take first and the most likely fix: (a) pay-run page p95 is 2 seconds, app CPU is 15%, database CPU is 90%; (b) uploads fail during evening bursts although average load is low; (c) pages are slow only for users on another continent. Then draw the architecture after your fixes using Mermaid or paper.

Exercise 3 — Advanced (half day): Build a tiny web service with an in-memory counter and run it behind a local load balancer with two instances. Observe the wrong results. Then move the counter into Redis or a database and observe the fix. Finally, design an autoscaling policy on paper: metric, target, minimum, maximum, cooldown, and a list of downstream limits you would check. Explain what would go wrong if you removed the maximum.

Related reading

• System Design: Load Balancing — how traffic gets spread across the servers you scale out.
• System Design: Caching Basics — the cheapest way to scale reads.
• System Design: Databases Basics — indexes and pooling, the usual first fixes for a slow data tier.
• System Design: Replication — read replicas and what replication lag costs you.
• Modular monolith architecture in Java — why a well-structured single deployable often scales further than expected.