System Design: The Big Picture

What you'll learn: By the end of this lesson you will be able to explain what system design is and ask the three questions that start every design. You will name the parts of a typical web architecture and say what each one is for. You will pick the right scaling technique for a given bottleneck, and you will know where the rest of this course goes from here.

Think of a restaurant. A tiny café can run on one cook, one till and a handwritten order pad. Now imagine the same café has to feed ten thousand people at lunch. You cannot just hire a faster cook. You need a kitchen layout, stations, a queue for orders, a plan for what happens when the fryer breaks, and a way to know the walk-in fridge is running low before it runs out.

System design is that kind of planning for software. It is the work of deciding which parts a system has, how they talk to each other, where data lives, and how the whole thing behaves when it is busy or when something breaks. Code answers "how does this function work?" System design answers "how do these twenty things work together, and still work next year?"

Open the full-size cheat sheet. Everything on it is explained in text below, so you can search and copy it.

The core idea

Every system design starts from three questions. Answer them in order, and most of the design falls out.

1. What must it do? These are the functional requirements: the features. "Users upload photos, follow other users and see a feed."
2. How well must it do it? These are the non-functional requirements: speed, availability, durability, security, cost. "The feed loads in under half a second. Photos are never lost."
3. At what scale? How many users, how many requests per second, how much data, and how fast is that growing? "Ten million users, mostly reading."

The same feature list can lead to wildly different designs depending on questions two and three. A photo app for a family of five and one for ten million people share a feature list and almost nothing else. Skip these questions and you will build something that falls over, or something absurdly over-built that costs ten times more than it should.

This lesson is a map. The course's running example is a photo-sharing app (users upload photos, follow each other, scroll a feed), and the "In practice" section near the end uses a different, more everyday product: a grocery-delivery app having a bad day. Later lessons go deep on every idea met here.

A tour of a typical architecture

Here is the shape most web products grow into. Not every product needs every box, and a small product should start with far fewer.

How to read it: the edge is what the user touches, the application tier is the code you write, and the data and async tiers hold state and do slow work.

The request lifecycle in five steps

When someone opens the app and loads their feed, roughly this happens. The Request Lifecycle lesson walks through every step.

1. Resolve. The client asks DNS for your service's address. Answers are cached along the way.
2. Connect. The client opens a connection and sets up encryption (TCP is the reliable connection, TLS the encryption on top). Doing this repeatedly is expensive, so connections are reused.
3. Route. The CDN may answer static content. Everything else passes the firewall to the load balancer, which picks a healthy instance.
4. Compute. The application checks identity, looks in the cache, queries the database if needed, and may queue slow work.
5. Respond. The response travels back, and parts of it are cached for next time.

Almost every performance and reliability question is really a question about one of these steps: which is slow, which can fail, and which can be skipped by reusing earlier work.

Scaling: how a system grows

Scaling means handling more load without falling over. There are only a handful of techniques, and the skill is choosing the one that fits the actual bottleneck.

The order matters, and measurement comes first. This flowchart is the order most products follow.

How to read it: follow the first "yes" from the top. The cheap, reversible fixes sit near the top and sharding sits at the bottom because it is the hardest to undo.

Reliability: assume things break

Servers crash, networks drop packets, disks fail, and dependencies get slow. In a system with many parts, something is always failing somewhere. Fault tolerance means the system keeps working, perhaps in a reduced way, anyway. The core tools:

• Timeouts. Never wait forever for another service; a call with no timeout can pin a thread until the whole service stops.
• Retries with exponential backoff and jitter. Retry after a growing, randomised delay so many clients do not retry at the same instant.
• Circuit breakers. If a dependency is clearly failing, stop calling it for a while and fail fast.
• Idempotency. Make doing an operation twice equal doing it once, so retries are safe. "Set the profile photo to X" is idempotent. "Add one dollar" is not, unless you attach a unique request key.
• Bulkheads. Isolate resources (separate thread pools per dependency) so one failing part cannot drain everything.
• Health checks. Let the load balancer ask each instance "are you OK?" and skip unhealthy ones.
• Backups. Copies in another place, with restores you have actually tested. A backup you have never restored is a hope, not a plan.

Data: SQL, NoSQL and CAP in brief

"NoSQL" is not one thing and not a badge of scale. Many relational databases handle very large workloads. Choose by access pattern and consistency needs, not fashion.

When data is copied across machines you meet the CAP theorem. Stated precisely: during a network partition (some machines cannot reach others) a system must choose between consistency (every read sees the latest write) and availability (every request still gets an answer). Partitions are a fact of networks, so the real question is what to do when one happens. See CAP and Consistency.

Security and operations

Two topics beginners leave to the end and regret it. For security: authenticate (who are you?) and authorise (what may you do?) separately, encrypt traffic with TLS and sensitive data at rest, give each service minimum permissions, validate every input, keep secrets out of code, and rate-limit to blunt abuse. Each layer assumes the one before might fail, which is called defence in depth.

For operations: a system you cannot see is a system you cannot fix. Use logs (what happened), metrics (numbers over time) and traces (one request's path through many services). Set an SLO (service level objective), such as "99.9% of feed requests succeed", and alert when you are burning through it, not on every blip.

Myths versus reality

In practice: a grocery-delivery app on its first big promo day

The situation

A regional grocery-delivery app serves one metro area. Six engineers run it, and the budget is small: no dedicated operations person. The stack is a Spring Boot monolith on one virtual machine behind Nginx, with PostgreSQL on a second machine. Product photos are served from the application's own disk.

Marketing plans the first big promotion. At 12:00 a push notification goes to 150,000 installed devices. Assume 5% open the app in the first ten minutes, which is 7,500 people at about 40 requests each: 300,000 requests in 600 seconds, or about 500 requests per second on average, with the first minutes roughly three times higher (an expected peak of about 1,500 requests per second). An earlier load test showed the single server degrading past about 150 requests per second. The pain on a normal Friday already shows: catalogue p95 latency around 1.2 seconds, and the application VM's CPU near 70%.

Starting architecture

How to read it: everything shares one machine's CPU and network, including photo bytes and notification calls made while the customer waits.

Options considered

The decision

Rollout and verification

Ship in that order because each step is independently reversible and earlier steps are the cheapest wins. Put each behind a feature flag where possible, and send 5% of traffic to the new instances first (a canary) before raising it. Run the load test (a tool such as k6 works) up to the expected peak of about 1,500 requests per second and then beyond it for headroom. The single server managed about 150 requests per second, so three instances alone would not be enough: the cache and the photo offload must make each request much cheaper, and the test shows whether more instances are needed. Watch Prometheus and Grafana dashboards: catalogue p95 and p99 latency, 5xx rate, cache hit ratio, CDN offload, database connections in use, and CPU per instance. The target, not a promise: p95 under 500 ms at the expected peak, and no database connection pool saturation.

The 2 a.m. failure

At 02:10 an alert fires: "checkout p95 above 2 s, database connections above 90%". The cause: the Redis node ran out of memory and restarted, losing sessions and the cache, and every catalogue request now goes straight to PostgreSQL.

The first three checks: Grafana (cache hit ratio dropped to zero), redis-cli INFO stats (keyspace hits and misses, evicted_keys), and in psql SELECT state, count(*) FROM pg_stat_activity GROUP BY state; to see how many connections are stuck. Mitigation: bring Redis back, then use a flag to serve a short-lived static catalogue snapshot so the database is not hit by every request. Follow-up fixes: split sessions from the cache so one failure does not log everyone out, set a maxmemory policy and an alert on memory, add request coalescing so a cold cache does not stampede, and add a cache-hit-ratio alert.

Architect's notes

• Measure before you buy. The load test number (about 150 requests per second) decided more than any diagram did. Get that number before the design meeting.
• Ask in the design review: what happens when each new box is down? Every added component is a new way to fail, and the Redis outage above is the price of the cache.
• Do the cheap, boring wins first. Photo offload and a CDN remove load at comparatively low risk.
• Name an owner and a pager. A component with no owner (who restarts Redis at night?) is a failure waiting for the worst hour.
• Defer sharding and microservices. Write the trigger (database CPU, p95 latency) in the decision record and let the numbers, not enthusiasm, call it.

A practical checklist

1. Write the functional requirements and what is out of scope.
2. Write the non-functional requirements with numbers.
3. Estimate load: requests per second, storage, bandwidth.
4. Sketch the API and the main data model.
5. Draw the simplest high-level design that works.
6. Find single points of failure and the likely bottleneck.
7. Add caching, queues and partitioning only where the numbers demand.
8. Add timeouts, retries, health checks, backups and monitoring.
9. Check authentication, authorisation, encryption and abuse limits.
10. Write down the trade-offs and what would make you revisit them.

Course roadmap

This is the first of three courses, each assuming the one before. See all three, the cheat sheet and the case studies on the System Design learning path.

Course 1: System Design Foundations (no prior knowledge needed)

• System Design: The Big Picture: this lesson.
• Requirements: turn a vague idea into scope and measurable goals.
• Estimation: back-of-the-envelope maths for load, storage and bandwidth.
• The Request Lifecycle: what happens between a click and a response.
• API Design: contracts, pagination, idempotency and versioning.
• Load Balancing: spreading traffic and surviving failures.
• Scaling Basics: vertical versus horizontal, and finding the bottleneck.
• Caching Basics: cache-aside, TTLs and eviction.
• Databases Basics: relational and NoSQL, indexes and transactions.
• CDN and Object Storage: serving files and media well.
• Reliability Basics: redundancy, retries and graceful degradation.
• A Repeatable Approach: a framework for any design problem.

Course 2: Building Blocks starts at Building Blocks Overview. It covers replication, sharding, consistency, advanced caching, queues, event streaming, architecture styles, gateways, rate limiting, search, storage engines, observability and security.

Course 3: Advanced starts at Advanced Overview. It covers distributed transactions, CQRS and event sourcing, consensus, clocks and ordering, resilience patterns, multi-region design and capacity planning, then full case studies including Photo Sharing.

Common mistakes

• Designing before asking. A diagram without requirements answers the wrong question. Fix: spend the first minutes on the three questions.
• Over-building on day one. Microservices, sharding and a message bus for a hundred users burn time and add failure modes. Fix: start simple and add parts when numbers justify them.
• Scaling the wrong thing. More application servers when the database is the bottleneck wastes money and can worsen the outage. Fix: measure first.
• Ignoring failure. Happy-path-only designs fail in production. Fix: for every box ask "what if this is slow, down or returns garbage?"
• Treating a cache as a database. If the only copy is in a cache, one restart loses it. Fix: keep a durable source of truth.
• Untested backups. Finding corrupt backups during an incident is too late. Fix: schedule restore drills.
• Skipping numbers. "A lot" and "fast" drive no decisions. Fix: use rough numbers labelled as assumptions.

Key takeaways

• Three questions first. What must it do, how well, and at what scale; most choices follow.
• Start small. The simplest design that meets the requirements is usually the right first design.
• Stateless tiers scale out. Keep application servers free of local state so you can add copies behind a load balancer.
• Match technique to bottleneck. Caching, replicas, queues and sharding each solve a different problem at a different cost.
• Failure is normal. Timeouts, capped retries with backoff and jitter, circuit breakers, health checks and tested backups are baseline.
• Idempotency makes retries safe. Design operations so a repeat does no extra harm.
• CAP is about partitions. During a partition you choose consistency or availability; otherwise you trade latency against consistency.
• Designs evolve. Revisit assumptions as real traffic shows where the bottleneck is.

Practice exercises

Exercise 1 — Starter (30 min): Pick an app you use daily, such as a music or messaging app. Write three functional requirements, three non-functional requirements and one guess at its scale. Then redraw the architecture diagram in this lesson with only the boxes you think that app really needs.

Exercise 2 — Intermediate (2 hours): For the photo-sharing app, list five things that could fail (for example, the thumbnail worker crashes or the database primary is down). For each, write what the user sees today, which reliability technique would help, and what it costs. Present it as a table.

Exercise 3 — Advanced (half day): Imagine the photo app grows from ten thousand to ten million users. Using the scaling table, write the order in which you would apply the techniques, and for each step the specific metric that would trigger it. Then argue against yourself: name one step you could reasonably skip, and one signal that would prove you wrong.

Related reading

• Requirements - the next lesson, where the three questions become concrete, measurable goals.
• The Request Lifecycle - a closer look at the five steps from click to response.
• Reliability Basics - the failure-handling techniques from this lesson in depth.
• Photo Sharing case study - the running example designed end to end in the advanced course.
• Notification service design - a worked example of a real design problem from the blog.