System Design: A Repeatable Approach

Step 6: Deep-dive the riskiest parts. You will never have time to detail everything. Pick the two or three parts where the design is hardest or failure hurts most: how IDs are generated, how data is partitioned, how a hot read path is cached, how a failure is recovered. Then go deep on data structures, failure behaviour and consistency.

Step 7: Find bottlenecks and failure points. Walk the diagram with two questions. "What breaks at ten times the load?" finds the single database, the shared queue, the hot key. "What happens when this box dies?" finds single points of failure, as in System Design: Reliability Basics. For each problem, propose a fix and say what it costs.

Step 8: Discuss trade-offs and alternatives. A good trade-off statement has three parts: the choice, what it buys, and what it costs. "We use asynchronous replication, which keeps writes fast but means a failover may lose the last few writes." If you cannot state the cost, you do not yet understand the choice.

How to think aloud and communicate

The way you communicate is part of the design.

• Structure your talk. Say which step you are on: "Let me move on to the data model."
• Check in after each major step: "Does this match what you had in mind?" Correcting course early is cheap.
• State assumptions as assumptions. "For illustration, I'll assume 100 million users."
• Explain why, not just what. "I'm using a queue here because uploads should not wait for thumbnail generation."
• Be willing to be wrong. If you spot a flaw, say so and fix it. That is what happens in real design reviews.

Common pitfalls

A small worked example: a paste-bin service

Let us apply all eight steps to a deliberately small problem: users paste a block of text, get a short link, and anyone with the link can read it. Pastes can expire. (Every figure below is an illustrative assumption.)

1. Requirements. In scope: create a paste, read it by short ID, optional expiry. Out of scope: accounts, editing, highlighting, search. Non-functional: reads under 200 ms at the 99th percentile (99% of reads faster than that), highly available reads, no loss before expiry, IDs hard to guess.

2. Estimation. Assume 1 million new pastes per day and 10 reads per write. Writes: 1,000,000 / 86,400 is about 12 per second, about 35 at a 3x peak. Reads: about 120 per second, 350 at peak. Storage at 10 KB per paste: 10 GB per day, roughly 3.6 TB per year. Conclusion: the request rate is modest, a handful of servers is plenty, and storage is the thing to plan for.

3. API.

POST   /v1/pastes        body: { content, expires_in_seconds? }   -> 201 { id, url }
GET    /v1/pastes/{id}                                           -> 200 { content, created_at, expires_at }
DELETE /v1/pastes/{id}   (needs delete token returned at create) -> 204

A missing or expired paste returns 404 (or 410 Gone). Creation accepts an idempotency key so a retry does not create two pastes.

4. Data model. Metadata (id, created_at, expires_at, size, object_key) is small and keyed by ID, so it goes in a relational or key-value store. The text is large, so it goes in object storage under object_key. The only queries are "get by ID" and "find expired rows".

5. High-level design.

Write path: generate an ID, insert the metadata row as pending (the unique constraint on id catches any collision before anything is overwritten), write the text to object storage, mark the row ready. Read path: check the cache; on a miss read the metadata, check the row is ready and expires_at is in the future, fetch the text, populate the cache.

6. Deep dive. IDs: 8 random base62 characters from a cryptographically secure generator give 62^8, about 2.2 × 10^14 IDs, so collisions are rare; on one, the unique constraint rejects the insert and the app retries. A base62 counter never collides but is guessable unless scrambled, so random wins for the unguessability requirement. Expiry: lazy expiry (return 404 on read if expires_at has passed) gives correctness; a background sweeper deleting expired rows and objects in small batches gives cost control. Cache TTLs never exceed the time left before expiry.

7. Bottlenecks and failures. The metadata database is a single point of failure, so add a standby (Replication). A viral paste is a hot key, which the cache absorbs. Bots need per-IP rate limiting and a size cap. A crash between the row insert and the text write leaves a stale pending row; a periodic cleanup removes those and any orphan objects.

8. Trade-offs. Random IDs cost a retry path but buy unguessability. Object storage costs an extra hop on read but keeps the database small. If traffic grew a hundred times, add read replicas and a CDN before thinking about sharding (Sharding). The design took about a page, which is the point: a good design is the smallest one that satisfies the numbers.

How to practise

• Do one problem a week, out loud, with a timer. Forty-five minutes is a realistic interview length.
• Always start with the same eight steps, even if it feels slow. The habit is what you are training.
• Compare with a reference and note what you missed. Our notification service design is a worked example of the same thinking.
• Vary the constraints. Redo the problem at ten times the traffic, or with a strict consistency requirement, and see what changes.

One-page checklist

• Did I ask about functional scope and non-functional goals, and write down my assumptions?
• Did I estimate requests per second, storage and bandwidth?
• Did I define the API and the main data entities?
• Does every box in my diagram have a stated reason, and did I trace one read and one write?
• Did I deep-dive the riskiest component and look for single points of failure and hot spots?
• Did I state the trade-off of each major choice, and summarise at the end?

In practice: a design review for a parcel-tracking page

The situation

A regional courier asks a team of five engineers to "design a parcel-tracking page". Customers receive an SMS with a tracking link. Today the link opens a page served by the Java monolith (two VMs behind Nginx) that queries the operations PostgreSQL database directly, the same database that scanners write to and the operations dashboard reads. The team has six weeks and an assumed extra cloud budget of 1,500 dollars a month. Two things hurt: about 4,000 "where is my parcel" support contacts a day, and a page p95 of about 4 seconds on delivery evenings, when the operations dashboard slows too. The page also refreshes itself every 30 seconds while it is open, so many views are repeats of the same parcel.

A prompt like that is vague on purpose, so the team walks the eight steps in the review and writes one line per step:

Starting architecture

Public tracking traffic, scanner writes and the internal dashboard all share one database and one application.

Options considered

The decision

Rollout and verification

Ship behind a flag in the SMS link: 5% of links first, then 25% for one delivery evening, then everyone, with the old page kept for one week. Before the canary, load test at 3x the assumed peak with real tracking numbers replayed. Dashboards that prove it worked: page p95 against the 500 ms target, cache hit ratio, replica lag, rate of 404 lookups (enumeration shows up here), operations dashboard latency, and daily support contacts about parcel location.

The 2 a.m. failure

The page reads "Tracking p95 above 2 s and cache hit ratio below 30%". Redis restarted during the evening peak, so the cache is cold and the replica takes the full read load.

First three checks: kubectl get pods -n tracking for restart counts, redis-cli INFO stats to compare keyspace_hits with keyspace_misses, and SELECT now() - pg_last_xact_replay_timestamp(); on the replica for lag. Mitigation: flip a config flag that raises the TTL to 120 seconds and slows the page's auto-refresh, accepting looser freshness until Redis is healthy. Follow-up fixes: run Redis with a replica or use a managed offering, add request coalescing so a cold cache does not become a stampede, and alert on hit ratio before latency.

Architect's notes

• The first deliverable is the list of questions, not the diagram. Bring an assumptions page to the review and let people edit it.
• Put numbers next to every requirement. "Fresh" becomes "within 60 seconds", which is what made option A and B comparable.
• Ask who else reads the database. Hidden consumers, like the operations dashboard here, are often the real reason for a redesign.
• Write the "revisit when" trigger for every decision. A measurable trigger beats a vague promise to look again.
• Cost the operations, not just the build. A new service or cache is a pager, a dashboard and a runbook someone must own.
• Defer the event platform. Option C may be right in a year; do not pay for it before a second consumer exists.

Common mistakes

• Treating the prompt as complete. Vague prompts are vague on purpose. Ask questions and state assumptions before drawing.
• Naming products before requirements. It signals that you know tools but not problems. Describe the need ("a durable key-value store") then name an example.
• Over-engineering the first draft. Extra caches, queues and shards add failure modes. Begin with the simplest design that meets the numbers.
• Silent thinking. An interviewer or teammate cannot see your reasoning. Narrate your steps, including the options you rejected.
• Skipping failure and scale analysis. A design that only works when nothing goes wrong is unfinished. Spend real time on step 7.
• Defending a flawed design. If you find a problem, fix it openly. Revising a design is a strength.

Key takeaways

• Order matters. Work from cheap-to-change decisions (requirements) to expensive ones (storage and partitioning).
• Eight steps. Requirements, estimate, API, data model, high-level design, deep dive, bottlenecks, trade-offs.
• Assumptions are allowed, hidden assumptions are not. State them and label every number as an assumption.
• Estimates steer the design. Orders of magnitude tell you whether you need one server or a fleet.
• Start simple. Add each component only when a requirement or number justifies it.
• Deep-dive selectively. Spend your time on the riskiest two or three parts, not on every box.
• Every choice has a cost. If you cannot say what a choice costs, keep thinking.
• Practise out loud, regularly. The framework becomes automatic through repetition.

Practice exercises

Exercise 1 — Starter (30 min): Take the prompt "Design a service that lets people share a link to a file for 24 hours." Write ten clarifying questions you would ask, split into functional and non-functional. Then write the assumptions you would make if nobody answered.

Exercise 2 — Intermediate (2 hours): Redo the paste-bin design for 100 million new pastes per day. Recalculate writes per second, storage per year and peak reads. Decide which parts of the original design still hold and which need to change. Redraw the diagram and justify each new component with a number.

Exercise 3 — Advanced (half day): Pick a service you use daily, such as a ticket booking site or a to-do app. Run all eight steps on it in writing, in about two pages, including one Mermaid-style diagram and a trade-off table. Then swap with a peer, or reread it after two days, and list three weaknesses you now see.

Related reading

• System Design: The Big Picture — the tour of a typical architecture that this framework helps you navigate.
• System Design: Requirements — go deeper on step 1, including the nines table and SLOs.
• System Design: Estimation — the back-of-envelope skills behind step 2.
• System Design: Building Blocks Overview — the next course, which gives you the components to choose from in steps 5 and 6.
• Designing a notification service — a full worked design that follows the same flow.