System Design Advanced: The Hard Problems Map

What you'll learn: By the end of this lesson you will be able to name the core hard problems of distributed systems and how they relate, choose which advanced lesson addresses a given failure, practise a case study out loud with a time-boxed method, and apply a reusable template to any new design.

Imagine running a restaurant with one kitchen. Orders come in, a cook makes them, a waiter delivers them. Problems are easy to see because everything happens in one room. Now franchise that restaurant across ten cities. Menus must stay in sync. A coupon redeemed in one city must not be redeemed again in another. If the phone line to head office drops, each branch must decide whether to keep selling or stop. Nothing about cooking got harder, yet everything about coordination did.

That is the jump from one machine to many. The first two courses gave you the parts: load balancers, caches, replicas, queues. This course is about what happens when those parts disagree, fail halfway, or cannot reach each other. It also gives you six full case studies to practise on.

The core idea

Almost every hard problem in distributed systems comes from three facts: the network is unreliable, clocks disagree, and machines fail independently and partially.

Because a message can be lost, delayed, duplicated or reordered, you cannot always tell whether a peer is dead or merely slow. Because clocks drift, you cannot rely on "whose timestamp is later" to order events. Because nodes fail on their own schedules, the system must keep working, or fail safely, with some parts missing.

Everything in the advanced course is a technique for living with those three facts. Each technique buys a guarantee and charges for it in latency, complexity, or availability. Learning to see that price tag is the real skill.

How the course is organised

The course has three layers.

1. Core theory and patterns (the concept lessons). These explain the recurring problems and their standard solutions.
2. Operational concerns (resilience, multi-region, capacity). These deal with surviving failure and paying for scale.
3. Case studies. Six complete designs, each using the same arc, where you apply the theory under realistic constraints.

The hard problems in one map

The following diagram shows how the problems depend on each other. Read it from the top: unreliable networks and clocks cause the problems below them.

Walk through the arrows once, because each one is a lesson.

• Network to agreement. If machines must act as one logical unit (one leader, one lock holder, one configuration), they need a protocol that stays correct even when messages vanish. That protocol is consensus.
• Clocks to ordering. When two servers record events, comparing wall-clock times is unsafe. We use logical clocks, version vectors, or carefully bounded physical clocks instead.
• Agreement and ordering to replication. Copies of data stay coherent only if replicas agree on the order of writes. Whether you pay for strong ordering or accept eventual convergence is the consistency choice from the building blocks course.
• Replication to transactions and events. Once data lives in several services, a business action touches several stores. You either coordinate them (transactions, sagas) or derive state from an ordered stream of facts (event sourcing).
• Network to resilience. A slow dependency is often more dangerous than a dead one, because callers pile up waiting. Timeouts, circuit breakers and load shedding keep the damage local.
• Transactions, events and resilience to multi-region. Spreading a system across regions multiplies every problem above: longer latencies, partitions that can last minutes, and conflicting writes.

When you are looking at a symptom rather than a theory, this flowchart points you to the right lesson.

Read it as a first guess, not a verdict. Real incidents often touch two of these lessons at once, and the point of the map above is that they are linked.

The questions that cut through every design

Strong engineers keep a short list of questions in their heads. Use them in every lesson and every case study.

1. What happens if this message is lost? And if it is delivered twice, or arrives late?
2. What happens if this node dies between step 2 and step 3? Which state is half-written?
3. What happens if these two nodes cannot talk to each other for ten minutes? Do we stop, or continue and reconcile?
4. What is the source of truth, and who is allowed to write it? Two writers without a coordination rule means conflict.
5. Which guarantee do users actually need? Banks need strong consistency on balances. A "like" counter does not.
6. What is the blast radius? If this component fails, who notices?

The lessons keep returning to these six questions. If your design cannot answer them, it is not finished.

How to use the case studies

Reading a finished design teaches you what the answer looked like. It does not teach you how to find it. The skill is built by attempting the problem first. Use this loop for each case study.

Step 1: Read only the prompt. Each case study opens with a one-line problem statement. Stop there.

Step 2: Practise out loud, with a timer. Speaking forces you to organise your thinking, and the timer forces you to prioritise. A reasonable budget for a 45-minute session:

Step 3: Compare. Read the lesson and mark where your design differed. A difference is not automatically a mistake: ask whether your choice is valid under a stated requirement. Many problems have several good answers.

Step 4: Write down the gaps. Keep a running list: "forgot idempotency", "never estimated storage", "did not discuss failure". Patterns will emerge, and those patterns are your study plan.

Step 5: Revisit. Redo the weakest section a week later without notes. Recalling material after a delay helps it stick.

A reusable case-study template

The case studies in this course all follow the same arc. You can use it on any problem, from a take-home exercise to a real design document.

You can copy this skeleton into a document and fill it in. A few habits make it work:

• Label every number as an assumption. "Assume 10 million daily users" is honest. Stating invented figures as fact is not.
• Pick the deep dives deliberately. The riskiest or least standard parts deserve your time, not the login page.
• State trade-offs as sentences with "because". "We use asynchronous replication because the feed tolerates seconds of staleness; we accept possible loss of the last writes on failover."
• Name a failure and its detection. A failure you cannot detect is a failure you cannot recover from.

A worked example to study first

Before starting the case studies, read a complete design written in this style. Designing a Notification Service works a real problem from requirements to failure handling. As you read, mark where it applies the template above, and notice how many of the six questions it answers without being asked.

Then try the following warm-up on your own: take the notification service and ask what happens if the queue delivers one message twice. Decide where idempotency has to live. You will meet exactly this question again in the lesson on distributed transactions.

Suggested path

If you are new to the advanced material, follow this order:

1. Read this lesson, then Consensus and Leader Election. It names the hardest problem directly.
2. Continue with Distributed Transactions and CQRS and Event Sourcing, which show how real systems cope without a single database.
3. Learn how to order things and generate IDs, then resilience and multi-region.
4. Start the case studies with the URL shortener, which is small enough to finish in one sitting, then move on to the photo-sharing and news-feed designs.
5. Finish with chat, video streaming and ride sharing, which push on connections, throughput and geography respectively.

If you already work with distributed systems, jump to whichever lesson matches a problem you have hit. The concept lessons are self-contained.

In practice: a design-review board for a new payments platform

The situation

A mid-sized payments company is building a new platform: a payment-intent API, a double-entry ledger, merchant payouts, and adapters to a card network and two banks. About 60 engineers touch it across five teams. Launch is planned in five months, regulators expect an audit trail, and the business target is no duplicate charges and no lost money. The stack is Kubernetes, Envoy, PostgreSQL, Kafka, Redis, Terraform and Prometheus with Grafana.

The pain is in the review process itself (illustrative): design documents run to 40 pages, review meetings last two hours and end without decisions, and in the last two launches three incidents traced back to failure modes nobody had asked about. The board is five people: two staff engineers, one SRE, one security engineer and a product lead.

Starting architecture

This is the proposal the board receives.

The review has to decide where the hard problems sit: the payment-to-ledger-to-network step, the payout path, and the Kafka event flow.

Options considered

The decision

Rollout and verification

The pre-read is the case-study template from this lesson, trimmed to five pages, with one mandatory section: the six questions answered for each component. Reviewers send questions in writing, so the meeting spends its time on the three hardest ones. Every decision lands in a log, and every accepted risk gets an owner, an alert and a revisit date. A game day before launch injects the failures the review named. What you would measure: the share of risks with a tested alert, review lead time, defects found before versus after launch, and the number of incidents that map to a review item.

The 2 a.m. failure

The page reads: "Payments in AUTHORISING state older than 5 minutes: 140". The card network adapter timed out and the payment service could not tell whether the charge succeeded. The review had flagged exactly this ambiguity and accepted it, with an age alert as the safeguard.

Detection is the payment_intents_oldest_pending_seconds gauge. The first three checks: count stuck intents by state and merchant, read the adapter's error rate and circuit breaker state, and look up a sample of intent ids at the network by idempotency key. Mitigation: route new authorisations to a secondary route if configured, and run the reconciler by hand. Do not let anyone retry a stuck payment with a new idempotency key, because that is how an approved but unconfirmed authorisation becomes a duplicate charge. Follow-up: make reconciliation automatic and add a timeout budget to the adapter.

Architect's notes

• Review risk, not size. Spend board time on money movement and irreversible actions; give low-risk services a checklist.
• Make the author do the work. A pre-read that answers the six questions is the cheapest defect finder you have.
• Ask "who gets paged?" An accepted risk with no alert and no owner is an undiscovered incident.
• Record rejected options. Next year's team will otherwise re-argue them.
• Defer on purpose. Multi-region active-active is rarely a launch need; write down the trigger and move on.
• Watch the gate. If teams avoid the board, the process is too heavy, so lighten it rather than enforce it.

Common mistakes

• Memorising architectures instead of reasoning. A memorised diagram collapses when requirements change. Practise deriving the design from constraints, then compare.
• Skipping estimation. Without numbers you cannot tell whether a database is enough or whether you need sharding. Even rough orders of magnitude change the design.
• Designing only the happy path. Real systems spend their lives in partial failure. Walk each component and ask what happens when it dies or slows down.
• Adding technology to look sophisticated. Kafka, a service mesh and CQRS all carry operating costs. Add them only when a stated requirement needs them.
• Reading the solution before attempting the problem. It feels productive but builds recognition, not recall. Attempt first, always.
• Never stating trade-offs. Every choice gives something up. A design that claims no downside is hiding one.

Key takeaways

• Three facts drive everything. Unreliable networks, disagreeing clocks and partial failures cause nearly every distributed-systems problem.
• Techniques are priced. Each guarantee costs latency, availability, or complexity; learn the price along with the technique.
• The problems form a map. Agreement, ordering and replication feed into transactions, events and multi-region designs.
• Ask six questions of every design. Lost, duplicated, or late messages; node death mid-operation; partitions; source of truth; needed guarantees; blast radius.
• Attempt before you read. Practise case studies out loud with a timer, then compare and record the gaps.
• Use one template. Requirements, estimate, API, data model, design, deep dives, bottlenecks, failures, trade-offs, cheat sheet.
• Label assumptions. Honest, illustrative numbers beat confident invented ones.
• Tools implement ideas. Judge any technology by which problems it solves and which guarantees it leaves to you.

Practice exercises

Exercise 1 — Starter (30 min): Without looking at the map, list five things that could go wrong when an order is placed across a cart service, a payment service and an inventory service. Then place each one on the hard-problems map and note which lesson addresses it.

Exercise 2 — Intermediate (2 hours): Read the notification-service design linked above. Fill in the case-study template for it from memory of the article: requirements, estimate, API, data model, high-level design, deep dives, failure modes, trade-offs. Mark every section where the article was silent, and write what you would add.

Exercise 3 — Advanced (half day): Pick a system you use daily (a ticketing site, a messaging app, a payments app). Run a full 45-minute out-loud session on it following the time boxes. Record yourself. Afterwards, listen back and list every moment you hand-waved, then write the six-question review of your design: lost message, node death, partition, source of truth, needed guarantees, blast radius.

Related reading

• Designing a Notification Service - a worked design that follows a similar arc: requirements, capacity estimate, architecture, component deep dive, trade-offs, scaling, failure handling and an interview cheat sheet.
• Building Blocks Overview - the toolbox this course assumes you already know.
• CAP and Consistency Models - the consistency vocabulary used throughout the advanced lessons.
• Reliability Basics - the failure-handling foundations that resilience patterns build on.