What you'll learn: By the end of this lesson you will be able to name the main building blocks of a large system and say what problem each one solves. You will match a symptom, such as slow reads or traffic spikes, to the block that addresses it. You will know in what order to study the rest of this course and where Foundations ideas reappear at scale.
Think about how a city works. Roads move people, power lines deliver electricity, water mains bring water, post offices move parcels, and libraries store knowledge. Each piece does one job and has its own rules. A city planner does not invent a new kind of road every time; she chooses from a known set of parts and decides where to place them.
Software architecture has a similar toolbox. When people say "use a cache", "put a queue there" or "shard the database", they are choosing from a small set of well-understood building blocks. Each one exists because a specific problem keeps showing up. This lesson is a map of those blocks. The lessons that follow zoom into each one.
The core idea
Every building block is a deliberate trade. It solves one problem by accepting a new cost, usually more complexity, some staleness, or a weaker guarantee. A cache makes reads fast but can be out of date. A queue smooths spikes but makes the system asynchronous. Sharding scales writes but makes cross-shard queries hard.
So the skill is not knowing every block. It is recognising the problem first, then picking the cheapest block that solves it, and knowing when not to use it. If you can say "we have problem X, so we use block Y, which costs Z", you are designing, not decorating.
The Foundations course gave you the vocabulary: requests, latency, load balancers, caches, databases. This course takes each of those ideas to the scale where simple versions stop working.
The map on one page
Here are the blocks grouped into layers, using our photo-sharing app as the running example.
Read it as a story. A request arrives at the edge, where a CDN may answer it right away. Otherwise a load balancer and gateway route it to a stateless service. The service checks the cache, and on a miss reads from the data layer. Anything slow, such as resizing an uploaded photo, goes onto a queue for a worker to handle later. Changes are published to an event stream so that other systems, such as the search index, can keep up.
The layers
Compute. The stateless services and background workers that run your code. Covered in the Foundations lessons on scaling.
Data layer. Where facts live. Includes the primary database, replicas that copy it, partitioning (sharding) that splits it, specialised stores such as search indexes, and object storage for large files. Lessons: Replication, Sharding, CAP and consistency, Search and indexing, Storage engines.
Caching. Remembers expensive results so they need not be computed again. Lesson: Advanced caching.
Messaging. Lets parts of the system talk without waiting for each other. Lessons: Message queues, Event streaming.
Edge and platform. The front door and the plumbing between services: gateways, service discovery, rate limiting, architecture style. Lessons: Architecture styles, API gateway and service mesh, Rate limiting.
Operations and trust. How you see what is happening and keep it safe. Lessons: Observability and SLOs, Security.
From problem to building block
Most real design conversations start with a symptom. This table maps common symptoms to the block that usually helps, what it costs, and what to try first.
| Problem you see | Block that usually helps | What it costs | Try this first |
|---|---|---|---|
| Reads are slow or the database is overloaded by reads | Cache, then read replicas | Staleness, invalidation logic | Add an index; cache the hottest queries |
| One database machine cannot hold or write all the data | Sharding | Cross-shard queries and transactions get hard | Bigger machine, read replicas, archive old data |
| Database is a single point of failure | Replication with failover | Replication lag, failover complexity | A standby with tested failover |
| Traffic spikes make requests time out | Queue between producer and consumer | Asynchrony, harder debugging | Autoscale the stateless tier |
| Slow work blocks the user (video, email, image resize) | Queue plus workers | Eventual completion, retries needed | Move only the slow step off the request path |
| Several systems must react to the same change | Event stream | Ordering and replay design | A simple queue or direct call if only one consumer |
| Users want search, typeahead or relevance ranking | Search index | A second copy of data to keep in sync | Database full-text features at small scale |
| A client or bug floods an API | Rate limiting | Some legitimate requests rejected | Basic per-key limits at the gateway |
| Many services, many teams, tangled deployments | Architecture style, gateway, service mesh | Operational overhead | A modular monolith |
| You cannot tell why the system is slow | Logs, metrics, traces, SLOs | Cost of telemetry | Start with request rate, errors, duration |
| Data or credentials may be exposed | Authentication, authorisation, encryption | Friction, key management | Least privilege and TLS everywhere |
Notice the last column. In most rows the "try this first" option is simpler than the block itself. Experienced engineers reach for the heavy block last, not first.
The same logic works as a decision flow. Start from the symptom, answer honestly, and stop at the first block that fits.
Read it as a priority order, not a menu. The first "yes" is usually the cheapest fix, and the last exit, measuring, is the answer more often than people expect.
How the blocks combine
Blocks rarely work alone. Real flows chain them. Look at what happens when a user uploads a photo.
Here the database is the source of truth, the queue protects the user from slow work, the worker does that work, and the search index is a derived copy. Each block is simple. The design skill is in the seams: what if the job is delivered twice, what if the search update fails, what if the worker crashes halfway? Those questions are why the later lessons spend so much time on retries, idempotency, ordering and consistency.
How Foundations ideas reappear at scale
Nothing in this course is entirely new. Each block is a Foundations idea under pressure.
| Foundations idea | What happens at scale | Where you will see it |
|---|---|---|
| Caching basics (cache-aside, TTL) | Stampedes, hot keys, cache and database drifting apart | Advanced caching |
| Databases basics (indexes, ACID) | One machine is not enough: replicate and partition | Replication, Sharding |
| Load balancing (algorithms) | Consistent hashing places data, not only requests | Sharding, Advanced caching |
| Reliability basics (retries, idempotency) | Delivery guarantees and duplicate handling in messaging | Message queues, Event streaming |
| Scaling basics (stateless tiers) | State must go somewhere, so state needs its own scaling story | Replication, Sharding |
| Request lifecycle | Many hops, so latency and failure multiply | Gateway, mesh, observability |
| Estimation | Decides whether you need the block at all | Every lesson |
If any row feels shaky, revisit System Design: Caching Basics, System Design: Databases Basics or System Design: Reliability Basics first.
Suggested reading order
You can read the lessons in the listed order, or by need. The order below builds each idea on the previous one.
- Data first: Replication, then Sharding, then CAP and consistency. These explain what happens to data once it lives on more than one machine, and they set the vocabulary of consistency used everywhere else.
- Speed: Advanced caching. It builds directly on replication and consistency.
- Asynchrony: Message queues, then Event streaming.
- Structure: Architecture styles, then API gateway and service mesh, then Rate limiting.
- Specialised data: Search and indexing and Storage engines.
- Running it safely: Observability and SLOs and Security.
If you are preparing for a specific problem, jump by the table above. If you simply want a strong base, follow the numbered order.
Choosing blocks responsibly
A few habits will serve you across the whole course.
Let numbers justify blocks. A site with 50 requests per second does not need sharding or an event stream. Estimation turns "feels slow" into "we need 50,000 reads per second and one database serves about a tenth of that" (an illustrative figure, since real capacity depends on hardware and queries).
Prefer fewer moving parts. Every block adds something to deploy, monitor, secure and wake up for at night. A modular monolith with a database, a cache and one queue can carry a surprising amount of load.
Plan the seams. Where two blocks meet, ask three questions: what if the message is duplicated, what if it is late, what if it is lost?
Decide the consistency you need, per feature. A photo's like count can be a little stale. A payment cannot. Different parts of one system can sit at different points on the trade-off.
Keep a way back. Blocks like sharding are painful to undo, so adopt them only after cheaper options are exhausted. Blocks like caching are easy to add and remove, which makes them good first experiments.
In practice: a two-sided marketplace platform
A realistic scenario, not a case history. This is a composite example built from common production patterns. It does not describe any specific company, and every number in it is an illustrative assumption.
The situation
A marketplace connects sellers of handmade goods with buyers. Assume 6,000 active sellers, 150,000 listings, 40,000 buyers a day and 18 engineers in three teams. It runs as a Spring Boot modular monolith on Kubernetes, with PostgreSQL as the only database and images in S3-compatible object storage. Search is a SQL ILIKE query. Order confirmation emails are sent inside the checkout request through an SMTP provider.
What hurts, all illustrative: search p95 is about 3 seconds and rising with every new listing, checkout p95 is 2.5 seconds because it waits on the email provider, database CPU sits near 85% on weekend evenings, and sellers complain that image uploads time out. The CTO asks where each building block belongs.
Starting architecture
Everything shares one process and one database. Any slow dependency, such as email or search, slows checkout.
Options considered
| Option | What it fixes | What it costs | Risk |
|---|---|---|---|
| A. Smaller change: PostgreSQL full-text search with a GIN index, and email sent from an in-process background thread | Search latency for now, email wait | Days of work | Lost emails if a pod restarts; search relevance stays basic |
| B. Add blocks one at a time: RabbitMQ for email and image processing, a CDN for images, Redis for listing pages, then OpenSearch if needed | Slow work off the request path, read load, search quality | About two engineer-months in phases | Index and cache are derived copies that can drift |
| C. Split search, orders and notifications into services with Kafka between them | Team autonomy, independent scaling | Six months, a platform team | Distributed-monolith risk with no measured need |
The decision
| Context | One database does everything; search and email dominate latency; teams are not blocked by deployments yet |
| Decision | Option B in this order: queue for email and image jobs (with direct-to-storage uploads), then CDN and cache, with A's PostgreSQL full-text search in the meantime, then a search index only if that misses a 300 ms p95 target. Keep the monolith |
| Consequences, good | Checkout stops waiting on email; database read load falls; the monolith stays one deployable unit |
| Consequences, bad | Email becomes at-least-once, so consumers must be idempotent; derived data needs rebuild tooling |
| Revisit when | Listings pass about 2 million, a team is blocked by shared deployments, or search needs facets and typo tolerance that PostgreSQL cannot give |
Rollout and verification
The target architecture shows where each block lands and which stores are sources of truth.
Roll out one block per release, each behind a flag. For the search index, run it in shadow first: send every query to both PostgreSQL and OpenSearch, compare results and latency offline, and switch users only when the numbers agree. Keep a dual-write period short, and rebuild the index from PostgreSQL before cutover to prove you can. Dashboards: checkout p95, search p95, queue depth and oldest message age, cache hit ratio, database CPU, and an index freshness gauge (newest listing change in PostgreSQL versus in OpenSearch).
The 2 a.m. failure
The page reads "Queue index-sync: oldest message age above 10 minutes". Sellers report that edited prices do not show in search and sold items still appear.
First three checks: rabbitmqctl list_queues name messages consumers, kubectl logs deploy/indexer --tail=50 for the repeating stack trace, and kubectl get pods for restart loops. Mitigation: if no dead-letter limit exists yet, move the offending message by hand, restart the indexer, and re-index affected listings from PostgreSQL. As a stopgap the listing page re-checks the sold flag in the database so buyers never see a stale "available". Follow-up: a delivery limit with a dead-letter queue, an alert on oldest message age (not only depth), and a nightly reconciliation that compares counts and recent timestamps between PostgreSQL and the index.
Architect's notes
- Draw the source of truth on the diagram. Mark each store "truth" or "derived", and make sure every derived store has a rebuild path.
- Measure before adding a block. Search p95 by query type and the share of checkout time spent on email decide the order, not fashion.
- Ask in the review: "What does a user see if this block is down for an hour?" Each block needs a degradation story.
- Count the pagers. Every block adds an on-call surface. Name the owning team and the dashboard before it ships.
- Beware the vendor-defaults trap. Managed queues and search clusters hide their costs in storage, replicas and data transfer; estimate them first.
- Defer microservices. Keep module boundaries clean inside the monolith so a later split is a cut along existing seams.
Common mistakes
- Collecting blocks like trophies. Adding Kafka, Redis and a service mesh to a small app multiplies failure modes. Start from a measured problem and add one block at a time.
- Using a block for the wrong problem. A cache does not fix write overload, and a queue does not fix a slow query. Diagnose the bottleneck first.
- Forgetting the cost column. Every block brings staleness, complexity or weaker guarantees. Write the cost next to the benefit in your design notes.
- Treating derived data as truth. Search indexes, caches and replicas can be behind. Keep one source of truth and build a way to rebuild the rest.
- Ignoring the seams between blocks. Duplicates, delays and partial failures live between components. Design retries, idempotency and reconciliation on purpose.
- Skipping the simple option. An index, a read replica or a bigger machine often removes the need for a big architectural change.
Key takeaways
- Blocks are answers to recurring problems. Learn the problem first, then the block.
- Every block is a trade. Name what you gain and what you pay.
- Layers. Edge, compute, caching, messaging and data, plus operations and security around them.
- Start simple. Indexes, bigger machines, read replicas and caches come before sharding and event streams.
- Seams are where bugs live. Duplicates, delays and partial failures need explicit design.
- Derived data needs a source of truth. Caches, replicas and search indexes should be rebuildable.
- Foundations ideas return at scale. Caching, load balancing, retries and estimation all reappear with sharper edges.
- Numbers decide. If an estimate says one machine is enough, do not add a block.
Practice exercises
Exercise 1 — Starter (30 min): Copy the map diagram on paper. Mark where each of these appears in the photo-sharing app: a photo's bytes, its metadata, its thumbnail job, its search entry and the cache of the home page. For each, write which block holds it and whether it is a source of truth or a derived copy.
Exercise 2 — Intermediate (2 hours): Choose a service with these symptoms: reads are 50 times writes, traffic doubles every Friday evening, and video processing takes several minutes. Using the problem-to-block table, propose a design. For each block you add, write the cost it brings and the simpler alternative you considered first.
Exercise 3 — Advanced (half day): Pick one flow, such as "user posts a comment that notifies followers and updates search". Draw it as a sequence diagram with at least four blocks. For each arrow, list what happens if the message is duplicated, delayed for ten minutes, or lost, and write the mechanism (retry, idempotency key, reconciliation job) that handles it.
Related reading
- System Design: The Big Picture — the Foundations tour of a typical architecture that this course builds on.
- System Design: Replication — the first block to study: keeping several copies of data in step.
- System Design: Architecture Styles — how to organise code and teams around these blocks.
- Microservices best practices — practical guidance for the service-oriented parts of the map.
- Kafka scalability patterns — a deeper look at one messaging block at scale.