What you'll learn: By the end of this lesson you will be able to (1) run the same eight-step method on a problem you have never seen, producing a named artifact at each step, (2) apply it end to end to a rate limiter, a flash-sale ticket booking system and a real-time leaderboard, (3) use the twelve shared practices (trade-off matrix, API contract, authentication flow, correlation ids, STRIDE, ADRs and more) as short, reusable tools, and (4) review any design with a one-page checklist and a list of anti-patterns.
A pilot flies a different route every day but never improvises the checklist. The route changes; the order of checks does not. System design is the same craft. The ten QuickCart stages gave you one route in depth; this lesson gives you the checklist.
This last lesson is not a QuickCart stage. It extends System Design: A Repeatable Approach, which introduced the eight steps on a small paste-bin service. Here each step gets its questions, its artifact and its QuickCart lesson, and the method meets three problems QuickCart never faced.
Where QuickCart is now
After ten stages, QuickCart is an online grocery marketplace with a load balancer, caches and a CDN (Stage 2), a deliberate mix of data stores (Stage 3), sharded orders (Stage 4), an asynchronous order pipeline (Stage 5), a ledger (Stage 6), search, notifications and media (Stage 7), services (Stage 8), resilience (Stage 9) and security and operations (Stage 10). Its targets, all illustrative assumptions, are 2 million daily users and 500,000 orders a day.
The pain now is different: a new team asks you to design something unrelated to groceries. QuickCart's answers do not transfer; the questions that produced them do.
The core idea
A design is a sequence of decisions, and a method makes the sequence explicit so you can inspect it. Each step produces an artifact (a list, a table, a figure) that the next step consumes. If an artifact is missing, the next step is a guess. If a later step contradicts an earlier artifact, change the artifact, not the story.
Figure 1. The eight steps and the artifact each one produces.
scope list"] --> S2["2 Estimate
numbers table"] S2 --> S3["3 API
operations list"] S3 --> S4["4 Data
entities and queries"] S4 --> S5["5 High-level
one figure"] S5 --> S6["6 Deep dive
mechanism"] S6 --> S7["7 Failures
failure table"] S7 --> S8["8 Trade-offs
decision record"] S8 -.->|"new fact"| S1
How to read this: the solid arrows are the normal order; the dotted arrow is the rule that any new fact (an estimate that breaks a requirement, a failure that needs a new component) sends you back to the artifact it changes.
The method in eight steps
For each step: what to ask, what to produce, and where QuickCart taught it.
Step 1: Clarify requirements and scope
- Ask: who uses it, what are the three key actions, what is out of scope, and what do "fast", "available" and "correct" mean in numbers?
- Produce: an in and out scope list, non-functional targets with numbers, written assumptions.
- Taught in Stage 1: functional versus non-functional requirements, SLOs.
Step 2: Estimate
- Ask: how many users, requests per second at average and peak, what read-to-write ratio, how fast does data grow, which resource runs out first?
- Produce: a numbers table with arithmetic, labelled as assumptions, ending "the hard part is ...".
- Taught in Stage 1: 40 million page views a day became about 463 per second, about 2,300 at peak.
Step 3: Define the API
- Ask: what are the operations, who calls them, which writes must be idempotent, how are lists paged, how are errors reported?
- Produce: an operation list with the idempotency rule for each write.
- Taught in Stage 8 and Stage 6.
Step 4: Model the data
- Ask: what are the entities, the top three queries, what must be strongly consistent, how big is each record?
- Produce: entities, keys, access patterns and a store per entity with a reason.
- Taught in Stage 3 and Stage 4.
Step 5: Draw the high-level design
- Ask: what is the smallest design that meets the numbers, and which requirement justifies each box?
- Produce: one figure tracing the main read path and the main write path.
- Taught in Stage 2: each stage added only what a number demanded.
Step 6: Deep-dive the riskiest parts
- Ask: where would a mistake cost the most money or trust?
- Produce: the mechanism step by step, a worked example and the invariant it protects.
- Taught in Stage 5 and Stage 6.
Step 7: Walk the failures
- Ask: what breaks at ten times the load, and what happens when each box dies, slows down or returns wrong answers?
- Produce: a failure table: failure, effect, defence, how you will notice.
- Taught in Stage 9.
Step 8: State the trade-offs
- Ask: what did each choice buy, what did it cost, and what would make it wrong?
- Produce: a trade-off table and, for the choices that matter, a decision record with a measurable "revisit when".
- Taught in every stage's decisions table, and Stage 10 for cost.
All figures in the three problems below are illustrative assumptions chosen to keep the arithmetic easy. They do not measure any real system.
Problem 1: a rate limiter shared by many APIs
A platform team runs 20 internal and public APIs. Today each API limits traffic in its own way. The team wants one shared service that any API can ask, "may this caller do this now?" For the algorithms themselves, see Rate Limiting; here the job is to design the service.
Requirements
- In scope: answer allow or deny for a (subject, rule) pair; per-tenant, per-plan and per-route rules; return remaining quota and a retry delay; change rules without a deploy.
- Out of scope: monthly billing quotas, bot detection, DDoS protection at the network edge.
- Targets: the check adds under 5 ms at the 99th percentile; the limiter is more available than the APIs it protects; limits may be exceeded by a few percent, never by a multiple.
Estimates
| Quantity | Assumption | Result |
|---|---|---|
| Peak traffic across all APIs | 50,000 requests per second | 50,000 checks per second |
| Active keys | 2 million (tenant by route combinations) | 2,000,000 keys |
| State per key | about 100 bytes with overhead | 200 MB in total |
| Check traffic | about 200 bytes per call | about 10 MB per second |
Conclusion: the state fits in memory on one machine; the pressure is request rate and tail latency, so availability and the failure policy matter more than storage.
API
POST /v1/check
{ "rule": "api-default", "subject": "tenant_42", "cost": 1 }
200 { "allowed": true, "remaining": 19, "retry_after_ms": 0 }
200 { "allowed": false, "remaining": 0, "retry_after_ms": 200 }A denial is a normal answer, so the call returns 200; the calling API turns it into 429 Too Many Requests (RFC 6585) with a Retry-After header (RFC 9110, section 10.2.3). Callers use a small client library that caches rules and enforces the 5 ms timeout.
Data model
Rules are small and read often: a table with rule, algorithm, capacity, refill_per_second and fail_mode. Counters are tiny, hot and disposable: one Redis hash per key holding tokens and a timestamp, with an expiry. A counter lost in a crash costs a short period of extra generosity, not data.
High-level design
Figure 2. Rate limiter service: callers, limiter, counter shards and rules.
How to read this: each key hashes to one shard, so a check touches exactly one counter; rules flow to the service as data, so changing a limit never needs a deploy; the client library, not the service, decides what to do when the limiter is slow.
Why a service, not each API talking to Redis? One place to change algorithms and observe decisions. The cost is one more hop and one more thing to keep up.
Deep dive: one atomic check
Two service instances can read the same counter at the same moment; if both see 1 token and both allow, one request is never counted. Read, refill, decision and write must be one indivisible step. Redis states that while a script runs, all other server activity is blocked, so its effects either have not happened or already happened. The script below (Lua, shown as plain text) is a token bucket. It uses the server's clock through TIME, avoiding skew between instances; Redis replicates a script's write commands rather than the script (the default since Redis 5.0), so calling TIME before a write is allowed.
-- KEYS[1] = bucket key; ARGV = capacity, refill per second, cost
local t = redis.call('TIME')
local now_ms = t[1] * 1000 + math.floor(t[2] / 1000)
local data = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local cost = tonumber(ARGV[3])
local tokens = tonumber(data[1])
local ts = tonumber(data[2])
if tokens == nil then
tokens = capacity
ts = now_ms
end
tokens = math.min(capacity, tokens + (now_ms - ts) * rate / 1000)
local allowed = 0
local retry_ms = 0
if tokens >= cost then
tokens = tokens - cost
allowed = 1
else
retry_ms = math.ceil((cost - tokens) * 1000 / rate)
end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now_ms)
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate * 1000) * 2)
return { allowed, math.floor(tokens), retry_ms }Worked example: capacity 20, refill 5 per second, cost 1. A burst of 20 checks in the same instant is allowed and the 21st returns {0, 0, about 200}: tokens arrive every 200 ms. The expiry is twice the time to refill from empty, so an idle caller's key disappears, which is equivalent to a fresh full bucket. Scripts are not persisted by the server, so clients call EVALSHA and reload the script if it is missing. Each script touches one key, so it works across cluster shards.
Figure 3. A check that fails open.
How to read this: the second request shows the policy that matters most; a login rule would instead fail closed and return a denial.
Failure modes
| Failure | Effect | Defence |
|---|---|---|
| A Redis shard dies | Keys on it lose state; checks time out | Replica takes over; clients apply fail_mode per rule (open for general traffic, closed for login and password reset); alert on the fail-open rate |
| One tenant is a hot key | One shard takes disproportionate load | Split its limit across sub-keys; a local deny cache in the client for callers far over limit |
| Bad rule pushed | A whole API throttled | Rules validated, rolled out per rule with a shadow mode that counts but does not block |
| Limiter service slow | Every API pays latency | 5 ms client timeout, small connection pool, bulkhead per caller |
Trade-offs
| Decision | Chosen | Cost | Revisit when |
|---|---|---|---|
| Central counters instead of local limits | Accurate per-key limits | A network hop and a shared dependency | Check rate makes the store the bottleneck: then batch token reservations |
| Token bucket | Bursts allowed, cost per request | Two numbers to tune per rule | A rule needs exact counts in a window (a login limit) |
Problem 2: ticket booking with a flash sale
An event has 50,000 seats and tickets go on sale at a fixed time. Far more people want seats than exist, they arrive in the same minute, and no seat may be sold twice. It resembles QuickCart's inventory step in Stage 5, with the contention concentrated on a few thousand rows.
Requirements
- In scope: view the seat map; hold up to 4 seats for 5 minutes; pay; release on expiry or cancel; a waiting room for the crowd.
- Out of scope: resale and dynamic pricing.
- Targets: a seat is never sold twice; a payment that succeeds always ends in a ticket or a refund; the seat map is at most a few seconds stale; the site survives a crowd far larger than the capacity.
Estimates
| Quantity | Assumption | Result |
|---|---|---|
| Seats | 50,000, about 200 bytes per row | about 10 MB of data |
| Fans arriving in the first minute | 400,000 | 400,000 / 60, about 6,700 arrivals per second |
| Seat-map polling, if everyone polls every 5 s | 400,000 / 5 | 80,000 requests per second |
| Orders at 2 seats each | 50,000 / 2 | at most 25,000 orders |
| Users in seat selection at once | cap of 2,000, average 120 s stay | admission rate 2,000 / 120, about 17 per second |
| Time to sell out if all admitted users buy | 25,000 / 17 | about 1,470 s, roughly 25 minutes |
The admission figure uses Little's law: users in a stage equal arrival rate times time spent there, so a cap of 2,000 and a 120 s stay admits about 17 per second. Conclusion: the data is tiny; the hard parts are row contention and crowd control, and the waiting room is what keeps the database inside its limits.
API
POST /v1/queue/join -> 202 { position, token }
GET /v1/queue/status -> 200 { state: "waiting" | "admitted", admit_token }
GET /v1/events/{id}/seats -> 200 seat map, cacheable for about 1 s
POST /v1/holds (Idempotency-Key) -> 201 { hold_id, expires_at } | 409 { unavailable: [...] }
POST /v1/holds/{id}/checkout -> 202 payment started | 410 hold expired
DELETE /v1/holds/{id} -> 204Only a caller holding an admission token can create holds. POST /v1/holds takes an idempotency key because mobile clients retry; a repeat returns the original hold.
Data model
Seats are the one table that must be correct. A relational database with row locking fits: the data is small and the invariant strict. One event's seats live on one shard.
CREATE TABLE seats (
event_id bigint NOT NULL,
seat_id int NOT NULL,
status text NOT NULL DEFAULT 'available'
CHECK (status IN ('available', 'held', 'sold')),
hold_id uuid,
hold_expires_at timestamptz,
order_id bigint,
PRIMARY KEY (event_id, seat_id)
);Orders sit in the same database, so confirming a sale is one transaction. The seat map and queue positions live in Redis and can be rebuilt.
High-level design
Figure 4. Flash-sale ticket booking.