System Design: Requirements and Scope

How to read it: a phrase is only a requirement once it passes all four questions: which action, what number, which percentile and window, and what it changes.

Latency: use percentiles, not averages

Latency is how long a request takes from the user's point of view. An average hides the pain. If 99 requests take 100 milliseconds and one takes 10 seconds, the average is about 200 milliseconds, yet one user in a hundred waited ten seconds. So we use percentiles:

• p50 (the median): half of requests are faster. The typical experience.
• p95: 95% are faster; one in twenty is slower.
• p99: 99% are faster; one in a hundred is slower. This is the "tail".

A requirement should read like "p50 under 100 ms, p99 under 500 ms for the invoice list endpoint". The tail matters more than it seems, because one page view often triggers many backend calls. If a page makes twenty calls and each has a 1% chance of being slow, about 18% of page views (1 minus 0.99 to the power 20) hit at least one slow call.

Throughput

Throughput is work per unit time: requests per second, invoices per hour. Quote it for the average and the peak, since you build for the peak. The Estimation lesson shows how to derive it.

Availability and the nines

Availability is the fraction of time the system works, quoted in "nines" (approximate, for a 365-day year).

Each extra nine cuts allowed downtime by ten and raises cost steeply. Two nines to three may mean running more than one copy of everything. Four or five usually means multiple data centres, automated failover and careful operations. Match the number to the cost of downtime: a photo app down for an hour annoys people, a payment processor that is down for an hour loses money and trust.

Durability and consistency

Durability means data, once acknowledged as saved, is not lost. It is separate from availability: a system can be down for an hour and lose nothing, or be up all the time while quietly losing data, which is a disaster. "A saved invoice never disappears" pushes you toward replicated storage and tested backups.

Consistency asks: after a write, when do readers see it? Strong consistency means immediately everywhere. Eventual means soon, not instantly. A dashboard total can lag by seconds. "Is this invoice number already used?" cannot. Decide per feature, because strong consistency costs latency and availability. See CAP and Consistency.

The rest of the list

• Security and privacy: who may see which invoice, how data is encrypted, which laws apply.
• Scalability: how load grows and whether the design must absorb it without a rewrite.
• Cost: a monthly budget turns "store everything forever in the fastest tier" into a decision.
• Maintainability: a simple design three people can run beats an elegant one that needs thirty.

How requirements drive architecture

A requirement is useful when you can say what it changes in the design.

A CDN keeps copies of files near users, a replica is a copy of the database, failover means switching to a standby when the main part fails, and a queue holds work for later. Each gets a full lesson soon.

• p99 under 500 ms with users worldwide suggests a CDN and caches near the application. One region serving everyone cannot beat the speed of light.
• 99.9% availability (about 8.76 hours a year) makes a single server that needs manual repair a poor bet: one bad night can use up the whole year's allowance. You will usually want at least two application instances and a database with a standby.
• Read-to-write ratio of 100 to 1 points to caches and read replicas. A write-heavy workload points to partitioned, append-friendly storage.
• Spiky traffic favours queues: absorb the burst, process at a steady pace.
• Strong consistency for payments points to a transactional relational database, not lagging replicas.

For the photo-sharing running example, the same discipline gives a table like this. Every number is an illustrative assumption.

Notice the deliberate asymmetry: less availability for uploads than viewing. That is a design decision, and it saves money.

In practice: an invoicing SaaS turns "fast and reliable" into SLOs

The situation

A B2B invoicing product serves about 800 small businesses. Eight engineers run it on managed Kubernetes: a Node.js API, PostgreSQL, Redis, RabbitMQ for background jobs, and S3-compatible storage for invoice PDFs, with Prometheus and Grafana for metrics. Sales has just signed the first larger customer, whose contract says the service must be "fast and reliable" and carries a 99.9% availability clause. Nobody can say whether the product meets it today.

Assume about 60,000 invoices a month, 70% of them in the last three days of the month. The pain is vague but real: support hears "invoices are slow" at month-end, and one incident left PDFs undelivered for three hours before anyone noticed, because every dashboard was green on CPU and memory.

Starting architecture

How to read it: the user-visible journey has a synchronous part (create invoice through the API) and an asynchronous part (PDF and email through workers). Each part needs its own requirement.

Options considered

The decision

The four SLOs, each with its indicator, target and window:

The internal availability SLO (99.95%, assumed to be what the baseline supports) is deliberately tighter than the 99.9% in the contract. Over 30 days it gives an error budget of about 22 minutes (43,200 minutes multiplied by 0.05%); the contract's 99.9% would allow about 43.

Rollout and verification

Measure first, commit later. For four weeks record the indicators in Prometheus without alerting, then set each SLO just inside what the baseline shows. Alert on budget burn, not on raw errors: a fast alert pages the on-call if the last hour burns budget at about 14 times the sustainable rate (that would spend 2% of a 30-day budget in an hour), and a slow alert opens a ticket for a steady leak. Add a queue-age metric, the age of the oldest unprocessed PDF job, because it is the one that would have caught the three-hour incident. Verify by replaying last month's traffic shape in a staging load test and by restoring last night's backup into a scratch database.

The 2 a.m. failure

The page reads "PDF ready SLO fast burn: 41% of invoices older than 60 s without a PDF". The cause: a customer uploaded a template with a 40 MB embedded logo, the PDF worker ran out of memory and was killed, the message was redelivered, and the next worker crashed on it too.

First three checks: the queue-age panel in Grafana, kubectl get pods -l app=pdf-worker (looking for OOMKilled and restarts), and rabbitmqctl list_queues name messages_ready messages_unacknowledged to see the backlog. Mitigation: pause the consumer, move the poison message to a dead-letter queue, and let the backlog drain. Follow-up: cap template asset size at upload, set a maximum redelivery count with a dead-letter route, and give the worker a memory limit that matches the cap.

Architect's notes

• Measure before you promise. Avoid signing a number you have not observed for a few weeks. The baseline decides the target, not the other way round.
• Set the SLA looser than the SLO. The gap is your safety margin and the room your team needs to ship changes.
• Pick journeys, not components. "Database CPU" is a cause. "Invoice PDF ready in 60 s" is what the customer feels. Alert on the second.
• Ask in the review: who is paged and what can they do? An SLO with no owner and no runbook is decoration.
• Defer the extra nine. Cost grows faster than the benefit. Write down the trigger (an actual customer need) that would justify it.

Common mistakes

• Starting with technology. Choosing Kafka or microservices before the requirements locks in decisions with nothing to justify them. Fix: write requirements first, then pick tools.
• Vague non-functional goals. "Fast", "scalable" and "reliable" cannot be tested. Fix: a number, a unit, a percentile and a window.
• Quoting the average latency. It hides the slow tail users feel. Fix: state p50 and p99, and measure them.
• Asking for five nines by default. Each nine multiplies cost. Fix: match availability to the real cost of downtime, per feature.
• No explicit non-goals. Unlisted scope creeps in. Fix: write an "out of scope" list and refer back to it.
• Treating requirements as fixed. Early numbers are guesses. Fix: record assumptions and revisit them.
• Confusing availability with durability. Fix: set the two goals separately.

Key takeaways

• Two kinds of requirement. Functional says what the system does; non-functional says how well, and shapes the architecture most.
• Scope is a fence. Write what is in, out and later, and keep the out list visible.
• Ask in groups. Users, features, usage pattern, data, quality and constraints cover most questions.
• State assumptions. When answers are missing, write one down so it can be challenged.
• Percentiles beat averages. Use p50, p95 and p99; the tail is what users complain about.
• Nines are expensive. Each extra nine cuts annual downtime tenfold and raises cost sharply.
• Availability is not durability. Set separate goals for staying up and for never losing data.
• An SLO is a requirement with an indicator, a target and a window. It implies an error budget the team can spend on risk.

Practice exercises

Exercise 1 — Starter (30 min): Take the idea "a service that shortens URLs." Write five clarifying questions you would ask, grouped using the table in this lesson. Then write three functional requirements and an out-of-scope list of at least three items.

Exercise 2 — Intermediate (2 hours): For the invoicing product, convert each of these into a measurable requirement with a number, percentile and window (or a labelled assumption): "invoices should be fast," "PDFs should arrive promptly," "the site should almost never be down." For availability, use the nines table to compute the allowed downtime per month for your target (divide the yearly figure by 12) and decide whether a single server could ever meet it.

Exercise 3 — Advanced (half day): Write a one-page requirements document for a food-delivery order-tracking feature. Include scale assumptions, a table of non-functional requirements with an SLI and SLO for each, and a "Conflicts" section listing two pairs of requirements that pull against each other (for example, freshness of location versus cost). For each conflict, state which side you favour and why. Then list which architecture choices (cache, queue, replica, region count) each requirement pushes you toward.

Related reading

• System Design: The Big Picture - the three questions in context, with the full architecture tour.
• Estimation - the next lesson, which turns scale assumptions into requests per second, storage and bandwidth.
• A Repeatable Approach - where requirements fit in the full design framework.
• Observability and SLOs - how requirements become SLIs, SLOs and error budgets in production.