System Design: Observability and SLOs

What to measure: RED and USE

Staring at an empty dashboard, what do you put on it? Two compact checklists cover most needs.

RED: for services (request-driven)

• Rate: requests per second.
• Errors: how many fail, as a count or fraction.
• Duration: how long requests take, as percentiles.

RED describes what users of a service experience. Put it on a dashboard for every service, per endpoint where it matters.

USE: for resources (things that get used up)

• Utilisation: how busy the resource is (percent of time in use).
• Saturation: how much extra work is waiting (queue depth, run queue).
• Errors: resource-level error events.

USE applies to CPUs, memory, disks, network links, connection pools and thread pools. Saturation is the early warning: a connection pool at 100 percent utilisation with a growing wait queue is about to cause latency, even if CPU looks fine.

Use RED to learn that users are hurting and where, then USE to learn why. A related set, the "four golden signals" (latency, traffic, errors, saturation), covers the same ground.

SLIs, SLOs, SLAs and error budgets

Measuring is not the same as deciding what is good enough. Four terms get confused often, so define them carefully.

• SLI (service level indicator): a measurement of one aspect of service quality from the user's point of view, expressed as a ratio. Example: the proportion of checkout requests that return a successful response in under 500 ms.
• SLO (service level objective): a target for an SLI over a time window. Example: 99.9 percent of checkout requests are good, measured over 30 days.
• SLA (service level agreement): a contract with customers that states consequences, often financial credits, if a target is missed. An SLA is usually looser than the internal SLO, so you have room to react before breaching it.
• Error budget: the amount of unreliability the SLO permits. It equals 100 percent minus the SLO.

A worked example

Assume a checkout API receives 10 million requests in 30 days (illustrative). The SLO is 99.9 percent good requests. The error budget is 0.1 percent, which is 10,000 bad requests in the window.

Suppose a faulty release causes 4,000 failures. You have spent 40 percent of the budget in one incident. If the budget is nearly gone halfway through the window, the team slows risky changes and focuses on reliability work. If plenty remains, the team can release faster and run experiments. The budget turns the argument "reliability versus features" into a shared, quantified decision.

An error budget policy says in advance what the team does at each level of spend.

How to read it: Healthy means normal release pace, Cautious means smaller releases and reliability work first in planning, and Frozen means only fixes and rollbacks ship until the budget recovers. Agree the thresholds with product before you need them.

Time-based thinking helps too: a 99.9 percent availability target over a 30-day month allows about 43 minutes of full outage (30 days x 24 x 60 = 43,200 minutes, times 0.001). See the table of nines in requirements.

Choosing good SLIs

• Measure from the user's side. Prefer the load balancer or client view over a server's own "I am healthy" signal.
• Pick user journeys, not components. "Can a user complete checkout?" beats "Is the pricing service up?".
• Use ratios of good events to valid events, covering availability, latency and, where relevant, correctness, freshness and durability.
• Do not aim for 100 percent. It is unreachable, extremely costly, and slows all change.

Burn rate alerts

The burn rate is how fast you consume the budget relative to a steady rate that would exactly use it up over the window. A burn rate of 1 uses the budget exactly at the end of the window; 10 would exhaust it in a tenth of the time. A well-known approach is multi-window, multi-burn-rate alerting: page if the burn rate is high over both a short and a longer window (a fast burn that threatens the SLO within hours), and open a ticket for slow burns that threaten it within days. The two windows prevent flapping on brief spikes while still catching real trends.

Alerting on symptoms, not causes

An alert should mean "a human must act now". Everything else is noise, and noise trains people to ignore pages.

Cause-level signals are still valuable, but as dashboards and diagnostic data, or as low-urgency tickets, not pages. The exceptions are leading indicators of certain failure, such as a disk that will be full in four hours or a certificate expiring tomorrow. Those are worth a notification because the user impact is close to certain if no one acts.

Good alert hygiene: every page has a runbook link; alerts are actionable and urgent, otherwise they become tickets or dashboard items; alerts that never led to action are deleted in a monthly review; routing follows ownership; and alert fatigue is tracked as pages per shift and the fraction that needed action.

This flowchart sorts a signal into page, ticket or dashboard.

How to read it: the question is always user impact and time left, never how alarming a raw number looks.

Dashboards that help

A dashboard is a tool for a specific question. Three layouts work well: a service overview (RED panels at the top, then the SLO and error budget remaining, then dependencies), a resource view (USE panels for databases, caches, queues and pools), and a journey view (sign-ups, orders per minute, payment success), since technical health that hides business drops is a false comfort. Put the most important signal top-left, show deploy markers, keep panels few, and remove dashboards nobody has opened in months.

Putting it together in a design

Treat observability as part of the architecture: list the critical user journeys; define an SLI, SLO and error budget policy for each; instrument every service with RED metrics, structured logs and trace propagation (including gateway and mesh layers) and shared resources with USE metrics; create burn-rate alerts with runbooks; and plan cost through retention, sampling and cardinality limits.

Telemetry has its own failure modes. If your pipeline falls over during an incident you lose visibility exactly when you need it, so keep observability infrastructure at least as reliable as what it watches, and know how you would debug without it.

In practice: onboarding a booking platform to SLOs, error budgets and burn-rate alerts

The situation

A table and salon booking platform serves consumers on web and mobile and merchants through a dashboard. Fourteen engineers in three teams run about a dozen services on Kubernetes with PostgreSQL, Redis, RabbitMQ for confirmation messages, and an SMS provider. Telemetry is Prometheus and Grafana with Alertmanager, plus unstructured log files.

There are around 60 alerts, mostly on causes such as CPU, pod restarts and queue length, and the on-call engineer gets about 25 pages a week of which roughly 70 percent need no action (all numbers assumed). Yet a bad release that broke "confirm booking" for 40 minutes was discovered through a customer support ticket, not a page. The team wants fewer, better alerts and a shared way to decide between shipping and stabilising.

Starting architecture

How to read it: alerts watch internals, and logs cannot be joined to metrics or traces, so a page tells the engineer something is odd but not who is hurt.

Options considered

The decision

The arithmetic matters. Assume 2 million valid create-booking requests in 30 days. A 99.5 percent SLO gives an error budget of 0.5 percent, which is 10,000 bad requests. A burn rate of 1 uses exactly that budget in 30 days (720 hours).

At a burn rate of 14.4 the whole budget would be gone in about 50 hours (720 divided by 14.4), which is why it pages.

Rollout and verification

Measure for two weeks before choosing final targets, then run the burn-rate rules in a non-paging channel and check that they would have caught the 40-minute outage when replayed over historical data. Seed a fault in a game day (return errors from confirm-booking in staging, then a small canary in production) and time how long it takes the page to arrive. Track page precision (the share of pages that needed action), pages per shift, time to detect, and the budget remaining panel on the service overview dashboard.

The 2 a.m. failure

The page does not come from an SLO. It comes from a watchdog: "Heartbeat from Prometheus missing for 5 minutes". A release added a user_id label to the booking duration histogram. The number of time series exploded, Prometheus ran out of memory and restarted in a loop, and with no metrics there was nothing for the burn-rate rules to evaluate. Silence looked like health.

First three checks:

1. kubectl get pods -n monitoring and kubectl top pod -n monitoring to see restarts and memory.
2. Prometheus's TSDB status endpoint (/api/v1/status/tsdb) after it starts, to list metrics and labels with the most series.
3. kubectl rollout history deploy/booking-api to find the release that introduced the label.

Mitigation: roll back the release (or drop the label with metric_relabel_configs), and confirm the series count falls once Prometheus is up. If it cannot get through replaying its write-ahead log within its memory limit, raise the limit temporarily rather than deleting data. Follow-ups: set label_limit and sample_limit on scrape jobs, add a CI check that rejects unbounded labels, and alert on head series growth as a leading indicator. Treat the watchdog as a standing rule: an always-firing alert routed to an external dead-man's-switch service that pages when the heartbeat stops, so the check does not depend on the Prometheus that just failed.

Architect's notes

• Measure first. Spend two weeks collecting the SLI before setting a target; invented SLOs are either ignored or constantly breached.
• Ask in the review: what is the SLI for, who owns it and what happens at zero budget? An SLO without an owner and a policy is a dashboard.
• Watch the telemetry bill. Logs and high-cardinality metrics dominate cost; set retention, sampling and label limits per signal before launch, not after the invoice.
• Pages are a shared resource. Give each page a runbook and an owner, and delete pages that never lead to action; fewer pages protect the on-call rota.
• Protect the pipeline. Monitor the monitor from outside, because a broken telemetry stack fails silently.
• Defer fancy things. Skip per-customer SLOs and automated budget gating until the basic loop of measure, review and adjust has run for a quarter.

Common mistakes

• Alerting on every cause. CPU, memory and restart alerts page people for harmless events. Alert on SLO burn and user symptoms; keep cause signals for diagnosis.
• Using averages for latency. Averages hide the slow tail that users feel. Use histograms and p95 or p99, aggregated correctly.
• High-cardinality metric labels. User or request ids make metric storage explode. Put them in logs and traces instead.
• Unstructured logs without correlation ids. You cannot follow a request across services. Emit structured logs with trace ids on every line.
• SLOs set at 100 percent or invented without data. They are unattainable or meaningless. Base the target on measured behaviour and user tolerance.
• Measuring from the server's point of view only. A service can report itself healthy while the load balancer or client sees errors. Measure at the edge.
• No ownership or runbooks for alerts. Pages arrive with no instructions, and response is slow. Attach a runbook and route to a clear owner.

Key takeaways

• Observability answers unknown questions. Monitoring checks known failures; observability lets you investigate surprises.
• Use all three signals together. Metrics detect, traces locate, logs explain, linked by shared ids.
• Percentiles beat averages. Record latency in histograms and watch p95 and p99.
• RED for services, USE for resources. Rate, errors and duration show user experience; utilisation, saturation and errors explain it.
• An SLI is a measurement, an SLO is a target, an SLA is a contract. Keep the SLA looser than the SLO.
• Error budgets turn reliability into a decision. Budget remaining guides whether to ship or stabilise.
• Alert on symptoms. Page on user impact and budget burn; keep cause metrics for dashboards and tickets.
• Watch cost and cardinality. Telemetry volume, labels and retention need design, just like any data system.

Practice exercises

Exercise 1 — Starter (30 min): For a photo-sharing app, list three critical user journeys. For each, write one SLI as a ratio of good events to valid events, with a latency threshold where relevant, and propose an SLO with a time window. Compute the error budget in minutes of downtime or in number of failed requests (assume a request volume).

Exercise 2 — Intermediate (2 hours): Sketch a RED dashboard for an orders service and a USE dashboard for its database and connection pool. For each panel name the metric type (counter, gauge or histogram), the labels you would allow, and one label you would deliberately forbid because of cardinality. Then write three alert rules: one fast-burn page, one slow-burn ticket, and one leading-indicator alert, each with its runbook outline.

Exercise 3 — Advanced (half day): An incident review: users report slow checkout, the average latency graph looks normal, CPU is fine, and error logs are sparse. Using the signals in this lesson, write the investigation plan step by step: which metrics, which trace queries (including a sampling concern), which logs and what each result would tell you. Then propose three instrumentation or alert changes that would have caught the problem earlier, and estimate the telemetry cost impact of each using stated assumptions.

Related reading

• Requirements and non-functional goals — percentiles and the table of nines that SLOs build on.
• Observability, monitoring and logging in microservices — a practical article on the same topic in a service architecture.
• Implementing observability step by step — hands-on instrumentation walkthrough.
• Reliability basics — the failure handling that your SLOs measure.
• Storage engines — how time-series data is stored and why cardinality matters.