What you'll learn: By the end of this lesson you will be able to explain why a plain error-rate threshold makes a poor alert, compare the six alerting strategies in the Google SRE Workbook, work out how fast a multiwindow burn-rate rule notices a given error ratio, route, group and inhibit alerts with Prometheus Alertmanager, write a paging policy that separates "wake someone" from "ticket" from "dashboard", and measure whether an on-call rota is healthy.
A smoke detector that goes off every time you make toast is soon unplugged. One that stays silent until the kitchen is on fire is useless. A good detector fires when there is a real fire, early enough to act, and stays quiet otherwise. Alerting is the same trade, with the added cost that the "detector" wakes a person.
Observability and SLOs defined SLIs, SLOs, error budgets and a first burn-rate table, and the ShopStream secure-operate stage shows one fast-burn rule. This lesson explains why those rules look as they do, how an alert travels to a human, and how to keep that human healthy.
Documentation pages read on 2026-11-08: Google SRE Book and Workbook, Prometheus (Alertmanager, alerting rules, alerting practices), PagerDuty Incident Response documentation.
The core idea
An alert is a request for a human to act, so every alert must be urgent, actionable and about something users feel. Anything else belongs on a ticket queue or a dashboard. The Prometheus alerting practices page puts the philosophy in one line: keep alerting simple, alert on symptoms, and have good consoles to pinpoint causes.
Three illustrations run through the lesson:
- Minimal example: one error-ratio alert, tested against the SRE Book's questions.
- Realistic example: ShopStream's checkout, with burn-rate rules and an Alertmanager routing tree.
- Failure example: an inhibition rule that silently suppresses the page that mattered.
What makes an alert good
The SRE Book chapter "Monitoring Distributed Systems" lists questions to ask before creating a page: will I ever be able to ignore this alert, knowing it is benign? Can I take action in response? Are other people paged for the same issue, making one page unnecessary? It adds that each page should be met with a sense of urgency, and that pages with rote, algorithmic responses should be a red flag, because a machine can follow a fixed recipe.
Figure 1. Three questions between a signal and a page. How to read this: start at the top for each candidate alert. Most signals leave on the left or the first "no" branch; the page is the narrow exit at the bottom.
PagerDuty's alerting principles say the same in operational terms: an alert requires a human to perform an action, and anything that wakes someone at night should be immediately actionable. Context-only events, such as a finished deployment, are notifications that page nobody.
The SRE chapter's "what versus why" distinction decides what to measure. A symptom says what is broken ("checkout fails for 1.5 percent of shoppers"); a cause says why ("a database replica is lagging"). Page on symptoms, which track user pain whatever the cause, and keep causes for dashboards. The exception is a cause with long lead time and near-certain impact, such as a certificate expiring in days.
Why a simple threshold fails: the six strategies
The SRE Workbook chapter "Alerting on SLOs" builds the argument step by step. It scores each approach on four properties:
- Precision: the share of alerts that correspond to a significant event.
- Recall: the share of significant events that raise an alert.
- Detection time: how long before an alert fires once the problem starts.
- Reset time: how long the alert keeps firing after the problem ends.
| # | Strategy | What goes wrong (from the Workbook) |
|---|---|---|
| 1 | Error ratio above the SLO threshold, short window | Low precision: fires on events that do not threaten the SLO (up to 144 alerts a day in the chapter's example) |
| 2 | Same, longer window | Better precision, but a long reset time (36 hours after a total outage in the example) |
| 3 | Add a duration (for:) |
Poor recall and slow detection: a brief return to normal resets the timer |
| 4 | Alert on burn rate | Good precision, low recall: a 35x burn rate never alerts, yet spends the 30-day budget in 20.5 hours |
| 5 | Several burn-rate alerts | Good recall thanks to the long window, but a longer reset time |
| 6 | Multiwindow, multi-burn-rate | Recommended: good precision and recall, fast detection of fast burns, and a short window that ends the alert soon after errors stop |
Strategy 6 is the earlier lesson's table: the long window decides whether the burn matters, the short window confirms it is still happening, and both must exceed the threshold. The Workbook suggests a short window of one twelfth of the long one.
Figure 2. How a Prometheus alerting rule moves between states. How to read this: without a for clause a rule becomes active on its first evaluation. for adds the pending state, which suppresses short blips; keep_firing_for stops an alert flapping when the condition wobbles around its threshold.
Both clauses are defined in the Prometheus documentation. They are blunt tools: burn-rate windows scale with the size of the problem, so a total outage trips the fast rule almost at once, while for: 10m waits ten minutes whatever the damage.
Detection time, worked through
Take ShopStream's checkout SLO of 99.9 percent over 30 days (illustrative), so the error budget is 0.1 percent. The table assumes a constant error ratio and constant traffic. A burn rate is the error ratio divided by 0.001, and the time to fire is the window length times the threshold divided by the error ratio.
| Constant error ratio | Burn rate | Rule that fires first | Time to fire | Budget spent by then |
|---|---|---|---|---|
| 100% (total outage) | 1,000 | 14.4 over 1 h and 5 min | about 52 seconds | 2% |
| 5% | 50 | 14.4 over 1 h and 5 min | about 17 minutes | 2% |
| 1% | 10 | 6 over 6 h and 30 min | 3.6 hours | 5% |
| 0.3% | 3 | 1 over 3 d and 6 h (ticket) | 24 hours | 10% |
Check the second row: the 1-hour average reaches 1.44 percent after 1.44 / 5 of an hour, 17.3 minutes. Each rule spends a fixed share of the budget before it fires, the 2, 5 and 10 percent of the Workbook table: you choose how much damage to tolerate before a human is told, and the thresholds follow.
From rule to human: Alertmanager
Prometheus evaluates rules and sends the firing alerts to Alertmanager. According to the Prometheus documentation, Alertmanager deduplicates, groups and routes those alerts to receivers such as email, PagerDuty or OpsGenie, and also handles silencing and inhibition. Four ideas matter.
| Idea | Definition (Prometheus docs) | Use it for |
|---|---|---|
| Grouping | Categorises alerts of similar nature into a single notification | One message when forty pods fail, not forty pages |
| Inhibition | Suppresses notifications for some alerts if certain other alerts are already firing | Muting downstream symptoms during a root-cause alert |
| Silences | Mute alerts for a given time, matched by labels | Planned maintenance |
| Routing tree | Sets grouping, timing and receiver per alert | severity=page to the pager, the rest to a queue |
Grouping is tuned with three timers. The configuration reference gives the defaults: group_wait is 30 seconds (how long to wait before the first notification for a new group), group_interval is 5 minutes (before notifying about changes to an existing group) and repeat_interval is 4 hours (before repeating an unchanged notification).
Figure 3. One notification instead of many. How to read this: the wait before the first message gives related alerts time to arrive and share it. A group that keeps changing produces at most one update per group_interval.
A routing tree for ShopStream
# alertmanager.yml (illustrative; integration settings omitted)
route:
receiver: ticket-queue
group_by: [alertname, service]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers:
- severity="page"
receiver: primary-pager
repeat_interval: 1h
inhibit_rules:
- source_matchers:
- alertname="PaymentsProviderDown"
target_matchers:
- dependency="payments-provider"
equal: [region]
receivers:
- name: primary-pager
- name: ticket-queueAlerts that match no child route stay with the root receiver, the ticket queue; severity="page" goes to the pager. The inhibition rule mutes alerts carrying dependency="payments-provider" while PaymentsProviderDown is firing, when both have the same region. The equal list is where inhibition goes wrong (see the 2 a.m. failure).
A paging policy
PagerDuty's alerting principles describe four classes, adapted here into a table everybody can read at a glance:
| Class | When it fires | Who and when | Example for ShopStream |
|---|---|---|---|
| Page, high urgency | A symptom is burning budget fast | Primary on-call, any hour, response measured in minutes | Checkout fast-burn rule |
| Page, business hours | A slower burn or a leading indicator | Action within about a day, not at night | Slow-burn rule; certificate expiring in days |
| Ticket | Needs work, no deadline today | Team queue | The 1x burn rule over 3 days |
| Notification | Context only | Nobody paged | Deployment finished |
Every page carries a specific title, the triggering metric and threshold, why it matters, a runbook link or first steps, and a dashboard link. PagerDuty states that an alert with neither steps nor a runbook is useless, and that an untested alert is equivalent to no alert, including the "no data" case: receiving no data is the same as breaking your threshold.
Sizing the rota
The SRE Book chapter "Being On-Call" gives figures from its authors' teams: treat them as examples, not rules. No more than 25 percent of time on call; at least eight engineers for a single-site rota (six per site for dual-site); at most two incidents per 12-hour shift, because one takes about six hours including follow-up; and example overload measures of fewer than 5 tickets a day and fewer than 2 paging events per shift.
For your own rota track pages per shift, the share of pages that needed action (page precision), time to acknowledge and time to detect. A rota that fails these has an alerting problem before it has a staffing problem.
Low-traffic services
Burn-rate maths assumes enough requests that one failure does not dominate. A ShopStream store-import endpoint with 10 requests an hour (illustrative) and a 99.9 percent SLO allows 7.2 failures in 30 days (7,200 requests times 0.001), and one failure in an hour is a 10 percent error ratio, a burn rate of 100. The Workbook's remedies: synthetic traffic (which can hide real errors), combining related services that share a failure domain, making single failures matter less (retries, fallbacks), or lowering the SLO or lengthening the window, a product decision.
The other side of this topic
The browser has the same problem in a different shape: client-side error rates and real-user timings are noisier than server metrics, yet a front-end alert still needs a symptom-first rule and an owner. The Frontend track has no published lesson on it yet.
In practice: ShopStream pages that mean something
This is the realistic example, an educational scenario on the course's running example. Every figure is illustrative.
This is a realistic scenario, not a case history.
The situation
ShopStream handles about 500,000 orders a day: about 5.8 per second on average, about 58 at an assumed peak. A four-person rota covers the platform. In a payments-provider incident, Checkout, Orders, Notifications and the workers produced 38 pages in ten minutes, each saying something upstream was failing. The on-call engineer could not tell the one real problem from the echoes.
Starting architecture
Figure 4. Starting point: every rule pages, nothing groups. How to read this: all rules route to the same receiver with the same urgency, so a single upstream failure reaches the human as dozens of independent pages.
Options considered
| Option | Gain | Cost | Risk |
|---|---|---|---|
A. Raise thresholds, add for: 15m |
Quieter | Extra delay for every real outage | Slow detection; slow burns missed |
| B. Burn-rate rules, grouping, tiered routing, one inhibition rule | Few pages, tied to user pain | Rule rewrite, paging policy, label discipline | A wrong inhibition rule hides real pages |
| C. Buy a correlation product | Less rule work | Cost, new dependency | Opaque when wrong |
The decision
Option B. Checkout gets the 14.4 and 6 page rules and the 1x ticket rule; cause alerts move to dashboards or ticket severity; routing follows the tree above; each page links a runbook (the next lesson). At the 14.4 threshold the error ratio is 1.44 percent: about 5 failed checkouts a minute at the average 5.8 a second, about 50 a minute at the assumed 58 a second peak.
Figure 5. The routing decision, in order. How to read this: grouping and inhibition come first, then severity chooses the receiver. A mistake in the inhibition step affects everything after it.
Rollout and verification
Run the new rules beside the old ones for two weeks, delivering to chat only. Replay the provider incident from stored metrics and count the notifications the tree would produce (the goal is one group and one page, not 38). Fire a synthetic failure in staging and time the page to a phone. Track pages per shift and page precision, and review every page weekly, deleting or downgrading those that needed no action.
The 2 a.m. failure
Customers cannot check out in one region and nobody is paged. "PaymentsProviderDown" has been firing for a separate, minor reason, and the checkout alerts also carry dependency="payments-provider". The inhibition rule's equal list names region, but these alerts have no region label. The Alertmanager configuration reference says a missing label and an empty-valued label are the same thing, so when all equal labels are missing from both alerts the rule applies. The minor alert therefore mutes the checkout page, and a customer support ticket is the first sign.
Prometheus still shows the burn-rate rule firing: its ALERTS series carries the firing state, which is how the gap between "firing" and "notified" is found. Mitigation: remove the inhibition rule and page the owner by hand. Lasting fixes: put region on every alert, narrow target_matchers so they cannot match page-severity alerts, and run a test that sends a fake page through the tree after every configuration change. An always-firing heartbeat alert that must reach an external receiver, as in the earlier lesson, catches silence in the pipeline itself.
Architect's notes
- Start from the SLO, not the metric list. One symptom rule per user journey; add causes only for long-lead-time risks.
- Write the paging policy before the rules. The table decides which rule gets which severity.
- Treat Alertmanager configuration as code, and prove that a fake page reaches a phone after each change.
- Give every alert an owner. An alert that needed no action in a quarter is a candidate for deletion.
Common mistakes
- Paging on every cause (CPU, restarts, queue length).
- Relying on a static threshold or only
for:, and getting noise or missed slow burns. - Paging for things nobody can act on, or whose response is a fixed script.
- Writing broad inhibition rules with labels missing from
equal.
Key takeaways
- A page is a request for a human to act now; otherwise it is a ticket, a dashboard item or automation.
- A plain error-rate threshold has low precision; a long window or
for:trades that for poor recall and slow reset. - Multiwindow, multi-burn-rate alerts (14.4 and 6 page, 1 tickets, short window one twelfth of the long one) give good precision and recall.
- Alertmanager groups, routes, silences and inhibits; its timers default to 30 seconds, 5 minutes and 4 hours.
- A missing label in
equalmakes an inhibition rule apply too widely. - Measure the rota: pages per shift and page precision.
Practice exercises
Exercise 1 (30 min). For a 99.5 percent checkout SLO (illustrative), compute the error ratio at which burn rates 14.4, 6 and 1 fire, and the failed requests per minute at 58 requests a second.
Exercise 2 (45 min). Run ten alerts from a system you know through Figure 1 and record page, ticket, dashboard or automate. Count the pages left.
Exercise 4 (1 hour). Write an Alertmanager tree for three teams routed by a team label, plus an inhibition rule that mutes dependent alerts when a database alert fires. List the labels every alert must carry for it to be safe.
Related reading
- Observability and SLOs: SLIs, SLOs, error budgets and the first burn-rate table.
- ShopStream Stage 10: secure and operate: the ShopStream SLO and fast-burn rule.
- Resilience patterns: what keeps one failure from becoming forty alerts.
- Reliability basics: availability targets and error budgets in context.
- Runbooks, Incidents and Postmortems: what happens after the page.
Official documentation
- Google SRE Workbook, Alerting on SLOs: the six strategies, window and burn-rate table, low-traffic services.
- Google SRE Book, Monitoring Distributed Systems: symptoms versus causes and the questions for a page.
- Google SRE Book, Being On-Call: quantity and quality balance, overload.
- Prometheus, Alertmanager overview and configuration.
- Prometheus, Alerting rules and alerting practices.
- PagerDuty Incident Response, Alerting principles.