What you'll learn: By the end of this lesson you will be able to decide when an event is an incident and how severe it is, assign the roles that keep a response coordinated, write a runbook a tired engineer can follow at 02:00, run a blameless postmortem, and write action items that have an owner and a verifiable end state, using ShopStream as the running example.
A hospital does not improvise when a patient collapses. Someone leads, others do specific tasks, a checklist covers the first minutes, and afterwards the team reviews what happened without hunting for the person to blame. Software teams that handle outages the same way recover faster and fail less often for the same reason twice.
Alerting and On-Call ends when a page reaches a human. This lesson starts there. Observability and SLOs and the ShopStream secure-operate stage mention runbooks and blameless postmortems in a sentence each; this lesson is the full treatment.
Documentation pages read on 2026-11-08: Google SRE Book and Workbook chapters on incidents and postmortems, and the PagerDuty Incident Response documentation. Where a practice is one company's own policy, the text says so.
The core idea
An incident process answers three questions in advance: who decides, what do we do first, and how do we learn. The incident commander decides, the runbook says what to do first, and the postmortem turns the cost of the outage into system changes. The earlier you settle these, the less a stressed team has to invent.
Three illustrations run through the lesson:
- Minimal example: the runbook skeleton in the runbook section.
- Realistic example: a ShopStream checkout incident at the evening peak.
- Failure example: a runbook step that no longer works.
Is this an incident, and how bad?
The SRE Book chapter "Managing Incidents" gives a simple test: if any of these is true, treat it as an incident: you need to involve a second team in fixing it; the outage is visible to customers; the issue is unsolved after an hour of concentrated analysis. The SRE Workbook adds a rule: declare incidents early and often, because managed incidents are resolved faster.
Figure 1. Declaring an incident. How to read this: any "yes" leads to declaration. Severity is chosen after, quickly, and corrected later.
Severity says how much response an incident gets. PagerDuty's published scale, as an example of one company's policy, runs from SEV-1 (a critical issue that warrants public notification and executive liaison) to SEV-5 (cosmetic, a ticket for the owner). Anything above SEV-3 is a "major incident" with a coordinator and a call. Its guidance for doubt is useful anywhere: if you cannot tell whether it is SEV-1 or SEV-2, treat it as the higher one, and argue about the label in the postmortem, not during the incident. Define your own levels by customer impact and required response, not by cause.
Roles: who does what
The SRE Book's incident roles come from the Incident Command System:
| Role | Responsibility | Does not |
|---|---|---|
| Incident commander (IC) | Holds the high-level state, assigns roles and tasks, removes roadblocks, keeps the incident document | Change the system; by default the commander holds every role they have not delegated |
| Operations lead | Applies operational tools to mitigate; the only group that modifies the system during the incident | Decide priorities alone |
| Communications lead | The public face of the response: periodic updates to stakeholders, keeps the document current | Debug |
| Planning lead | Longer-term items: file bugs, arrange handoffs, track divergence from normal | Run the call |
PagerDuty's guidance agrees and is blunt: the commander coordinates, delegates remediation to subject matter experts, and does not check graphs or read logs. It also adds a scribe, a deputy and time-boxed task assignments.
Figure 2. Roles and the shared document. How to read this: all coordination goes through the commander, and the document is the single record of what is known and what was tried.
Two practices keep this working. The incident document is the commander's most important artefact: editable by several people, it becomes the timeline for the postmortem. Handoff is explicit: the outgoing commander says "you are now the incident commander, okay?" and waits for a firm yes. The Workbook describes one long incident in which the engineers and commander rotated every four hours.
What the response is trying to do
The SRE Book's first priority is to stop the bleeding, restore service and preserve the evidence for root-causing. The Workbook is explicit that responders must prioritise mitigation above all else, because customers mostly want the errors to stop, and you need to know where the root cause is, not all of it, before you can act.
Figure 3. From detection to closed action items. How to read this: the loop on "Mitigating" is normal. The incident ends at user recovery; the learning ends only when the action items are done.
Mitigations are usually reversible and generic: roll back the last release, turn a feature flag off, shift traffic away from a failing region, shed load, scale out. Root-cause fixes come later.
Runbooks
A runbook is the documented first response to a specific alert or failure. Its reader is tired, rushed and may not own the service. The earlier lesson listed what it holds; here is a skeleton:
Runbook: CheckoutFastBurn
Owner: checkout team Last verified: 2026-10-30 (game day)
What this means: checkout errors exceed 1.44 percent (14.4x burn rate).
Who is hurt: shoppers trying to pay.
First checks (5 minutes):
1. Was there a deploy or flag change in the last hour? Link: <deploy log>
2. Which dependency is failing? Dashboard: <dependency errors>
3. Is the error confined to one region? Dashboard: <errors by region>
Mitigations (stop when errors drop):
- Roll back the last release: <exact command or pipeline link>
- Disable flag <name>: <exact command>
- Fail checkout reads over to the standby region: <link, approval needed>
Escalate to: <payments on-call> if the payments provider is failing.
After: open an incident if not resolved in 15 minutes.What makes a runbook work:
- One alert, one runbook, linked from the alert, with an owner.
- Exact commands and links, not "restart the service".
- Decision points, "if X, do Y, else Z", and a stop condition.
- A verified date. An untested runbook is a guess; verify during a drill.
- Automate what is rote. The SRE Book flags pages with rote, algorithmic responses as a red flag: a script can do those, and a runbook step that never changes is a script waiting to be written.
Postmortems
The SRE Book defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause and the follow-up actions to prevent recurrence. It lists triggers: user-visible downtime above a threshold, data loss of any kind, an on-call intervention such as a rollback or rerouting of traffic, a resolution time above a threshold, and a monitoring failure, which usually means a human found the problem.
The postmortem is blameless: it assumes everyone involved had good intentions and acted on the information they had. The SRE Book's line is that you cannot fix people, but you can fix systems and processes. Wording matters. "Engineer X ran the wrong command" invites defence; "the tool accepted a command with no confirmation" invites a fix. Removing blame, the Book says, gives people the confidence to escalate without fear.
The Workbook's example template gives the structure: owner and status, summary and impact, root causes and trigger, detection, resolution, a timeline, lessons learned (what went well, what went poorly, where we got lucky) and action items.
Action items are the point. The Workbook asks for a type, a priority, an owner and a tracking bug for each, and a verifiable end state: "improve monitoring" is vague, "alert when renewal fails for more than 10 minutes" can be checked. It notes that changing human behaviour is less reliable than changing automated systems, and recommends rewarding the closing of items, not just writing postmortems. The SRE Book adds that an unreviewed postmortem might as well never have existed, so review and publish.
PagerDuty publishes its own cadence, as one example: postmortems for SEV-1 and SEV-2 incidents, a meeting within 3 calendar days for SEV-1 and 5 business days for SEV-2, lasting 15 to 30 minutes, with tickets limited to what matters. Pick numbers that fit your size and write them down.
| Choice | Lighter process | Heavier process |
|---|---|---|
| Declaring | Only for outages customers see | Any need for a second team |
| Roles | Commander and one operator | Commander, operations, communications, planning, scribe |
| Postmortem | Short summary and actions for SEV-1 and SEV-2 | Full template for every incident and near miss |
| Gain | Little overhead | Consistent learning |
| Cost | Misses cheap lessons | Process fatigue |
The other side of this topic
A front-end team meets incidents as bad releases and broken client behaviour, where the fix is a rollback or flag the client honours, and where communication includes a status page shoppers see. The Frontend track has no published lesson on this yet.
In practice: a checkout incident at the evening peak
An educational scenario on ShopStream. All timings and figures are illustrative.
This is a realistic scenario, not a case history.
The situation
ShopStream handles about 58 checkouts a second at an assumed evening peak. A release adds a new pricing step to checkout. Twelve minutes later the page from the previous lesson, "Checkout is burning its error budget 14.4 times too fast", reaches the on-call engineer. Until now, incidents were handled by whoever was online, in a shared channel.
Starting architecture
Figure 4. The starting point. How to read this: several people change production at once, and customer support has no one to ask.
Options considered
| Option | Gain | Cost | Risk |
|---|---|---|---|
| A. Keep the channel, ask people to be careful | No change | None | Repeats the same confusion |
| B. Severity levels, a commander role, a live document, runbooks, blameless postmortems | Clear decisions, record for learning | Training, drills, writing | Process outliving its use |
| C. Adopt a dedicated incident platform | Less manual coordination | Cost, a new dependency | Tool without the habits |
The decision
Option B, in light form: three severity levels, a commander and an operations lead for any SEV-1 or SEV-2, a communications lead from SEV-2 up, a document template, and a postmortem for SEV-1 and SEV-2 and for any monitoring failure. Runbooks are linked from alerts and verified in a quarterly game day, with one of the previous incidents replayed as a "wheel of misfortune" exercise (the SRE Book describes reenacting a past postmortem with engineers playing the roles).
Figure 5. The shape of a managed response. How to read this: the commander never touches the system. Each task is assigned, time-boxed and reported back.
Rollout and verification
Train three engineers as commanders by shadowing. Run the game day with a seeded fault (a bad flag in staging) and time the page to declaration, declaration to mitigation, and the quality of updates. After each real incident, check whether the postmortem was written within the policy window and track the share of action items closed by their due date.
The 2 a.m. failure
A later, similar page. The runbook says "disable flag checkout.new_pricing". The operations lead runs the command and nothing changes: the flag was renamed during a refactor months ago and the old name is silently ignored. Ten minutes pass. The commander time-boxes the step, asks "what else could we do?" and orders the rollback, which works. The Workbook's advice to consider alternatives when something is not working is the rule applied.
The postmortem finds no one at fault. It finds that the runbook had no verified date, that the flag system accepted unknown names without an error, and that the refactor had no step to update runbooks. Actions, each with an owner and a verifiable end state: the flag tool rejects unknown flag names; a CI check fails when a runbook mentions a flag that does not exist; every runbook shows its last verified date and the game day refreshes it.
Architect's notes
- Decide the declaration criteria and severity table before the incident, and say who may declare.
- Separate deciding from doing. The commander does not type commands.
- Mitigate first. Roll back, flag off, shed load; investigate after.
- Make the postmortem a ticket system, not a document. Track items to closure and report the closure rate.
- Practise. Drills are where stale runbooks and unclear roles show up.
Common mistakes
- Everyone debugging at once, with nobody communicating.
- Waiting for certainty before declaring.
- Runbooks that say "investigate" instead of giving steps.
- Postmortems that name a person or end with "be more careful".
- Action items with no owner or no way to tell they are done.
- Writing a postmortem and never reviewing or sharing it.
Key takeaways
- Declare an incident when a second team is needed, customers see it, or an hour of focused work has failed; if unsure of severity, take the higher.
- The commander coordinates and delegates; operations is the only group changing the system; communications keeps stakeholders informed.
- Stop the bleeding first; root cause later.
- A good runbook has an owner, exact steps, decision points and a verified date; rote steps become automation.
- Blameless postmortems fix systems and processes; they assume good intent.
- An action item needs an owner and a verifiable end state, and the postmortem is done when the items are done.
Practice exercises
Exercise 1 (45 min). Write a three-level severity table for ShopStream by customer impact, with the response each level triggers and who may declare.
Exercise 2 (1 hour). Write a runbook for a "payments provider error ratio high" alert in the skeleton format. Mark every step that could be automated.
Exercise 3 (1 hour). Rewrite these in blameless form: "Priya deployed without testing"; "ops forgot to scale the cluster". Then turn each into an action item with an owner and a verifiable end state.
Exercise 4 (2 hours). Design a wheel-of-misfortune session using the 2 a.m. failure above: roles, the facts you reveal at each step, and what the group should decide.
Related reading
- Alerting and On-Call: the page that starts an incident.
- Observability and SLOs: the signals responders read.
- ShopStream Stage 10: secure and operate: ShopStream's alerts and runbooks.
- Resilience patterns: timeouts, breakers and fallbacks that reduce incident size.
- Multi-Region and Disaster Recovery: failing over when a region is lost.
Official documentation
- Google SRE Book, Managing Incidents: roles, command post, incident document, handoff.
- Google SRE Book, Postmortem Culture: Learning from Failure: triggers, blameless principle, reviews.
- Google SRE Workbook, Incident Response and Postmortem Culture: declaring early, template, action items.
- PagerDuty Incident Response documentation: severity levels, incident commander and postmortem process.