sitereliabilityengineering sitereliabilityengineering/book sitereliabilityengineering/sre-book sitereliabilityengineering/concepts
Core idea
A postmortem is a learning and reliability-improvement mechanism, not an exercise in assigning blame. An incident is an opportunity to understand the conditions that made failure possible, reduce the chance or impact of recurrence, and share the lessons across teams.
Why postmortems matter
Complex systems will fail. Restoring the service solves the immediate problem; a postmortem addresses why the system was vulnerable and what must change.
The goals are to:
- Preserve an accurate account of the incident and its user impact.
- Understand triggers, contributing conditions, and underlying system weaknesses.
- Convert lessons into owned, prioritized, verifiable actions.
- Share knowledge so other teams avoid the same failure modes.
- Improve System Resilience and Incident Response over time.
A postmortem is not the finish line
The value comes from actions completed and lessons reused—not from merely publishing a document.
When to write one
Define objective triggers before incidents occur. Google’s examples include:
- User-visible outage or degradation beyond an agreed threshold.
- Any data loss.
- An incident requiring on-call intervention (e.g., rollback or traffic rerouting).
- Recovery taking longer than an agreed threshold.
- Failure of monitoring, especially when people discover the incident before alerts do.
A stakeholder can also request a postmortem even when a threshold is not crossed. Thresholds should fit the service’s SLOs and risk profile.
The blameless mindset
Blameless does not mean detail-free or accountability-free. Record the actions taken, but investigate why those actions made sense with the information, tools, constraints, and incentives available at the time.
| Avoid | Prefer |
|---|---|
| “The operator carelessly retried the job.” | “The retry path did not enforce idempotency, and the interface did not make the risk visible.” |
| “The team failed to notice the outage.” | “The alerting coverage did not detect the failure until user impact was already occurring.” |
| “People need to be more careful.” | “Add a preflight validation, scoped permissions, and a safe rollback path.” |
Ask: What did responders see? What did they believe would happen? Which safeguards, signals, or processes were missing? Why did the system permit a single action to have such a large effect?
The purpose is to make the safe path easier and the unsafe path harder.
Anatomy of a useful postmortem
| Section | Questions it should answer |
|---|---|
| Metadata | Incident ID, date, service, severity, author/owner, status, reviewers |
| Executive summary | What happened, for how long, and how was service restored? |
| User/business impact | Who was affected? How many requests or users? What about latency, data, revenue, and Error Budget? |
| Detection | What first identified the problem? Did monitoring work? |
| Trigger and contributing causes | What initiated the incident, and what weaknesses allowed it to escalate? |
| Response and resolution | What mitigations were attempted? Which worked or failed? |
| UTC timeline | What happened, when, and in what order? |
| Lessons learned | What went well, what went poorly, and where were there fortunate near misses? |
| Action items | What changes will reduce likelihood, impact, or detection/recovery time? |
| Evidence | Links to graphs, logs, tickets, dashboards, and incident communications |
Distinguish trigger from causes
A traffic spike may trigger an outage, while a resource leak, lack of load shedding, weak capacity isolation, and missing safeguards contribute to its severity. Stopping at the trigger is not deep enough.
What makes action items effective?
An actionable item should have:
- Specific change: say what will be built, changed, removed, or tested.
- Category: prevent, mitigate, detect, investigate, or improve process.
- Single accountable owner: collaborators can be many; accountability must be clear.
- Priority: based on risk and the potential effect of recurrence.
- Tracking reference: ticket/issue URL or identifier.
- Acceptance criteria: a verifiable definition of done.
- Target date (recommended): helps ensure follow-through; adapt to your team’s workflow.
| Weak action | Stronger action |
|---|---|
| “Improve alerts.” | “Alert when the error rate breaches the agreed threshold for five minutes; validate the page path in a test.” |
| “Document the retry hazard.” | “Reject empty-selection destructive operations and add an automated regression test.” |
| “Train everyone on cascading failures.” | “Add a load-shedding safeguard and rehearse the recovery playbook in a game day.” |
Watch for action-item theater
A long list of vague tasks is not progress. Prefer risk-reducing fixes that have owners, priorities, evidence of completion, and follow-up review. Avoid overfitting expensive fixes to an unusual incident.
Lessons from Google’s examples
Example 1 — Shakespeare Sonnet++ search outage
Source: SRE Book — Example Postmortem (illustrative postmortem).
- Impact: 66 minutes of downtime; estimated 1.21 billion lost queries.
- Trigger: news of a newly discovered sonnet caused a massive traffic spike; searches for a term absent from the indexed corpus were especially problematic.
- Contributing causes: an existing file-descriptor/resource leak on failed searches amplified load into a cascading failure.
- Detection: monitoring paged on high HTTP 500 rates.
- Mitigation: traffic was isolated to a sacrificial cluster, capacity was increased to roughly 10Ă— the original level, and an updated index was rolled out.
- Follow-up themes: repair the leak, add load shedding and regression tests, improve the cascading-failure playbook, and account for Error Budget exhaustion.
Takeaway: Separate the external trigger from the internal failure mechanisms. Robustness depends on graceful failure under exceptional demand—not only on predicting the next traffic spike.
Example 2 — Satellite decommissioning incident
Source: SRE Workbook — Postmortem Culture (real case study).
- A maintenance operation was retried after part of the workflow had already run.
- A software/API defect interpreted an empty selection as no filter, allowing a destructive operation to affect satellite machines globally.
- Production traffic was rerouted to core datacenters; users experienced increased latency while affected edge machines were restored.
- Corrective work added safety checks and improved decommissioning automation. When a related incident occurred years later, prior fixes reduced its blast radius.
Takeaway: The useful question is not who retried the operation. It is why the system allowed an unscoped destructive retry. Strong fixes include fail-closed validation, idempotency, bounded blast radius, and emergency-stop mechanisms.
Practical postmortem workflow
- Stabilize: restore the service and hand over from response to learning.
- Capture evidence: preserve incident logs, dashboards, decisions, and a timestamped timeline.
- Draft promptly: include responders and impacted service owners while details are fresh.
- Quantify impact: distinguish measured values from estimates and unknowns.
- Analyze contributing factors: examine system design, observability, process, and incident response—not just the final failure.
- Define changes: assign owners, priorities, acceptance criteria, and tracking tickets.
- Review collaboratively: check technical depth, blameless wording, and feasibility of actions.
- Share widely (safely): make lessons discoverable without exposing user-sensitive data.
- Track to closure: periodically review outstanding actions and the risk they leave open.
- Revisit recurring incidents: look for cross-team/systemic themes and verify whether previous fixes worked.
Healthy postmortem culture
- Leadership models curiosity, not public blame.
- Engineers are recognized for raising issues, writing useful analyses, and closing actions.
- Reviews invite participants from multiple teams; knowledge travels beyond the incident owners.
- Postmortems are stored in a searchable repository with consistent metadata.
- Teams track useful trends: time to detect, time to recover, user impact, blast radius, repeat incidents, and action-item completion.
- Tools can prepopulate timelines, incident metadata, and links, reducing documentation toil.
Warning signs: fear of being named in a postmortem; vague or unowned fixes; late, unread, or unreviewed documents; recurring outages with the same failure mode; feature delivery continually displacing reliability work.
Reusable incident postmortem skeleton
Suggested file name:
INC-YYYY-NNN — Short incident title.md
Incident metadata
- Incident ID:
- Service / severity:
- Start / mitigation / end (UTC):
- Postmortem owner / contributors:
- Status: Draft / In review / Final / Actions in progress / Closed
- Related incident:
[[...]]
Summary
What happened, the main consequence, and how it was resolved in 3–5 sentences.
Impact and detection
- User-visible impact:
- Scope / duration / affected requests:
- SLO or error-budget impact:
- Detection source and delay:
- Uncertainty / missing data:
Trigger, contributing factors, and resolution
- Trigger:
- Contributing technical/systemic factors:
- How the failure propagated:
- What stopped the impact:
Timeline (UTC)
| Time | Event / observation / decision | Evidence |
|---|---|---|
| 00:00 |
Learning
- What went well:
- What went poorly:
- Where we got lucky / near misses:
- What would make a future response easier:
Action items
| Action / acceptance criteria | Type | Priority | Owner | Target | Ticket | Status |
|---|---|---|---|---|---|---|
| Prevent / Mitigate / Detect / Process | Open |
Review and distribution
- Timeline and impact verified
- Contributing causes reviewed with participants
- Wording checked for blame
- Actions approved, owned, and tracked
- Relevant stakeholders notified
- Lessons indexed and linked to related incidents
Incident Management provides the operational lifecycle that leads into these blameless reviews, while AI-Powered Root Cause Analysis at Scale shows how automated investigation can support the learning loop.
Sources
- Google SRE Book — Postmortem Culture: Learning from Failure — philosophy, triggers, blamelessness, collaboration, and reviews.
- Google SRE Book — Example Postmortem — illustrative incident structure, timeline, and follow-up actions.
- Google SRE Workbook — Postmortem Culture: Learning from Failure — real case study, contrast between weak and strong postmortems, organizational practices, tooling, and follow-through.