sitereliabilityengineering/concepts devops/observability
Core Idea
Incident management is a fixed sequence for outages: detect, respond, mitigate, resolve, then run a blameless postmortem. Mitigation restores service first and asks why later.
- The lifecycle runs detection (alerts or user reports, tied to recent changes) → response (notify on-call, isolate the blast radius) → mitigation → resolution → postmortem, triaged by SEV-1 to SEV-4 with runbooks and on-call rotations.
- Mitigation deliberately ignores root cause: restart pods, roll back, or switch traffic to a healthy version (blue/green) so the service is back before the investigation starts.
- The postmortem is blameless because failures are systemic issues, not personal mistakes; removing blame is what gets people to report honestly.
Incident management can be defined as the structured approach to addressing and managing the aftermath of outages or service interruptions.
Goal - Restore normal service operation as swiftly as possible and minimize the adverse impact on business operations.
Key Concepts
| Concept | Description |
|---|---|
| Incident | Any unplanned interruption to a service or reduction in its quality. |
| Severity (SEV) Levels | A classification (e.g., SEV-1 to SEV-4) that reflects the impact of an incident. |
| Runbook | A predefined guide or procedure for handling specific incidents. |
| On-call rotation | Designated team members scheduled to handle incidents during off-hours. |
| Postmortem | A retrospective analysis of the incident: what happened, why, and how to improve. |
Incident Lifecycle
Scenario: Outage in a food delivery app, Customers can’t lace new orders. API request to /place-order are timing out.
Detection
- Look for problem using tools or user reports.
- Try to connect the issue to any recent changes in the app or system(like a new release or update)
- This helps you guess what might have caused it.
Monitoring tool (like Datadog) detects a high error rate on /place-order. Alert trigger and alert categorized as SEV-1 (coz it create critical business impact)
Response
- Let the right people know and start working on the issue.
- Try to isolate the problem —> find out which parts of the system are affected so the rest can keep running smoothly.
On call engineer gets notified and acknowledges the incident. Checks the logs and dashboard (Grafana, Prometheus) to indetify the problem.
After identifying the problem it communicate with responsible team to fix the issue
Mitigation
- The goal here is to bring the system back online quickly, not to fix the root cause.
- Use Backups or switch to a healthy version (like with blue/green deployments) to keep service running.
Restarts a few application pods
Blue/Green Deployment
When the
/place-orderAPI breaks due to a bad deployment or DB overload, Blue/Green deployment lets you quickly switch traffic from the broken version (Blue) to a working version (Green) without downtime.
Steps:
- Keep a healthy Green version ready (with fixes or old stable code).
- Reroute user traffic to Green using a load balancer or feature flag.
- Monitor the new version to ensure it’s stable.
- Fix or rollback the Blue version in the background.
Resolution
- Once things are working again, focus on finding the real cause and fixing it for good.
- Don’t worry about writing postmortems right away — first, make sure everything is stable.
System is back to normal.
Postmortem
Postmortem Culture - Learning from Failure
Postmortem is a structured review of an incident after it’s resolved. The goal is to understand what went wrong, why it happened and how to prevent it from happening again. Without blaming anyone.
Blameless Culture
In SRE, a blameless culture focuses on learning from incidents rather than blaming individuals. It sees failures as results of systemic issues, not personal mistakes. This approach encourages openness, honest discussion, and continuous improvement. By removing fear of blame, teams feel safer to report issues, take responsibility, and innovate — leading to more effective postmortems and a stronger reliability culture.