SRE & incident management · Lesson 3 of 7 · 4 min read
Incident management
Severity levels
Define them in advance, with examples, so nobody debates at 3 a.m.:
| Sev | Definition | Response |
|---|---|---|
| SEV1 | Critical customer impact: outage of a core flow, data loss, security breach | All hands, incident commander, status page, exec updates every 30 min |
| SEV2 | Significant degradation or partial outage | On-call + owners, IC, regular updates |
| SEV3 | Minor impact, workaround exists | Business hours |
| SEV4 | No customer impact | Ticket |
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Roles (ICS-style)
- Incident Commander (IC) — coordinates, decides, delegates; does not debug hands-on in big incidents.
- Operations / tech lead(s) — investigate and mitigate.
- Communications lead — status page, stakeholders, customer support.
- Scribe — timeline of actions, decisions and timestamps.
- Subject matter experts paged as needed.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
The incident lifecycle
- Detect — alert, customer report, synthetic check.
- Triage — confirm impact and severity; declare the incident early (it's cheap to downgrade).
- Mobilise — incident channel (
#inc-2026-10-27-checkout), bridge call, assign roles. - Mitigate first, diagnose later — restore service: roll back the last deploy, fail over, scale up, disable a feature flag, shed load, block bad traffic. Root cause can wait.
- Communicate — regular updates at a set cadence (what's affected, what we're doing, next update time); internal and external.
- Resolve — confirm with metrics/SLIs that users are fine; monitor for recurrence.
- Postmortem — within days, blameless, with tracked actions.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
"What changed?" is the most productive first question: deploys, config/feature-flag changes, infrastructure changes, traffic shifts, dependency incidents, certificate expiry, quota limits. Deployment annotations on dashboards make this fast.
Metrics for incident response
- MTTD (time to detect), MTTA (to acknowledge), MTTM (to mitigate), MTTR (to restore/resolve), MTBF (between failures).
- Use them as trends for improvement, not as targets that encourage gaming (e.g. closing incidents early).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.