Opscraft
SRE & incident management · Lesson 4 of 7 · 3 min read

Postmortems and learning

Blameless postmortems

Assume people acted reasonably with the information they had; focus on systems, processes and conditions that allowed the failure. Blame leads to hidden mistakes and less learning.

Template:

  1. Summary — what happened, impact (users, duration, revenue, SLO budget consumed), severity.
  2. Timeline — detection, key decisions, mitigation, resolution (UTC/IST consistently).
  3. Root cause and contributing factors — use "5 whys" or causal analysis; usually several factors (a bad config + missing validation + alert gap + slow rollback).
  4. What went well / what went poorly / where we got lucky.
  5. Action items — specific, owned, prioritised, tracked to completion (prevent, detect faster, mitigate faster). E.g. "Add schema validation for feature flag config in CI (owner, due date)".
  6. Lessons shared broadly — postmortem reviews, a searchable library.

Avoid action items like "be more careful" — they're not actions. Prefer mechanisms: automation, guardrails, tests, alerts.

Enlarge diagramSummaryimpact, duration, budget usedTimelinedetection to resolutionCausesseveral factors, not one personWent well / poorlyand where we got luckyActionsowned, dated, trackedSharereview, searchable library123456
Figure 1. The parts of a blameless postmortem.
Swipe sideways to see all of it, or tap to open it full screen.

Example root-cause chain (5 whys)

Checkout failed for 25 minutes → why? pods couldn't connect to the database → why? connection pool exhausted → why? a new release opened connections per request without pooling → why? the change wasn't load-tested and the canary had too little traffic to show it → why? canary analysis only checked error rate, not DB connection saturation. Actions: pool configuration review, connection-saturation metrics in canary analysis, load test stage for DB-touching changes, RDS Proxy.

Enlarge diagramCheckout failed for 25 minutesPods could not connect to the databaseConnection pool exhaustedNew release opened a connection per requestNot load-tested; canary too small to show itCanary checked errors, not DB connectionsroot cause → fix the analysis
Figure 2. The 5 whys from this lesson: keep asking why until you reach something you can fix with a mechanism.
Swipe sideways to see all of it, or tap to open it full screen.