Ask a question Share knowledge Edit the page

Incident Analysis (Post Mortem)

Post Mortem (post mortem) It is a structured process and report that is generated after a major technical incident or failure, and its main purpose is to understand the causes of what happened, learn from them, and implement changes that will prevent similar situations from happening again in the future.

Key principles of effective post-mortem

  1. The Unaccused Culture (Blameless Culture)
    Most important, if engineers fear punishment or public reprimand for mistakes, they will hide details of failures, delay reactions, and distort chronology. Postmortem is based on the belief that people acted with good intentions based on the information they had at the time, and if there was a failure, it is the design of the system or process, not the individual performer, who is to blame.

  2. Search for systemic causes instead of superficial ones
    Instead of saying, "Developer made a typo in the configuration file," the team digs deeper: why did the system allow the file to be rolled out? Why didn’t the autotest or the canary depot work? Why did the monitoring not detect the problem until 20 minutes later?

  3. Focus on mental models (Mental Models)
    The analysis is to reconstruct the participants' thinking during the incident: What did they see on dashboards? What alerts worked? What documentation misled them? It helps to find and repair blind spots in interfaces and processes.

  4. Transparency
    The results of the analysis should not be stored in closed folders, but should be shared with the entire company or engineering league, so that other teams can learn from the experience and test their services for similar vulnerabilities.

Post Mortem Process

  1. Stabilization of the system: The first priority is to eliminate the impact of the failure on users, and the analysis begins only after the recovery of functionality.

  2. Fact-Collection and Chronology: The log of events is collected: when the first anomalies appeared, when the alert worked, what actions were taken by engineers when the system came back to normal.

  3. Completing a draft report: The responsible person (usually the engineer on duty or the owner of the affected service) fills out the report template.

  4. The following is the postmortem meeting: It’s the engineers involved, the allied teams and the stakeholders, and it’s a constructive chronology and reasoning process.

  5. Formation of Action Items: A list of concrete steps to improve the system is being developed.

Requirements for Action Items (Selectivity and Quality)

To prevent post mortem from becoming a formality, each Action Item developed must meet strict criteria:

  • Specific owner: The task is assigned to one specific person, not to the “development team”.

  • Clear deadline: The exact date of execution is indicated instead of vague wording like “next quarter”.

  • Measurable result: The readiness criterion should be easily verifiable (e.g., “add an alert to exceed the database connection limit in Slack” rather than “make the database more reliable”).

  • Mandatory entry in the Task tracker: Every improvement should be a ticket in Jira/Linear/ClickUp. Without a ticket, there is no task.

Advanced Level: Blameless 2.0 and the Reliability Committee

In mature IT organizations (for example, as is arranged at Google or large retailers like Cooper), a special format for sharing experiences is created. Reliability Committee (Reliability Committee).

Regular committee meetings publicly (in-house) examine 2-3 of the most significant incidents over a period, including the scale of the business damage, the chronology, the root causes, and the decisions made, and create a powerful horizontal knowledge exchange: engineers in neighboring departments see where they are, and proactively fix similar problems in their own before they fall.

Post Mortem template of document

  • Title: [Date.] [Service name] [Brief essence of the failure]

  • Status: [In progress / Completed]

  • Authors and participants: [Names]

  • Business effect: [How many users are affected, financial losses, reputational damage]

  • Timeline of events: [Time - Event]

  • Five Whys (5 Whys): [Method of searching for root cause]

  • Which worked well: [Where the team reacted quickly, which backups saved]

  • Where We Were Unlucky / What Went Wrong: [Blind areas, delays of alerts, non-working instructions]

  • List of Action Items: [Table: Task | Owner | Ticket link | Term]

Useful materials and references

Creative Commons License | by Igor Tsupko, Lana Novikova, Rodion Nagornov & community