Postmortem Root Cause Analysis: Why One Root Cause Means the Incident Comes Back

Here's a quick exercise. On the morning of August 1, 2012, all five of these were true at a US trading firm called Knight Capital. By the end of roughly 45 minutes, the firm had lost more than $460 million.

Which one was the root cause?

Every answer is defensible. Every answer, fixed alone, leaves four conditions loaded for the next bad morning. That's the problem with the Root Cause field in your postmortem template, and it's why "fixed" incidents keep coming back.

TL;DR: Root cause analysis doesn't find where an incident started. It finds where the investigation stopped. Incidents need several conditions at once, so a single cause yields a single fix and leaves the rest armed. Test every action item by replaying the incident with only that fix in place, and spread fixes across four layers: less likely, known sooner, smaller, back faster.

Why a single root cause produces repeat incidents

Richard Cook's 1998 paper How Complex Systems Fail makes two points that explain most repeat incidents:

1. Serious failures require several smaller failures to line up. Systems are full of defenses, so one fault alone is rarely enough.
2. So attributing an accident to one root cause is a mistake. Cook argues the pull toward a single cause is social: people want something to point at.

Now trace what the template does with that:

• One text box → one cause
• One cause → one action item
• One action item → one condition removed
• The incident needed five → four remain, waiting for a different trigger

That's the mechanism behind "didn't we already fix this?" You did. You fixed one of them.

Root cause analysis is a stopping rule

Postmortems don't stop when they reach the truth. They stop somewhere comfortable. In practice there are three stop points:

1. The Person: "human error." Produces training, reminders and "be more careful." None of these change the environment, so the next engineer faces the same setup. Ask instead: How did this action make sense with what they could see at the time?

2. The Line: "bug fixed in PR 4812." Feels rigorous because the code really was wrong. But the line is the trigger. Ask instead: What let this line reach production, and what let it run so long before anyone noticed?

3. The Wall: "vendor outage," "network blip." Analysis stops where ownership ends. Ask instead: Where did our design assume the other side would never fail? (Timeouts, retries and fallbacks are yours.)

Why the 5 Whys makes it worse

The Five Whys follows one chain of "why?" until something feels fundamental. Alan Card's 2017 paper in BMJ Quality & Safety summarizes the problems:

• It walks a single causal path, while incidents have many.
• Different people asking the same whys reach different root causes.
• The stopping point is arbitrary; five is just a number.

John Allspaw's 2014 essay The Infinite Hows makes the software version of the argument: "how" questions open up the situation; "why" questions tend to hunt for a culprit. Add hindsight bias (also one of Cook's points: knowing the outcome makes warning signs look obvious), and "why" chains drift toward The Person.

Case study: how AWS handled "human error" in the 2017 S3 outage

On February 28, 2017, Amazon S3 in US-East-1 went down for hours. Per AWS's public summary, the trigger was an engineer running an established playbook who entered one input incorrectly, removing far more servers than intended.

A textbook Person stop. AWS didn't take it. Their summary attributes the problem to a tool that allowed too much capacity to be removed too quickly, and the published fixes map onto four different layers:

The working question was "how did one mistyped value become a regional outage?" not "why did the engineer mistype it?"

The Replay Test: how to evaluate postmortem action items

For each action item, replay the incident with only that fix in place and note where it stops:

Example: an incident where someone ran a cleanup script against production instead of staging (illustrative, not a real case). Action item: "add a confirmation prompt." Replay it. The incident stops at the final keystroke, for one script. The shared config holding both connection strings, the lack of a row limit, the slow alert and the four-hour restore are all untouched. The next tool with production credentials has no prompt.

If all your action items stop in the same place or live on one layer, you have a gap.

How to prioritize when you can't fix everything

• Prefer shared conditions. Knight's missing deploy check sat under every deploy, not just this one. A fix there covers incidents you haven't had yet.
• Delete "be more careful." It fails the Replay Test by definition. Training can be an addition, never the only fix.
• Owner and date, or drop it. Three owned items beat twelve unowned ones.
• Look for near misses. Cook notes that complex systems usually run partly degraded and that reviews often find earlier near misses. A condition that showed up before is a shared condition.

Trade-off: conditions-first postmortems take longer and produce fewer, harder action items. Some need roadmap time. The cheap fixes are cheap because they change little.

A postmortem template that lists conditions, not a cause

Drop this into your existing doc template in place of the Root Cause field:

CONTRIBUTING CONDITIONS  (a list, never one)
  - What had to be true for this to happen?
  - What usually stops this, and why didn't it this time?
  - Has this almost happened before?

HOW IT MADE SENSE
  - For each key action: what could the person see,
    and why was this reasonable with that information?

DETECTION AND RECOVERY
  - When did it start? When did we know? When did users know?
  - What made recovery slower than it needed to be?

ACTION ITEMS  (each with an owner and a date)
  - Layer: less likely / known sooner / smaller / back faster
  - Replay Test: with only this fix, where does it stop?
  - Does it cover other incidents, or only this one?

The single highest-leverage change: rename Root Cause to Contributing Conditions and make it a list. People fill in the form they're given.

Key Takeaways

• Root cause is where the investigation stopped, not where the incident started.
• Incidents need several conditions at once; one cause → one fix → the rest stay armed.
• Analysis usually stops at The Person, The Line or The Wall. Each has a better question.
• The 5 Whys walks one causal path and stops at an arbitrary depth.
• Run the Replay Test on every action item: with only this fix, where does the incident stop?
• Spread fixes across four layers: less likely, known sooner, smaller, back faster.
• Prefer fixes to shared conditions; delete "be more careful."

FAQ

What is wrong with root cause analysis in software postmortems?

Nothing is wrong with looking for causes. The problem is looking for one. Incidents in complex systems need several conditions to be true at the same time, so naming a single root cause leads to a single fix that leaves the other conditions in place. The incident returns through a different trigger.

Why is the 5 Whys technique criticized?

It follows a single chain of causes when incidents have many, different teams asking the same questions reach different "root causes," and the stopping point (five) is arbitrary. These criticisms are laid out in Alan Card's 2017 paper in BMJ Quality & Safety.

What should replace "root cause" in a postmortem template?

A list of contributing conditions: everything that had to be true for the incident to happen, plus what normally prevents it and why that didn't work this time. Pair it with questions about detection, recovery, and how each key action made sense to the person at the time.

How do you write good postmortem action items?

Give each one an owner and a date, label its layer (less likely, known sooner, smaller, back faster), and run the Replay Test: replay the incident with only that fix and see where it stops. Prefer fixes that remove conditions shared by many possible incidents. Avoid items that only ask people to be more careful.

What is a blameless postmortem, and is "human error" ever the root cause?

A blameless postmortem examines the conditions that made an action reasonable at the time instead of assigning fault. "Human error" is a starting point, not a conclusion: if an action made sense to someone, something in their environment made it make sense, and that is what you can change. AWS's 2017 S3 summary is a good public example: a mistyped input was the trigger, and the fixes targeted the tool.

Why do the same incidents keep happening after a postmortem?

Usually because the action items fixed the trigger, the last step before the outage, while the conditions underneath it stayed in place. A different last step then produces a near-identical incident. Fixing shared conditions and adding detection and containment breaks that cycle.

Related reading

• Nobody Deployed Anything. Production Broke Anyway.: config, flag and data changes as the unreviewed fast lane (Knight's repurposed flag is one).
• The Cause Was Fixed. The Outage Wasn't.: when recovery, not the trigger, is the real incident.
• Your Career Isn't Measured in Code. It's Measured in Blast Radius.: the "smaller" layer, as a career lens.
• Your Best Work Erases Its Own Evidence: why prevention work goes unseen, and how to make it visible.

The bottom line

The Root Cause field asks for one answer, and your team will always give it one. Whatever lands there becomes the fix. Ask "how" instead of "why," list the conditions, and test every action item by replaying the incident without the others.

A postmortem that finds one cause will meet the incident again. One that finds the conditions might not meet it at all.

New here? Start Here: A Plain-English Map to Understanding AI is the front door to everything I write. Get one of these in your inbox each week → subscribe to the Simply Explained newsletter.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.