Column
Before and after an incident
An incident is often several things lining up rather than one bad moment. This column reads from the small exceptions that build up before it, through the moment it is noticed, to what a team does afterwards so the next one is less likely.
It often starts well before anything breaks, with a shortcut that worked.
Normalization of Deviance
A broken rule that causes no immediate harm quietly becomes the new normal.Keep in mind: An exception that caused no harm is not proof that it is safeAn accepted exception does not stay alone. Left in plain sight, it invites the next one.
Broken Windows Theory
Visible disorder left unrepaired can signal that standards are optional, inviting more.Keep in mind: Fix the defect in front of you before it becomes the standardWhen the first real warning arrives, it looks like one more of the exceptions everyone has learned to live with.
Normalcy Bias
Faced with a real anomaly, people reach for the familiar, harmless explanation.Keep in mind: Ask what it would mean if it were real before you explain it awayNoticing is half of it. The other half is whether whoever noticed feels free to say so out loud.
Andon (Stop the Line)
Letting anyone who sees an abnormality stop the line keeps a defect from traveling downstream, where it costs more to find and fix.Keep in mind: Let anyone stop the line, and never punish a stopAfter the incident the same question returns. Will the people closest to it say what they saw?
Blameless Postmortem
Punishing the person nearest a failure teaches people to hide what they did.Keep in mind: Fix what made the mistake easy, not the person who made itAn honest account gives the analysis something to work with. How far it digs decides what the fix can reach.
Five Whys
The first answer to "why did this fail" is usually a symptom, and fixing a symptom leaves the cause in place to fail again.Keep in mind: Keep asking why until you reach something you can changeOne chain of whys follows one path. Looking back, there is often more than one place the incident could have been stopped.
Swiss Cheese Model
Every safeguard has holes; accidents happen when the holes in several layers line up.Keep in mind: Never stop at the one layer that failedAdding another layer is the obvious response, and in some systems the new layer is one more way to fail.
Normal Accident Theory
In complex, tightly coupled systems, accidents are built into the structure.Keep in mind: Loosen the coupling before you add more safeguardsWhatever the analysis finds, it ends as action items, and the kind of item decides whether the next incident is less likely.
Mechanisms, Not Good Intentions
Asking people to try harder after a failure changes little, because the failure happened to people who were already trying.Keep in mind: Replace "be careful" with something that works without anyone rememberingThe postmortem steps all start from an incident that already happened. The last step is to learn some of the same lessons without one.
Chaos Engineering
Resilience that has never been exercised is a hypothesis, and production will test it whether or not you schedule the test.Keep in mind: Break it on purpose before production does it for you
