Column
When the numbers run the work
Teams measure to see what is happening, and can end up working to the measure. This column reads through how that can go wrong, from choosing what to count, to rewarding it, to reading the result, and then how to keep a number useful.
Numbers earn their place by pointing at where to look first.
Pareto Principle
Most of the effect comes from a small share of the causes, such as bugs in a few modules.Keep in mind: Find the few causes that matter and start thereA ranking only covers what someone thought to count.
McNamara Fallacy
Decisions drift toward whatever can be counted, until what cannot be counted is treated as if it did not exist.Keep in mind: Name what you cannot count, and keep it beside the numberThat harm comes from what the number leaves out, even while the number is honest. Tie it to a reward and the number itself can stop measuring well.
Goodhart's Law
A number stops measuring well once people are rewarded for moving it.Keep in mind: Watch the outcome, not the number that stands for itGaming a number is the mild case. The severe case is a reward that makes the problem itself grow.
Cobra Effect
An incentive meant to reduce a problem rewards people for producing more of it.Keep in mind: Ask how each incentive could be earned without fixing anythingReading the results has its own trap, because the cases that make it into the report are not the only ones that happened.
Survivorship Bias
Conclusions drawn only from successes ignore the failures that used the same strategy.Keep in mind: Count the failures before you copy the successEven the numbers that are in the report can mislead, most of all the surprising ones, and a surprisingly good one is easy to leave unchecked.
Twyman's Law
When real effects that large are rare, a metric that moves surprisingly far is more often a bug in the measurement or the analysis than a real effect.Keep in mind: Treat a surprising number as a bug report until it survives checkingChecking the plumbing still means checking a number against other numbers. At some point someone has to look at the work itself.
Genchi Genbutsu (Go and See)
Decisions made from reports drift away from what is actually happening, because each layer of summary drops the detail that did not fit its form.Keep in mind: See the real thing yourself before you trust the summaryAnother way is to agree, before the number moves, what the team will do when it does, so nobody argues over it in the moment.
Error Budget
An SLO below 100% leaves a measurable allowance of failure, and that allowance can be spent.Keep in mind: Spend the allowed failure on shipping, and stop when it runs out
