§ 115
Twyman's Law
When real effects that large are rare, a metric that moves surprisingly far is more often a bug in the measurement or the analysis than a real effect.
Keep in mind
Treat a surprising number as a bug report until it survives checking
What to do
When a result is far better or worse than past experiments, hold the announcement and check the plumbing first: how the metric is logged, whether the treatment changed what gets recorded, and whether the analysis compares like with like. Then replicate on a fresh sample. Ordinary, expected moves do not need this treatment.
Why it happens
In its usual wording, "Any figure that looks interesting or different is usually wrong!" Kohavi and his co-authors explain it with Bayes' rule. Breakthroughs are rare: if experiment results centre on 0 with a standard deviation of 0.25%, a +2.0% gain in sessions per user is eight standard deviations out, so even a statistically significant result is more likely a bug, such as an instrumentation error. With 5% significance and 80% power, if real breakthroughs are one in 500 experiments, the chance that a significant result is a true positive drops to 3.1%. People also check results unevenly: a bad result for a favoured feature gets questioned, a great one gets celebrated, so the errors that flatter us survive longest.
In practice
An MSN experiment with a new auto-suggest algorithm looked like a huge success: searches on Bing referred from MSN rose dramatically. Investigating because of this rule, the team found the new code was effectively issuing two searches whenever a user picked a suggestion, one of which the browser always dropped.
Read the accountWhen it stops applying
Trivial Changes Can Have a Big Impact: the same paper's Rule #1: trivial changes sometimes move key metrics a lot, like a Bing font-colour change worth over $10M a year; a team that treats every big win as a bug stops running the cheap long-shot experiments that find them.
Source
Folklore, attributed to Tony Twyman via A. S. C. Ehrenberg, The Teaching of Statistics: Corrections and Comments (Journal of the Royal Statistical Society, Series A, 1975); documented in Ron Kohavi, Alex Deng, Roger Longbotham & Ya Xu, Seven Rules of Thumb for Web Site Experimenters (KDD, 2014) — the wording, the Bayes argument and the examples.
Read the sourceCite this
Read in a column
- When the numbers run the work Next in the column: Genchi Genbutsu (Go and See)
Related laws
- 14Goodhart's LawA number stops measuring well once people are rewarded for moving it.
- 21Survivorship BiasConclusions drawn only from successes ignore the failures that used the same strategy.
- 111Most Ideas FailIn online controlled experiments at large web companies, most ideas that survived the team's own pruning do nothing or make things worse once they are measured, and experts tend to be poor at telling in advance which ones.
- 114McNamara FallacyDecisions drift toward whatever can be counted, until what cannot be counted is treated as if it did not exist.