§ 111
Most Ideas Fail
In online controlled experiments at large web companies, most ideas that survived the team's own pruning do nothing or make things worse once they are measured, and experts tend to be poor at telling in advance which ones.
Keep in mind
Treat your favourite idea as a hypothesis that will probably fail
What to do
Before building, write down the metric the idea is meant to move and by how much, then ship it behind a flag to a randomised share of users. Drop or rework what does not move the metric, however sure the team was. Where you cannot randomise (too little traffic, a long-term bet, a change you cannot roll back), say so and decide on judgement without claiming it was measured.
Why it happens
Every feature is built because someone believes it will have value, and the review that approves it runs on the same beliefs. When Microsoft measured ideas in online controlled experiments, "only about 1/3 of ideas improve the metrics they were designed to improve"; the authors estimate that a team launching ten ideas without measuring gets roughly a third good, a third flat and a third negative, so the total value is small. Teams tend to run experiments when they are less sure, but the paper notes that Amazon evaluated every new feature and still saw a success rate below 50%. Without a measurement nobody finds out: "For most ideas, the change in key metrics will be too small to detect over time." Guessing does not fix it: "It is humbling to see how bad experts are at estimating the value of features (us included)." In the authors' survey on the outcomes of eight of their A/B tests, none of more than 200 respondents got six or more right.
When it stops applying
Conviction Bets: some decisions cannot be put through a controlled experiment at all; the same paper agrees with Davenport that experiments suit strategy execution, not strategy formulation, and are "not suited for assessing a major change in business models, a large merger or acquisition", and grants that measuring is sometimes "non-economical"; by this entry's reading, there someone has to commit on judgement.
Source
Ronny Kohavi et al., Online Experimentation at Microsoft (Microsoft ThinkWeek, 2009) — "only about 1/3 of ideas improve the metrics they were designed to improve", the estimate of a third good, a third flat and a third negative, and the survey of eight A/B tests. The counter-force is the same paper's section 6.7, which agrees with Thomas H. Davenport, How to Design Smart Business Experiments (2009, as cited in the paper), that experiments are not suited to strategy formulation, and distinguishes "not measurable" from "non-economical to measure"; reading that as a licence to commit on judgement is this entry's, not the paper's.
Read the sourceCite this
Hammurabi, § 111, “Most Ideas Fail.” https://kanywst.github.io/hammurabi/laws/most-ideas-fail/
BibTeX
@misc{hammurabi-most-ideas-fail,
title = {Most Ideas Fail},
author = {{Hammurabi contributors}},
note = {Hammurabi, § 111},
url = {https://kanywst.github.io/hammurabi/laws/most-ideas-fail/}
}Read in a column
- Deciding what to build Next in the column: Cost of Delay
Related laws
- 21Survivorship BiasConclusions drawn only from successes ignore the failures that used the same strategy.
- 98Genchi Genbutsu (Go and See)Decisions made from reports drift away from what is actually happening, because each layer of summary drops the detail that did not fit its form.
- 115Twyman's LawWhen real effects that large are rare, a metric that moves surprisingly far is more often a bug in the measurement or the analysis than a real effect.
- 107Cost of DelayWork gets ordered by size, by gut feel or by whoever asks loudest, while what each item loses every week it waits goes uncounted, so the most valuable work often waits behind the merely visible.
- 75Working BackwardsTeams that start from a solution often build things no customer needed; starting from the announcement exposes that early.
- 110Wicked ProblemsFor some problems, working out what the problem is and choosing a solution are the same act, so the requirements cannot be settled before the design starts and there is no point at which the problem is done.