§ 95
Pesticide Paradox
Every way of finding bugs leaves behind the bugs it cannot find, so a test suite that never changes finds fewer and fewer.
Keep in mind
Keep changing how you look for bugs
What to do
Keep the regression suite, but do not count on it to find new bugs. Add tests aimed at what the suite does not exercise: new inputs, fuzzing, property-based tests, exploratory sessions. When a bug escapes to production, ask which kind of test would have caught it and add that kind, not only that one case. Review the suite on a schedule and remove tests that duplicate each other.
Why it happens
A test method catches a particular kind of bug, and once a team has run it for a while those bugs are fixed and developers learn not to write them. What remains are the bugs the method is blind to, so the same tests keep passing while the defects move to places they do not look. As with insects after seasons of the same pesticide, what survives is what the treatment cannot kill. A green suite then shows how well the code avoids yesterday's bugs, not how few bugs it has.
When it stops applying
Old Tests Still Guard: a test that has not failed in years still keeps a fixed bug from coming back and pins behavior others rely on, so retiring old tests to chase new bugs gives up protection they still provide.
Source
Boris Beizer, Software Testing Techniques (2nd ed., Van Nostrand Reinhold, 1990), "The Pesticide Paradox and the Complexity Barrier" — "Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual", and "Test suites wear out."
Read the sourceCite this
Hammurabi, § 95, “Pesticide Paradox.” https://kanywst.github.io/hammurabi/laws/pesticide-paradox/
BibTeX
@misc{hammurabi-pesticide-paradox,
title = {Pesticide Paradox},
author = {{Hammurabi contributors}},
note = {Hammurabi, § 95},
url = {https://kanywst.github.io/hammurabi/laws/pesticide-paradox/}
}Related laws
- 87Beyoncé RuleIn a shared codebase, the behavior no test checks is the behavior that changes without warning.
- 84Chaos EngineeringResilience that has never been exercised is a hypothesis, and production will test it whether or not you schedule the test.
- 105Test Oracle ProblemA test can only catch a wrong answer if something independent of the code knows the right one, and for many programs nothing does.