§ 105Maintainability

Test Oracle Problem

A test can only catch a wrong answer if something independent of the code knows the right one, and for many programs nothing does.

Keep in mind

Before trusting a passing test, ask what it was checked against

What to do

For every test that guards something important, be able to say where its expected value came from: a spec, a hand calculation, an independent implementation, a domain expert. Treat snapshot tests and generated tests whose expectations were copied from the current output as change detectors, not as evidence the code is right. Where no oracle exists, test what must hold anyway: invariants, round trips, and relations between the outputs of related inputs. For a reversible change behind good monitoring, a canary can be the check; keep hand-built oracles for changes that are expensive to undo.

Why it happens

Elaine Weyuker pointed out that "A frequently invoked assumption in program testing is that there is an oracle": the tester, or some other mechanism, can tell whether an output is correct. "A program is non-testable if either an oracle does not exist" or checking the output would take the tester an extraordinary amount of time. When nobody can say what the right answer is, the expected value in a test tends to come from running the code itself, and then the test records what the code does, not what it should do; it passes on every bug that was already there when it was written. Tests generated from the implementation are especially prone to this. Chen, Cheung and Yiu, building on Weyuker, wrote that "the availability of test oracles is pragmatically unattainable in most situations," and proposed metamorphic testing: derive follow-up inputs from tests that passed and check that the outputs relate the way they must, which needs no oracle.

When it stops applying

Production as the Oracle: for a cheap, reversible change, a partial and time-limited deployment judged against real traffic and SLOs catches problems no test author thought to write down.

Source

Elaine J. Weyuker, On Testing Non-Testable Programs (The Computer Journal, 1982) — "A frequently invoked assumption in program testing is that there is an oracle", and a program as non-testable when no oracle exists or checking takes an extraordinary amount of time; T. Y. Chen, S. C. Cheung & S. M. Yiu, Metamorphic Testing: A New Approach for Generating Next Test Cases (Technical Report HKUST-CS98-01, Hong Kong University of Science and Technology, 1998) — "the availability of test oracles is pragmatically unattainable in most situations", and tests that need no oracle. The counter-force is from Alec Warner & Štěpán Davidovič, "Canarying Releases", in The Site Reliability Workbook (O'Reilly, 2018) — canarying as "a partial and time-limited deployment of a change in a service and its evaluation."

Read the source

Cite this

Hammurabi, § 105, “Test Oracle Problem.” https://kanywst.github.io/hammurabi/laws/test-oracle-problem/

BibTeX

@misc{hammurabi-test-oracle-problem,
  title  = {Test Oracle Problem},
  author = {{Hammurabi contributors}},
  note   = {Hammurabi, § 105},
  url    = {https://kanywst.github.io/hammurabi/laws/test-oracle-problem/}
}

Text under CC BY-SA 4.0: credit Hammurabi and link the article.

X Bluesky Hatena LinkedIn

Read in a column

When AI writes the code 6 of 10Next in the column: Kernighan's Law

Related laws

Suggest a correction on GitHub