Chaos Engineering

Failover paths, retries and redundancy run only when something breaks, so they decay between failures: a config drifts, a dependency is added, a timeout changes, and nobody notices until the real outage. Staging cannot reproduce production's traffic, data and coupling, so a test there tells you less than you would hope about the system people actually use. Injecting failure on purpose, in production, against a stated hypothesis about steady-state behavior, makes those weaknesses show up on a weekday afternoon with engineers watching instead of at 3 a.m. Netflix built Chaos Monkey early in its move to AWS to kill instances at random, so every service had to survive losing one.

In late September 2014, AWS rebooted about a tenth of its EC2 fleet on short notice to patch a Xen vulnerability. Of Netflix's 2,700+ production Cassandra nodes, 218 were rebooted and 22 did not come back, yet Netflix reported zero downtime that weekend. The team credited the Chaos Monkey runs against Cassandra it had started a year earlier, which had forced it to build the automation that detected and replaced the failed nodes.

First, Do No Harm: deliberately degrading a system real customers depend on spends their trust, and much of the same knowledge comes cheaper from fault-injection tests, game days and design review; on a system already known to be fragile an experiment only confirms what you knew, so fix it first.