Static Stability

Recovering from a failure usually means doing something the system does not do in normal operation: launching replacement instances, calling a control plane, changing routes. Control planes have more moving parts than data planes and so are more likely to be impaired, which means recovery leans on the weaker half of the system at the moment it is needed most. A statically stable design keeps working when a dependency is impaired, because the capacity and state it needs are already in place. Amazon runs its stateless services across three Availability Zones overprovisioned by 50 percent, so losing one leaves the other two able to carry the load without launching anything.

On December 7, 2021, an internal network problem in AWS's us-east-1 region impaired the EC2 APIs that launch and describe instances for hours, and new launches did not fully recover until 2:40 PM PST. AWS's summary noted that "running EC2 instances were unaffected by this event": workloads already running kept serving because the data plane did not need the impaired control plane, while anything that depended on launching new capacity to recover was stuck.

Cost of Idle Capacity: capacity held for a failure that rarely happens is paid for every day, and scaling to demand is cheaper wherever a slow recovery is acceptable; the source itself concedes that building services zone by zone rather than regionally adds complexity.