§ 79
Avoid Fallback
A fallback path runs only after something has already failed, so the least-tested code in the system runs under the worst conditions.
Keep in mind
Distrust code that runs only when things are already broken
What to do
Before adding a fallback, ask whether the same effort would make the primary path more available: push data to the caller ahead of time instead of pulling it at request time, add redundancy. If you keep a fallback, run it in production continuously, not only on failure, and treat its output as an equally valid source. Delete fallback paths nobody has exercised.
Why it happens
Fallback code sits idle in normal operation, so its bugs, its capacity limits and its stale data go unnoticed until it is needed. It then switches on in the middle of an outage, when load is unusual and other parts are degraded, and adds a new mode to a system that is already behaving strangely. A fallback that routes around a cache straight to a database, or onto a secondary service, can move load onto something never sized for it and widen a partial failure; Amazon.com's outage around 2001 began that way. Amazon's conclusion is that the effort is usually better spent making the primary path more available.
In practice
On November 2, 2023, a power failure at Cloudflare's main Portland data center took down its control plane and analytics, and Cloudflare failed over to its disaster-recovery sites in Europe. The fallback had never been tested at that scale: Cloudflare had "never tested fully taking the entire PDX-04 facility offline," some newer products had no tested DR procedure, log processing was not supported at the DR site at all, and retrying clients swamped the services that did come up. Full restoration took until November 4.
Read the accountWhen it stops applying
Graceful Degradation: returning a reduced answer beats returning an error when the reduced path runs all the time, as when a recommendations panel falls back to a static list it serves anyway; the danger is the path that never runs, not degradation itself.
Source
Jacob Gabrielson, Avoiding fallback in distributed systems (Amazon Builders' Library, 2019) — the Amazon.com outage of around 2001, when every web server fell back to querying the database directly once the caches failed together, and "At Amazon, we avoid fallback in our systems because it's difficult to prove and its effectiveness is hard to test."
Read the sourceCite this
Hammurabi, § 79, “Avoid Fallback.” https://kanywst.github.io/hammurabi/laws/avoid-fallback/
BibTeX
@misc{hammurabi-avoid-fallback,
title = {Avoid Fallback},
author = {{Hammurabi contributors}},
note = {Hammurabi, § 79},
url = {https://kanywst.github.io/hammurabi/laws/avoid-fallback/}
}Related laws
- 78Static StabilityA system that has to take new action to survive a failure depends on that action succeeding at the moment things are going wrong.
- 80Constant WorkA system whose work grows with the number of changes or failures does the most work exactly when it has the least headroom.
- 48Ironies of AutomationAutomation leaves people only the hardest cases, after their skills have faded from disuse.
- 84Chaos EngineeringResilience that has never been exercised is a hypothesis, and production will test it whether or not you schedule the test.