§ 72
Error Budget
An SLO below 100% leaves a measurable allowance of failure, and that allowance can be spent.
Keep in mind
Spend the allowed failure on shipping, and stop when it runs out
What to do
Agree on the SLO and the policy before the incident, not during it: while budget remains, ship; when it is spent, freeze feature releases and put the time into reliability until it recovers. Keep data loss and security outside the budget entirely.
Why it happens
Product teams are judged on velocity and operations teams on reliability, so every release becomes a negotiation decided by whoever argues harder. Past a point users cannot tell the difference (someone on a 99%-reliable phone cannot tell 99.99% from 99.999%), but the last fraction costs the most and blocks the most change. Once the target is explicit, the gap between it and perfect is a budget: 99.9% over 30 days is about 43 minutes of failure, and both sides can see how much of it is left.
In practice
After moving to Google Cloud, Evernote set its first SLO at 99.95% uptime per calendar month for certain services, counting even maintenance windows as downtime, and said it used the SLO and error budget to decide where to spend engineering effort, reviewing performance monthly with Google. In 2017, while chasing a complex bug, it applied an SLO calculation to quantify customer impact and cut its weekly release windows from five to two.
Read the accountWhen it stops applying
Normalization of Deviance: a standing allowance for failure teaches a team that each failure is routine; the outage that should have alarmed people gets logged against the budget and never investigated.
Source
Marc Alvidrez, "Embracing Risk" (with "Motivation for Error Budgets" by Mark Roth), in Beyer, Jones, Petoff & Murphy (eds.), Site Reliability Engineering (O'Reilly, 2016), ch. 3 — where the error budget is defined and "100% is probably never the right reliability target" is argued.
Read the sourceCite this
Read in a column
Related laws
- 14Goodhart's LawA number stops measuring well once people are rewarded for moving it.
- 52The Tail at ScaleWhen one request fans out to many servers, the slowest one sets the response time.
- 73ToilManual, repetitive operational work grows with the service and, left unchecked, crowds out engineering.
- 02Normalization of DevianceA broken rule that causes no immediate harm quietly becomes the new normal.
- 82Andon (Stop the Line)Letting anyone who sees an abnormality stop the line keeps a defect from traveling downstream, where it costs more to find and fix.