All insights

Reliability

Error budgets: deciding when to stop shipping

Every team argues about speed versus stability. An error budget ends the argument by turning it into a number that both sides agreed on in advance.

Start from the target

A service level objective says how reliable a service needs to be, measured from the user's side. If the objective is 99.9% of requests succeeding over 30 days, the remaining 0.1% is the error budget. For a service measured on uptime, that is roughly 43 minutes a month.

That budget exists to be spent. Releases, migrations and experiments all carry risk, and the budget is the amount of risk the business has already accepted.

Use it as a decision rule

  • Budget remaining: keep shipping. Reliability is good enough and extra caution only slows delivery.
  • Budget nearly gone: slow down. Prioritise fixes, smaller releases and safer rollouts.
  • Budget exhausted: stop feature releases until the service is back inside its objective.

Alert on the burn rate

Alerting on every error is noisy. Alerting on how fast the budget is burning is not. A burn rate of 1 means the budget lasts exactly the 30 days. A burn rate of 14.4 sustained for one hour consumes 2% of the monthly budget, which is worth waking someone for. A burn rate of 6 over six hours uses 5% and is also worth a page. A slow burn, such as a rate of 1 over three days, deserves a ticket rather than a page.

Pairing each long window with a short one (for example one hour and five minutes) stops the alert from firing long after the problem has passed.

Where to begin

  1. Pick one user journey that matters, such as checkout or sign-in.
  2. Measure its success ratio where requests enter the system.
  3. Set the objective slightly below what the service achieves today.
  4. Review the budget monthly and adjust the target when the data says so.

One well-chosen objective changes more conversations than a dashboard with fifty graphs.