Observability · SLO Practice

Error Budget Policy: Freezes, Exemptions, and Escalation Arithmetic

How a budget becomes an enforceable rule, why a freeze that everyone can override is worse than none, and how exemption quotas are computed from the budget itself.

An error budget is a measurement until a policy is attached to it, and the policy is what turns it into a control. The measurement says that 62% of the month’s budget remains. The policy says what happens at 0%, who decides whether something at 0% still ships, and what “happens” means concretely enough that two people reading it at 3 a.m. reach the same conclusion.

Most published budget policies are written as an escalation ladder: at some remaining fraction, changes require additional review; at zero, non-critical changes stop; at some negative threshold, the incident process takes over. The ladder is easy to write and hard to implement, because every step of it is a constraint on someone’s ability to do their job, and constraints that block product delivery are tested immediately.

The policy must be evaluated on the same clock as the budget

The first correctness requirement is that the condition triggering a policy step be computable from the same series that define the objective, over the same window chosen in objective windows and time equivalence. A policy that says “the budget is exhausted” is ambiguous under a rolling window, because under a rolling window the budget is never exactly exhausted for more than an instant — as the bad hours age out, the budget returns, and the policy would flip off and on continuously near the boundary.

Two disambiguations are used. The first evaluates the policy on budget consumed over the full remaining window: the condition is that the projected end-of-window consumption exceeds the budget, which is stable under a rolling window as long as the projection uses a long enough lookback. The second applies hysteresis: enter the frozen state when burn over the full window exceeds the target, and leave it only when the budget has recovered to some positive threshold, typically a few percent. Hysteresis is the simpler mechanism and it has a cost: the written policy has to state both thresholds, because a single threshold with implicit hysteresis is not what any reader will implement.

The arithmetic of when a freeze lifts is worth making explicit, since it drives the schedule. If a service is frozen with a full window of burned budget, and it then runs with a burn rate of 0.5, the budget recovers at half a window’s worth per window, so the recovery to a 5% cushion takes about two windows. Under a twenty-eight-day window, that is nearly two months of frozen state. Teams that have not done this arithmetic are surprised by it, and the usual reaction is to declare an indefinite freeze that nobody enforces.

Exemptions are a quota, and the quota is computable

Every budget policy that is actually used has an exemption path. Without one, the policy is either ignored or it blocks work that is genuinely safe, and both outcomes teach the organization to route around it. The design question is how much exemption is permitted, and it can be derived rather than argued.

If a service is over its budget and a class of changes is exempted from the freeze, the exempt class can spend budget. Its total allowed spend is whatever the budget can tolerate over the window, so an exemption quota expressed as a fraction of the window budget is consistent with the rest of the accounting. A quota of 10% of the monthly budget for emergency changes, evaluated as a running total with its own alert at 80% consumption, keeps the exemption bounded and makes it visible before the quota is exhausted rather than after.

Two qualifiers on the arithmetic. First, the quota should be expressed in the same units as the budget — counts of failures, not numbers of changes — because a change that causes a failure is not the same size as a change that does not. Second, quota consumption should be attributed to the change that caused it, which requires the change to be identifiable in the telemetry. Without that attribution, the quota becomes a budget for changes in general and measures nothing.

The exemption that causes the most trouble is the one for a failure whose cause is unknown. If the cause of a burn is unknown and the exemption is granted on the grounds that the change is unrelated, the exemption is a hypothesis rather than a fact, and the hypothesis is being trusted at exactly the moment when the system’s behaviour is not understood. The defensible version restricts unknown-cause exemptions to changes that are reversible and observable within the incident’s detection window.

What “freeze” has to mean to be enforceable

A freeze expressed as “non-critical changes stop” is enforced by social pressure, which works until it does not. A freeze expressed as a required approval from a named role, with the approval recorded somewhere the deploy pipeline can read, is enforceable, and the enforceability is what makes the rest of the policy credible.

The mechanical version usually has three pieces: a change-classification step that identifies whether a change touches the service under freeze, a gate in the deployment pipeline that fails closed for that class while the freeze is active, and an override that requires an explicit role and leaves a record. The gate does not need to be sophisticated. It needs to be checked at the point where changes are applied, not at the point where they are reviewed, because review happens before the freeze may begin.

Two failure modes recur. A gate that can be bypassed by deploying through a different path — a hotfix script, a manual rollout, a different pipeline for the same service — is a gate that only constrains the careful. A gate that fails open on its own error is worse: a policy service that cannot be reached at 3 a.m. should not be the reason a broken change ships, so the intended failure direction has to be chosen deliberately, and the choice made visible. Failing closed during incidents is safer for reliability and more dangerous for recovery, since rollbacks are also changes.

Rollbacks deserve explicit treatment in the policy, because they are the one change class that should never be blocked. A freeze that prevents a bad change from being reverted has inverted its purpose. Most policies name rollback and configuration reverts as always-permitted, and the exception is written before the first freeze, not discovered during one.

The ladder should reflect the cost of being wrong

The escalation steps are usually written in three tiers, and each tier should be justified by what the system does at that burn level rather than by symmetry. A common and defensible structure:

Remaining budgetChange postureNamed decision-maker
above 25%normalservice owner
25% to 0%additional review, no large rolloutsservice owner plus peer
below 0%reliability work first, only fixes and reversalsengineering manager

The thresholds are not derived from anything; they are chosen. What is derived is the response: once the ladder exists, the number of times it is entered is a property of the service, and a service that enters the frozen tier every month has a threshold set too tightly or a system that cannot meet its target. The frequency of entering the tier is itself a signal, and one that is cheap to track since the policy state is already a series.

The asymmetry that most ladders get wrong is the recovery direction. Going down the ladder is automatic; coming back up requires either a full window of clean behaviour (under a rolling window) or a window boundary (under a calendar window). If the policy does not say which, the team will use whichever gives the faster exit, and the documented policy will not describe what happens.

Policy and the incentive it creates

A budget policy changes behaviour before it is ever enforced, because the existence of a threshold changes what people propose. The most important property of the writing is therefore that the exempted cases be narrow and named. A policy with a broad exemption is read by everyone as a policy with no teeth, and the teams that follow it carefully are disadvantaged relative to the teams that do not.

The second-order effect is on targets. A service whose target is set too high will live permanently in the frozen tier, and the rational response to a permanent freeze is to argue the target down. That is the intended outcome, since a target the system cannot meet is a mis-set target, and the policy is what makes the mis-setting uncomfortable enough to correct. The argument only works if targets are revisable on evidence, and if the evidence — the distribution of consumption, not just the total — is available.

That distribution is the part most accounting misses: a budget consumed in one hour and a budget consumed in a long shallow slope have the same remaining percentage and different causes, as discussed in error budget units and burn rate. The response should differ accordingly, and a policy that reacts identically to both will spend its credibility on the wrong problem.

When there is no budget to spend

The policy has a degenerate case when the objective’s target is met by construction or the volume is too thin for the budget to be a number, which is the situation described in low-volume objectives and counting qualifiers. In that case the ladder has no operating range, and the honest policy says so: the service is governed by review of individual failures rather than by a budget, and the review is the control. Writing a percentage ladder for a service with eleven permitted failures per month produces a policy that triggers on a single event and is therefore permanently in the lowest tier, which is the same as having no policy at all.

References

  1. The Site Reliability Workbook, Chapter 2: Implementing SLOs — Google
  2. Site Reliability Engineering, Chapter 3: Embracing Risk — Google
  3. Accelerate: The Science of Lean Software and DevOps — Forsgren, Humble & Kim, IT Revolution Press, 2018