Software Engineering · Engineering Process
Error Budgets as a Release Gate: The Arithmetic Behind the Freeze
Converting a service level objective into a measured allowance of failure, normalising it with burn rate, alerting on two windows at once, and the places where the arithmetic stops matching user experience.
A service level objective states the reliability a service is meant to provide. Its complement states how much unreliability is therefore permitted, and that quantity is what makes the objective operational. The permitted failure, expressed in minutes of downtime or in a count of failed requests, is the error budget. It is spent by the same events that would otherwise be argued about one at a time, and when it is gone, the argument about whether to ship has already been settled.
The design is an accounting device. Its power comes from making a promise quantitative enough to be consumed, and its weaknesses come from all the places where the accounting unit does not match what users experience.
From an objective to an allowance
The choice of indicator determines the arithmetic. A time-based indicator asks what fraction of wall-clock time the service met its criterion; a request-based indicator asks what fraction of requests succeeded. For a monthly target of 99.9 percent, a time-based budget over a thirty-day month is 0.001 multiplied by 43,200 minutes, or 43.2 minutes of total unavailability. A request-based budget over the same month with a hundred million requests allows a hundred thousand failures.
The two units diverge sharply in behaviour. A time-based indicator treats a ten-minute outage at 3 a.m. and a ten-minute outage at noon as identical, and treats a degradation affecting one percent of requests for a day as nothing at all, provided the criterion counts the day as failed only if the service is unavailable. A request-based indicator does the opposite: it charges by volume, so a fault that hits only a small share of traffic barely moves the number, and it charges a heavy client’s requests exactly as much as a light client’s.
| Indicator | Unit | 99.9 percent monthly allowance | Blind to |
|---|---|---|---|
| Time-based availability | minutes | 43.2 minutes | partial degradation |
| Request success ratio | requests | 0.1 percent of volume | a single tenant with total failure |
Neither is correct in general. The choice should follow what the service’s users lose when it fails, which for most request-serving systems is closer to the request unit, provided the requests are counted where the user experiences them.
Burn rate as a common currency
Comparing an incident’s cost with the budget in raw units requires knowing how much of the window has elapsed, which makes the comparison awkward during the window. Burn rate removes the awkwardness by normalising: it is the observed error rate divided by the permitted error rate.
A burn rate of one consumes the entire budget exactly at the end of the window. A burn rate of two exhausts it in half the time. A burn rate of 14.4 sustained for one hour consumes 2 percent of a thirty-day budget, since one hour is 1/720 of thirty days and 14.4/720 is 0.02. That identity is the basis for alert thresholds: a threshold at a given burn rate over a given window corresponds to a fixed share of the budget, independent of the service, the traffic, and the objective.
WINDOW_MIN = 30 * 24 * 60 # a 30-day objective window
def burn_rate(slo, observed_bad_ratio):
# Observed failure share relative to the permitted share.
return observed_bad_ratio / (1.0 - slo)
def budget_fraction_spent(slo, observed_bad_ratio, elapsed_min):
return (observed_bad_ratio * elapsed_min) / ((1.0 - slo) * WINDOW_MIN)
def remaining(slo, observed_bad_ratio, elapsed_min):
return max(0.0, 1.0 - budget_fraction_spent(slo, observed_bad_ratio, elapsed_min))Because the units cancel, a single alerting configuration can serve services with different objectives: the threshold is expressed as a multiple of the permitted rate rather than as an absolute error rate. A service at 99.99 percent and one at 99.5 percent both alert when they burn the same fraction of their own allowance in the same time.
Two windows in the alert condition
A single long window detects slow burns but recovers slowly: an alert computed over an hour stays true for most of an hour after the fault is fixed, which delays the all-clear and trains responders to ignore the alert while it drains. A single short window reacts quickly but fires on transient spikes that would never consume a meaningful share of the budget.
The standard construction requires both a long and a short window to exceed their thresholds. The long window is sized so that the combination corresponds to a budget fraction worth waking someone for, typically a few percent of the monthly allowance. The short window, on the order of minutes, exists to make the alert stop: once the condition clears in the short window, the combined condition is false, so the alert resets even though the long-window average is still elevated.
Two or three such pairs, at different burn rates and window lengths, cover the useful range: a fast burn that justifies immediate response, a moderate burn that justifies a ticket and an investigation within the day, and a slow burn that justifies examining the trend at the next review. The tiers should map onto the response tiers described in severity tiers and escalation thresholds, so that a burn rate produces a page or a ticket by rule rather than by judgment.
What the freeze does and does not buy
The policy layer converts the number into a decision. A typical policy states that while the budget is exhausted, changes that are not explicitly justified as reliability fixes or incident remediations are halted, and that the halt lifts when the budget recovers above a stated fraction. The important property is that the rule is agreed before the incident, because the moment of exhaustion is exactly when a team is least willing to stop.
The cost of the gate is real and should be stated in the policy rather than discovered. A freeze penalises the changes that would improve reliability unless they are carved out, so the carve-out must name the categories: the fix for the incident that consumed the budget, work that reduces the probability of that class of incident, and security patches. Without the carve-out, a freeze slows the remediation more than it slows the risk.
A second cost is accumulation. Work deferred during a freeze does not disappear, and it lands in a burst when the freeze lifts, which produces a change volume spike. Some policies throttle the resumption rather than removing the constraint at once, and the queueing behaviour of a change queue explains why the spike is hazardous: a sudden increase in arrivals to a stage with fixed capacity lengthens the queue and increases the time each change waits for review and deployment.
The trade-off of the policy as a whole is a choice of operating point. A budget that never binds is decorative and provides no signal. A budget so tight that ordinary change exhausts it converts the policy into a permanent freeze, which is functionally a decision to stop changing the system and is rarely intended.
Where the arithmetic stops matching experience
Four mismatches recur, and each has a mitigation that is cheaper than changing the objective.
Averaging over requests hides small populations. If 0.1 percent of requests fail but they all belong to one tenant, that tenant experiences a total outage while the budget reports negligible consumption. Per-tenant indicators, or a maximum over tenants, surface the case that the aggregate conceals.
The denominator is not stable. Failures measured as a share of requests shrink when traffic shrinks, so a partial outage that reduces incoming traffic can consume less budget than a smaller outage at full volume. Worse, a fault that prevents requests from arriving at all may not be counted as a failure, because nothing failed; it simply did not exist. Health checks and synthetic probes exist to detect exactly this case.
Missing data has to be assigned a value before it happens. If the indicator cannot be computed because the telemetry collection is down, treating the gap as good conceals an outage in the monitoring, and treating it as bad spends the budget on a measurement failure. Neither is universally right; what matters is that the policy states which one applies, so that the gap is not resolved by whoever notices it first.
The measurement point decides what counts. A server-side indicator records a success as soon as the response leaves, while the client may time out before it arrives, and a proxy in between may fail a request that the server counted as good. The indicator should be computed as close to the user as the system can afford, and the choice should be documented alongside the objective.
One budget per service, or one per relationship
A budget attached to a single service measures that service against its own objective. For a dependency whose failures propagate, the useful unit is often the relationship: the caller’s view of the callee, which is measurable from the caller’s side and includes the failures the callee never saw. Aggregating several services into one budget hides the case where one consumes most of the allowance, and splitting by caller multiplies the accounting without necessarily adding information. The decision follows from who acts on the number: a budget is worth maintaining only where a team can change something when it is exhausted.