Observability · Alerting
Error Budgets and the Burn Rate That Consumes Them
How an error budget turns a service level objective into a spendable quantity, and how the burn rate compresses a window into a single number that can be compared against a threshold.
An alert threshold needs a number to compare against. The usual candidates are absolute ones: error rate above five percent, latency above two seconds, queue depth above a thousand. Each of those numbers is invented locally, by whoever wrote the rule, and none of them carries a reason. An error budget supplies the missing reason. It converts the service level objective from a promise into a quantity that depletes, and it makes every threshold a statement about how fast that quantity is being spent.
The budget is a fraction of a window
A service level indicator is a ratio of good events to valid events. A service level objective fixes a target for that ratio over a stated window: 99.9 percent of requests succeed over thirty days, for example. The complement of the target is the error budget, so a 99.9 percent objective permits 0.1 percent of the requests in the window to fail.
The same fraction can be read as time. Thirty days is 43,200 minutes, and 0.1 percent of that is 43.2 minutes. A 99.95 percent objective over the same window permits 21.6 minutes, and 99.99 percent permits 4.32 minutes. The time reading is a convenience, not an equivalence: a service can be fully down for four minutes and be well inside a 99.99 percent budget measured in requests, provided the failure happened when traffic was light. Request-based and time-based indicators disagree whenever the failure is correlated with load, which is the normal case.
Two properties of the ratio matter for anything built on top of it. First, the denominator is a count of valid events, not of all attempted events, so a definition of validity is part of the objective and not an afterthought. Requests rejected at the edge before instrumentation, or dropped by a load balancer that never forwards them, are invisible to a ratio computed at the server. Second, the ratio is only meaningful at scale. At ten requests per minute, a 99.9 percent objective over an hour has a budget of 0.6 requests, and a single failure consumes nearly two hours of allowance. The arithmetic is correct and the alerting built on it is not, because the measurement granularity is coarser than the budget.
A third choice, rarely stated, is whether the window rolls or resets. A rolling thirty-day window moves one day forward each day, so the budget recovers continuously and a large incident stops mattering after thirty days. A calendar window resets at a boundary, which produces a visible discontinuity: a service that has spent its entire budget on day twenty-nine starts the next period with a full one. Rolling windows are the safer basis for alerting because the quantity being monitored has no step change that is not caused by the service itself.
Burn rate as a unit of time compression
The burn rate is the ratio of the observed error ratio to the error budget ratio:
$$\text{burn rate} = \frac{\text{observed error ratio}}{1 - \text{SLO}}$$
A burn rate of one means the budget is being consumed at exactly the rate that exhausts it at the end of the window. A burn rate of two exhausts it in half a window, and a burn rate of 14.4 exhausts a thirty-day budget in about two days. The useful property is that the burn rate is dimensionless and comparable across objectives: a burn rate of six means the same thing to a 99.9 percent service and to a 99.99 percent service, even though the absolute error ratios differ by a factor of ten.
Time to exhaustion follows directly from the definition, since the budget is spent at a constant multiple of the nominal rate:
$$t_{\text{exhaustion}} = \frac{W}{\text{burn rate}}$$
where $W$ is the window length. This is the number that turns a burn rate into an operational decision. At 14.4 the remaining budget lasts roughly two days if nothing changes, which is long enough to schedule work and short enough to matter. At 1.0 it lasts a month, which is what the objective intends and is not an emergency.
The standard alerting construction inverts the arithmetic and asks how much budget a short window is allowed to consume. Consuming 2 percent of a thirty-day budget within one hour requires a burn rate of $0.02 \times 720 / 1 = 14.4$. Consuming 5 percent within six hours requires $0.05 \times 720 / 6 = 6$. Consuming 10 percent within three days requires $0.1 \times 720 / 72 = 1$. Those three pairs cover most of the useful alerting space: a fast severe condition, a moderate condition, and a slow drift that would otherwise be invisible.
| Budget consumed | Window | Required burn rate |
|---|---|---|
| 2% | 1 hour | 14.4 |
| 5% | 6 hours | 6 |
| 10% | 3 days | 1 |
Because the threshold is derived rather than chosen, it can be recomputed when the objective changes. Moving from 99.9 to 99.95 percent halves the error budget, so every burn-rate threshold stays numerically identical while the absolute error ratio that trips it halves. That is the intended behaviour: the alert encodes a spending policy, and the policy is expressed in units of the budget.
Why a raw error-rate threshold cannot express severity
A rule written as error_ratio > 0.01 looks like a burn-rate rule for a 99 percent objective, and it is one, but only for that objective. Change the objective and the rule silently changes meaning. Worse, the same rule evaluated over a one-minute window and over a six-hour window describes two completely different operational situations, because the window is the other half of the threshold. Severity is a product of depth and duration, and an error-rate comparison fixes only the depth.
The burn rate fixes both, at the cost of requiring the window to be stated explicitly in the rule. A rule that pages must name the window it evaluates over, and the window must be long enough to contain several samples of the underlying series. A one-hour window evaluated against a fifteen-second scrape interval contains 240 samples, which is ample. A one-minute window contains four, and the ratio computed from four samples is dominated by whichever requests happened to land in them.
Aggregation is the second place where the naive form fails. An error ratio for a fleet is the ratio of summed errors to summed requests, not the mean of per-instance ratios. The two differ whenever traffic is unevenly distributed, and the mean-of-ratios version overweights idle instances whose error ratio is computed from a handful of requests. Prometheus expresses this directly: sum(rate(errors[5m])) / sum(rate(total[5m])) sums first and divides once, while an avg over per-instance ratios divides first and averages after. Only the first matches the definition of the indicator.
What the arithmetic assumes and where it stops
The burn-rate construction assumes the objective is meaningful at the timescale being alerted on. A thirty-day budget consumed at 14.4 times the nominal rate is an emergency because there is only a month of allowance; a service with a weekly release cycle and a budget that is intentionally spent by the end of each week has a different tolerance, and the same 14.4 would fire on ordinary behaviour.
It also assumes the indicator is stationary enough that a ratio over a short window estimates the ratio over the long one. Deployments, cache warmups, and diurnal traffic shifts all violate that, which is why the burn-rate rules that page usually evaluate two windows at once and require both to agree. The pairing exists to suppress exactly the cases where a short window is unrepresentative; Multiwindow Burn-Rate Alerts and the Short Window as a Reset works through the mechanics and the failure that the second window is there to catch.
The last assumption is that the budget is a resource worth defending at all. An objective set far below what users tolerate produces a budget that is never spent, and an alerting policy built on it pages for conditions nobody has noticed. The complement of that error is worse: an objective set above what the service actually delivers means the budget is permanently exhausted and the burn rate is permanently above one, which turns every threshold into a constant firing condition. The number is only useful when it sits between those two failures, and choosing it is a product decision that Symptom-Based Alerting and the Cause Signals It Replaces depends on just as much.
References
- The Site Reliability Workbook, Chapter 5: Alerting on SLOs — Google
- Site Reliability Engineering, Chapter 4: Service Level Objectives — Google
- Prometheus documentation: Query functions — Prometheus project