Forecasting Exhaustion for Saturation Alerts

A disk that is eighty percent full is either in no danger at all or a few hours from writing errors, and the single number says nothing about which. What distinguishes the two cases is the rate at which the remaining space is being consumed. Saturation alerts are the class where the useful signal is not the current value but the time remaining, and expressing the alert that way changes which conditions page and which become planned work.

From a percentage to a deadline

The threshold on a resource that fills monotonically is usually written as a fraction of capacity, and the fraction is chosen to leave enough room to react. The choice encodes an implicit assumption about the growth rate, so it is correct only for the growth rate the author had in mind. On a volume growing one percent per day, ninety percent full means ten days of lead time, which is ample and probably means the alert was set too high to be useful. On a volume growing five percent per hour, seventy percent full means six hours, and a threshold at ninety would fire with two hours left.

The projective form replaces the fraction with a horizon. Compute the recent rate of change, extrapolate it to the capacity limit, and alert when the projected time to exhaustion falls below a stated horizon. The horizon is a policy choice with an operational meaning: a four-hour horizon fits within a shift, a thirty-day horizon fits within a planning cycle. The alert then says something a responder can act on directly, because the action is “add capacity before the horizon elapses” rather than “the resource is above some percentage”.

PromQL provides the projection directly. predict_linear(v range-vector, t scalar) fits a least-squares line to the samples in the range and returns the value it predicts t seconds ahead, and deriv(v range-vector) returns the fitted slope per second. Both are documented for use with gauges; applying them to a counter measures something else, because a counter’s rate has to be computed before the fit means anything.

# Alert when the fitted line over the last 6 hours reaches the volume
# limit within the next 4 hours.
predict_linear(node_filesystem_avail_bytes{mountpoint="/data"}[6h], 4 * 3600) < 0

The expression is written against available bytes rather than used bytes, so the condition is a prediction that the quantity crosses zero rather than a prediction that another quantity crosses a constant. That form travels with the volume: a resized filesystem changes the constant in the other form and leaves this one correct.

Why the horizon bounds the fit

A least-squares line through the samples in a window is a statement about the linear trend within that window. Extrapolating it further than a small multiple of the window length assumes the trend continues, and the assumption weakens with distance. A fit over six hours predicting one hour ahead is a modest extrapolation; the same fit predicting thirty days ahead is eight times the window length and is wrong often enough that it should not page.

The relationship between window and horizon is therefore the rule that keeps the alert honest. The window must be long enough to average out the sampling and workload variation in the series, and the horizon must be short enough that the modelled trend is plausible across it. In practice the horizon is rarely more than a small factor of the window, and where a longer horizon is needed, the alert belongs in the ticket class rather than the paging class, since a thirty-day projection is a capacity-planning statement. The routing distinction is the one drawn in Routing Alerts by Required Response: a projection that leaves days of lead time is planned work by construction.

The window’s length has a second effect, on the fit’s sensitivity. A long window is insensitive to a recent change in slope, so a volume that doubled its fill rate an hour ago is projected as though it were still filling slowly. A short window is sensitive to the recent slope and also to noise, and the noise in a slope estimated from a handful of samples can be large enough to flip the sign of the prediction. The window should be a small multiple of the time over which a genuine change in growth rate is expected to matter, which for most saturation conditions is hours.

The failure modes of a linear projection

Non-linear growth. The most common failure is a fill pattern that is not linear at all. Log rotation, snapshot expiry, scheduled batch jobs, and garbage collection produce a sawtooth that is monotone over days and violently non-monotone over hours. A fit over a window that lands inside one tooth can be steeply positive or steeply negative depending on where it starts. The mitigation is to fit over a window that spans at least one full cycle of the pattern, and to check the cycle length before choosing the window.

Deletions and resets. A series that is not monotone cannot be projected meaningfully. A memory gauge that drops after a restart, a connection count that falls when a deployment drains, and a queue depth that empties overnight all produce negative slopes that project away from the limit and therefore suppress the alert. If the quantity can reset, the alert must either be written on the monotone part of it or be paired with a second alert that fires on the value itself.

Outliers inside the window. A least-squares fit is not resistant to outliers, and one extreme sample in a six-hour window can dominate the slope. A single stalled scrape that reports a duplicated value, or a single burst that consumed a gigabyte, shifts the projection for as long as that sample remains in the window. A median-based or trimmed fit is less influenced by a single extreme sample and is not what a least-squares function computes, which is a reason to be careful with any projection whose output decides whether someone is woken.

Extrapolating a rate that is itself an extrapolation. Where the underlying quantity is a sum of rates computed elsewhere, the projection compounds the assumptions of both steps. The inner rate computation already decides how to treat gaps and counter resets, and the outer fit then assumes the resulting series is linear. Each layer is defensible on its own; the combination is a chain of approximations whose error is not visible in the final number.

A projection with no data. Very short windows on sparse series produce fits from very few points, and some engines return no value at all rather than a fit from too few samples. The alert then silently stops evaluating, which is the failure covered in Alerting on Absent and Stale Data and applies to projections as much as to any other expression.

Hysteresis and the flicker at the horizon

A projection that hovers near the horizon crosses it repeatedly, because the slope changes slightly with each new sample and the horizon is a fixed line. The condition has no natural stable point, so the alert oscillates for hours while the resource is genuinely close to the boundary.

The fix is the hysteresis pattern rather than a longer hold duration. Fire when the projection falls below the horizon, and clear when it rises above a multiple of the horizon, which is the two-threshold construction described in Alert State, Hysteresis, and the Cost of Flapping. The gap between the two thresholds should be large enough to exceed the variation of the projection under normal growth, and the multiwindow pattern from Multiwindow Burn-Rate Alerts and the Short Window as a Reset does not fit here, because the quantity being tested is already a rate and a second window would test the same fit twice.

A second guard is to require the projection to be positive before alerting. A negative slope means the resource is being freed, and treating that as a prediction of exhaustion inverts the sign of the signal. This is a one-clause guard in the expression, and it removes an entire class of obviously wrong pages.

What the projection is good for

The projective alert is strongest for quantities that grow monotonically at a rate that is slow compared with the window, and whose limit is a hard boundary. Filesystem space, partition count, inode usage, certificate lifetimes, and quota consumption all fit. It is weakest for quantities that are bursty or that self-correct, such as queue depth, where the linear model is wrong by construction and a value threshold with hysteresis is more honest.

The last limitation is that a projection answers a question about the resource, not about the service. It says when a disk will fill, and says nothing about whether the effect will be visible to users, which depends on what uses the disk and how it fails. The projection therefore belongs with the cause signals that are recorded and displayed but do not page on their own, as argued in Symptom-Based Alerting and the Cause Signals It Replaces. The exception is a resource whose exhaustion is immediately user-visible, such as a hard quota that rejects writes; there, the projection is a symptom alert with a lead time, which is the most useful shape a saturation alert can take.