Anomaly Detection on a Bill That Is Mostly Flat

A cost anomaly alert has to answer a question that ordinary monitoring avoids: what should this figure have been? For latency or error rate the expected value is often obvious, since the healthy state is near zero. For spend the healthy state is a nonzero number that drifts upward with traffic, steps at deployments, and oscillates on a weekly period. A detector that flags any deviation from a fixed threshold will either fire constantly or never fire, and the choice between those failure modes is set by the threshold rather than by the data.

The series is periodic, trending, and stepped

A daily cost series for a working system has at least three components that must be separated before an anomaly can be defined. There is a weekly cycle: weekday load differs from weekend load by a factor that is stable for a given business. There is a trend, usually upward, from growth in traffic or data. And there are steps, from deployments, migrations, and configuration changes, that persist rather than decay.

A detector built on a fixed threshold handles none of the three. If the threshold sits above the weekly peak it will miss an anomaly that is small relative to the cycle but large in absolute terms. If it sits below the peak it fires every weekday. If it tracks the trend, it adapts to growth and stops distinguishing growth from a leak.

The standard treatment decomposes the series into these components and detects on the residual. A median-based form uses the same phase of the cycle rather than a mean, because the median is not moved by the events the detector is supposed to find. If an expensive incident lasts three days, a mean over the preceding four weeks absorbs part of it into the baseline, and the next incident of the same size produces a smaller residual. The median over the same weekday, taken across several weeks, is unaffected by a small number of extreme days.

def baseline(series, day, weeks=6):
    """series: {day_index: cost}. Median of the same weekday over the preceding weeks."""
    weekday = day % 7
    vals = [series[d] for d in range(day - 7 * weeks, day, 7)
            if d in series and d % 7 == weekday]
    if not vals:
        return None
    vals.sort()
    n = len(vals)
    return vals[n // 2] if n % 2 else 0.5 * (vals[n // 2 - 1] + vals[n // 2])

The residual is the observed value minus the baseline, and the threshold is applied to the residual rather than to the raw figure. The threshold itself must be relative, because the noise in a cost series scales with the size of the figure: a service spending a thousand units per day fluctuates by tens, and one spending a hundred thousand fluctuates by thousands. A threshold in percentage terms with an absolute floor below which nothing is reported gives both a stable alert rate and a bound on wasted attention.

What the detector is actually looking for

There are three distinct failure patterns in a cost series, and they require different detectors.

A step is a persistent level change. It is detected by comparing a recent window against a baseline window and requiring the difference to persist for several periods, so that a one-day spike is not reported as a step and a genuine migration is. A step that is caused by a deployment is legitimate and must be acknowledged, otherwise the baseline adapts and the alert is lost; a step that is not explained by any known change is the interesting case.

A drift is a slow change in the growth rate. It is the hardest to detect because each day’s figure is within the noise of the previous day’s, and the cumulative difference only becomes large after many days. Detecting it requires a detector on the trend component rather than on the residual: the slope of a fitted line over a window, compared against the slope over the preceding window. A service whose spend grows by a few percent per week doubles in a matter of months, and no per-day residual will exceed its threshold at any point during that growth.

A spike is a transient. It is the easiest to detect and the least important, because a spike that ends is a spike that has already ended. The useful spike detector reports the cause rather than the amount: a spike in compute hours without a spike in request count indicates a provisioning event or a runaway loop, and a spike in both indicates demand.

The distinction matters because the three patterns have different costs of false positives. A step alert that fires on every deployment trains its readers to ignore it. A drift alert that fires late still has value, since the correction is a configuration change rather than an emergency. A spike alert that fires on legitimate traffic bursts is pure noise.

Price and quantity have to be separated

A cost figure is a product of quantity and price, and an anomaly in the product can come from either factor. The distinction determines the response, and a detector that reports only the product hides it.

A change in quantity appears as a change in the usage records: more instance-hours, more gigabytes, more requests. A change in price appears as a change in the effective rate with usage unchanged. Price changes come from several sources: a commitment expiring, a discount no longer applying to a resource because it moved outside the covered scope, a resource changing family so that it no longer matches a reservation, or a tier boundary being crossed.

The last of these is the one that looks like an anomaly and is not. When a service prices usage in tranches, crossing a boundary raises the marginal rate for the increment, and the total rises faster than the quantity. A detector on the total will report a step at the crossing, and the correct response is not to reduce usage but to recognize that the operating point has moved. Detecting on quantity and on effective rate separately makes the boundary visible as a rate change with no quantity change.

The same decomposition explains why a bill can be flat while the underlying usage grows: a commitment that covers the growth absorbs it into the discounted rate until the committed quantity is exhausted, at which point the incremental usage is priced at list and the bill steps. The step is a signal that the commitment is fully consumed, which is a decision point rather than an incident. That relationship is set out in commitment instruments and break-even utilization.

Attribution decides which alerts are actionable

An alert on a total tells a reader that something changed but not where. The response requires drilling from the total to a dimension, and the dimensions available are those the usage records carry: account, service, region, resource tag, and whatever else the provider exposes. A cost anomaly programme is therefore bounded by its tagging discipline, and an untagged resource produces an alert that cannot be resolved without manual search.

A practical structure reports each alert at the finest dimension that both changed and is attributable. If the change is isolated to one account, the alert names the account. If the change is spread across many accounts within one service, the alert names the service, because that pattern indicates a price or configuration change rather than a workload change. The distinction between a change localized in one dimension and a change distributed across many is the most informative single feature of a cost anomaly, and it is available before any drill-down.

There is a tension between sensitivity and attribution. A detector with a high threshold reports few alerts, all of which are large, and large alerts are usually attributable because they are caused by large changes. A detector with a low threshold reports many alerts, most of which are small fluctuations within a single resource, and the attention cost exceeds the money saved. The threshold is therefore a budget decision: the expected value of an alert is the money it saves times the probability it is real, minus the cost of investigating it, and small alerts have negative expected value regardless of their accuracy.

Limits of what a detector can know

A cost anomaly detector has no access to intent. It cannot distinguish a deliberate capacity increase from an accidental one, so every alert requires a human or a correlated signal to classify. The correlated signals that help are deployment records, configuration change logs, and traffic counters, and a detector that can join against them converts a raw anomaly into a candidate explanation.

A second limit is that the baseline is historical, so a system that has just changed shape has no valid baseline. A migration that moves a workload from one service to another produces a step down in one series and a step up in another, and both will alert. Suppressing alerts for a known change window is necessary, and the window must be long enough for the new steady state to be observed several times, otherwise the baseline is fitted to the transition rather than to the new level.

A third limit is that a detector tuned to the current scale will miss an anomaly that is small relative to the total. A leak of one percent of a large bill is invisible against the noise, and it accumulates slowly enough that the drift detector may attribute it to growth. The only remedy is to detect at a finer granularity than the total, which returns to attribution: a leak inside a single resource group is visible in that group’s series even when it is invisible in the aggregate. This is the same argument as allocating shared cost to a unit of work, applied to detection rather than to chargeback, and it makes attribution quality the limiting factor for everything downstream of the invoice.