Recording Rules, Pre-Aggregation, and the Cost of Evaluating at Read Time

An expression evaluated at read time costs whatever the data underneath it costs, multiplied by the number of times it is evaluated. A dashboard with twenty panels refreshed every thirty seconds by ten viewers is six hundred evaluations per minute of expressions that may each scan millions of samples. The arithmetic is not the problem; the problem is that the cost scales with the number of people looking, which is the one quantity a system cannot control.

A recording rule moves the evaluation to write time. The expression runs once per interval regardless of viewers, its output is stored as a new series, and every consumer reads the pre-computed result. The trade is that the stored series is frozen at the rule’s interval and its label set, so anything the rule did not compute cannot be recovered from it later.

The multiplication that motivates pre-aggregation

Consider a histogram with fifteen buckets, five label dimensions of ten values each, and ten thousand instances. The raw series count is in the millions. A query that computes a fleet-wide 99th percentile has to read every one of those series, merge the histograms, and interpolate. That work happens per evaluation.

If ten dashboards each evaluate the same expression once per refresh, the work is repeated ten times per interval for an identical answer. If the expression is materialised by a rule, it runs once. The saving is the ratio of evaluations to rule runs, which for a widely viewed dashboard is large and for an unviewed one is negative, because the rule runs even when nobody is looking.

That asymmetry is the first design constraint. A rule costs a fixed amount of work per interval forever, so it should exist for expressions that are read often or that are too expensive to evaluate on demand. A rule for an expression nobody reads is pure cost, and rules accumulate: each one adds series, adds evaluation load, and adds a name that someone eventually depends on.

What a rule fixes and what it forecloses

A rule stores the output of an expression with the labels the expression produced. Two properties become permanent:

The aggregation level. If the rule sums across instances, the per-instance detail is gone. A later question about one instance cannot be answered from the rule’s output, only from the raw series, which may have been deleted by retention. This is the irreversible part, and it is the reason to record at a finer level than the current dashboards need: the cost is linear in the extra labels, while the loss is permanent.

The time resolution. The rule evaluates at its interval, so the stored series has one sample per interval. A rule evaluating every minute cannot answer a question about a thirty-second spike, and the spike is attenuated by whatever window the expression used. A rule that computes a rate over five minutes and runs every minute produces overlapping windows: each sample is the rate over the previous five minutes, so adjacent samples share four-fifths of their input and are correlated. That correlation is invisible in the graph and makes the series unsuitable for any statistic that assumes independent samples.

The interval choice also determines the phase relationship between the rule and the scrape. A rule evaluating at the same period as the scrape will, depending on scheduling, sometimes see a fresh sample and sometimes see one that is one interval old. Setting the rule interval to a multiple of the scrape interval and aligning the phases removes the jitter; leaving them coprime produces a series whose effective sample spacing varies between evaluations, for the aliasing reasons set out in Scrape Interval, Lookback Delta, and the Resolution of a Rate.

The staleness the rule introduces

A rule’s output is a normal series, so it is subject to the same lookback rules as anything else. When the rule stops producing output, whether because its input series disappeared or because the rule was removed, the last value it wrote remains answerable for the lookback period, conventionally five minutes. During that window, a dashboard shows the last pre-removal value as if it were current.

This is a real hazard for rules that encode health. A rule that computes whether a target is up will, after the target is removed, keep reporting the last value for five minutes. If the last value was healthy, the removal looks like health. The mitigation is the same one that applies to scraped series: an explicit staleness marker when the rule’s input is absent, so the series terminates immediately rather than lingering.

A second staleness appears when the rule’s expression produces no result for a label combination that existed before. The output series simply stops, and the lookback fills the gap with the old value. A rule that aggregates with sum by will drop a group entirely when its members vanish, which is correct for the group but indistinguishable from a scrape failure for the consumer.

Cardinality, again, but in the other direction

Rules usually reduce series count by aggregating, but they can increase it. A rule that computes several expressions from one input produces several outputs. A rule that joins two metrics by a label produces an output with the union of both label sets, which can multiply. And a rule whose expression has a label that the input lacks, such as a constant label added with a literal, produces a new dimension that downstream rules may then combine with others.

The accounting is worth doing explicitly: a rule adds (output series count) minus (input series count that would otherwise have been retained) to the total. When the rule aggregates heavily, that is negative. When it computes a per-instance ratio, it is roughly zero and the benefit is purely that the ratio is precomputed rather than the inputs being cheaper. When it adds labels, it is positive and the rule is a cardinality cost that must be justified by read volume.

There is a related trap: a rule whose input is another rule. Chains of rules are common and each hop adds an interval of delay. A three-hop chain at one-minute intervals has a minimum latency of three minutes between an event and its appearance in the final series, plus the lookback effects at each hop. The delay is not a bug, but it is invisible unless the chain is documented, and it becomes a correctness problem when someone writes an alert against the last hop expecting low latency.

Why downsampling is a different operation

Pre-aggregation reduces series; downsampling reduces samples per series. A long-retention store commonly keeps raw samples for a short window and downsampled aggregates for a long one, computing the aggregate once at compaction time rather than per query.

Downsampling a counter is straightforward: the increment over the downsample interval is the difference between the boundary values, which is exact. Downsampling a histogram requires merging buckets, which is exact for identical layouts and lossy when the layouts differ, as covered in Exponential Histograms and the Mergeable Bucket Layout, and a layout with many small buckets preserves more of the distribution than a handful of wide ones. Downsampling a gauge requires a choice: keep the last value, the minimum, the maximum, or the average. Keeping only the last value loses the extremes, which are usually the interesting part; keeping all four multiplies storage by four but preserves what the raw data contained.

The mistake that downsampling invites is treating a downsampled average as if it were a raw value. Averaging the downsampled averages over a longer window is the mean-of-means problem in another guise, and it produces the same weighting error. The fix is the same: downsample to sums and counts, not to averages, and divide at the end.

Choosing what to materialise

The decision rule that follows from the costs above has three parts. Materialise expressions whose evaluation cost is high relative to their read frequency, since that is where the saving is. Materialise expressions whose result is needed for alerting, since an alert that depends on a query timing out is an alert that fails at the moment it matters. Do not materialise expressions that are cheap, rarely read, or exploratory, because the flexibility of evaluating against raw series is worth more than the saved work. The cost of that flexibility is dominated by selection, since the matchers in the expression decide how many series the engine must touch before any arithmetic happens, which is the subject of Label Matchers, Posting Lists, and How a Series Is Selected.

The last point is the one most often ignored. An exploratory query is one whose shape is not known in advance, and a pre-aggregated series can only answer the shapes that were anticipated. Keeping raw data long enough to explore is a retention cost; keeping only pre-aggregated data is a capability cost. Both are real, and the balance is a decision about which questions the system will be able to answer later, not about which queries are slow now.