Aggregation Laws: Which Metric Operations Commute

Aggregation is the operation that turns many series into few, and it is where most incorrect dashboards are born. The reason is that aggregation operators do not all behave the same way when applied in different orders, and the difference is not a rounding error. Two paths through the same data can differ by a factor that depends on the group sizes, so the error is not even stable across time.

The property that separates the safe operators from the unsafe ones is a form of associativity: whether aggregating subgroups and then aggregating the results gives the same answer as aggregating everything at once. Sums, counts, minimums and maximums have it. Averages do not. Percentiles do not. Ratios do not. Everything else follows from that single distinction.

The operators that survive regrouping

A sum is associative and commutative over real numbers, so $\sum_i \sum_{j \in G_i} x_j = \sum_k x_k$ regardless of how the groups are chosen. Counts behave identically. Minimum and maximum are associative and commutative too, since the minimum of a set of minima is the minimum of the union.

These four are the operators that can be pushed to any point in the pipeline without changing the result. A collector can pre-aggregate per host, the query layer can sum across hosts, and a recording rule can sum across regions, and the answer is the same as a single pass over everything. This is what makes hierarchical aggregation feasible at all: without it, every query would have to reach raw series.

There is a caveat on floating-point sums that has nothing to do with associativity in the mathematical sense. Addition of float64 values is not associative in the machine sense, because each partial sum is rounded to 53 bits of mantissa. Regrouping a long sum of values with very different magnitudes can change the low-order bits of the result. For counters that increment by one, the values are small integers and the effect is invisible until the total exceeds $2^{53}$, as noted in Counter Resets and the Extrapolation Inside rate(). For gauges holding large values, the difference between summation orders can be visible at the seventh significant digit, which matters only if something downstream compares two such sums for equality.

Why the mean of means is a different number

An average is a ratio: the sum of the values divided by the count. It is a sum of one quantity divided by a sum of another. Grouping does not commute with that division, because the denominator differs between groups.

Three instances serving 100, 100, and 900 requests with average latencies of 10, 10, and 100 milliseconds illustrate the difference. The true mean over all 1,100 requests is $(100 \cdot 10 + 100 \cdot 10 + 900 \cdot 100)/1100 \approx 83.6$ milliseconds. The mean of the three per-instance means is $(10 + 10 + 100)/3 = 40$ milliseconds. The second number is less than half the first, and the direction of the error is determined entirely by which instance carried the traffic: the two fast instances had a tenth of the requests of the slow one, and averaging their means gave them equal weight.

The correct computation keeps the two sums separate until the end. If the data available is per-instance sums and counts, the ratio of the summed sums to the summed counts is exact. If only per-instance means are available, the information needed to weight them has been destroyed and no arithmetic recovers it. This is the mechanism behind the familiar rule that a ratio metric should be aggregated as a ratio of sums, never as an average of ratios. The argument is the same one, applied to two different instruments instead of one.

The same structure appears in error rates. A service with 1,000 requests and 10 errors has a one percent error rate; a service with 10 requests and 5 errors has a fifty percent error rate. The mean of those two rates is 25.5 percent. The true error rate over both is $15/1010 \approx 1.5$ percent. The mean-of-rates number is wrong by more than an order of magnitude, and it moves whenever the low-traffic service’s request count changes, which makes it useless as an alerting signal. The alerting consequences of that instability are the subject of Delta and Cumulative Temporality in Collection Pipelines.

Rate does not distribute over sum

The intuition that a rate is a linear operator suggests that the rate of a sum equals the sum of the rates. For counters with no resets, that is true: the sum of the increments equals the increment of the sums, and the extrapolation factors are the same when the samples share timestamps.

With resets it fails, and the failure is structural. The correction adds the value observed immediately before the drop, on the assumption that this entire value was accumulated and then lost. For a single series that assumption is right. For a summed series, the pre-drop value includes every component, and only the components that actually restarted lost anything. The correction therefore credits the sum with the pre-reset totals of series that never reset.

Three samples make the arithmetic concrete. Series A reads 100, then restarts and reads 5, then climbs to 15. Series B reads 100, holds at 100, then restarts and reads 20. The summed series reads 200, 105, 120. It decreases once, from 200 to 105, so the correction adds 200 and the computed increment is $120 - 200 + 200 = 120$.

Corrected per series, A yields $15 - 100 + 100 = 15$ and B yields $20 - 100 + 100 = 20$, for a total of 35. That is the true increment: A accumulated 5 before the second sample and 10 more after, and B accumulated 20 after its own restart. The summed path overcounts by 85, which is exactly B’s pre-restart value of 100 minus the 15 that A had legitimately accumulated. The algorithm cannot tell which of the two series caused the drop, so it credits both.

The practical rule is to rate before summing when resets are possible, and to sum before rating only when the series are known to be free of resets, which is rarely knowable. The cost is that the per-series rate must be computed over aligned windows, and series that were absent for part of the window contribute nothing, which biases the sum downward in exactly the way extrapolation was designed to avoid.

Percentiles and the shape of the error

A percentile over a pooled set is not derivable from per-group percentiles, for the same reason the mean is not: the rank position of the $q$-th value depends on all the counts below it, not on each group’s own $q$-th value. Averaging per-group 99th percentiles produces a number that is not the 99th percentile of anything and can be arbitrarily far from it. A single group with a pathological tail drags the average up even if it holds a tiny fraction of the traffic.

Histograms sidestep this because they carry the counts, and the pooled histogram can be built by adding bucket counts, as described in Histogram Buckets, the +Inf Bucket, and How Quantiles Are Interpolated. Summaries do not, and their per-group quantiles cannot be merged at all. That asymmetry is the main practical argument for histograms in any system that aggregates across instances.

Aggregating across time is a separate question

Everything above concerns aggregation across series at one instant. Aggregating across time, which means averaging a per-minute series to get an hourly figure, has the same trap in a different form. A counter’s per-minute rate averaged over an hour weights each minute equally regardless of how many requests occurred in it. If traffic is concentrated in ten minutes of the hour, the hourly average rate understates the rate during those minutes and overstates it during the quiet ones, and neither number is the hourly rate.

The fix is the same: keep sums and counts, divide at the end. A rate averaged over time is only meaningful when the underlying quantity is itself a rate of a stable process, and even then the average is not the same statistic as the rate computed over the whole hour, because it ignores the distribution of the increments within the hour.

Where the regrouping happens matters more than the operator

Because the unsafe operators are unsafe, the pipeline design has to decide where each one runs. The safe operators can be evaluated early, in the collector or in a recording rule, and the unsafe ones must be evaluated last, at the point where the final answer is needed, from components that have been carried through the pipeline separately.

Concretely, that means carrying sums and counts through the pipeline and computing averages, ratios and percentiles only in the query. A recording rule that stores an average has discarded the weighting information permanently, and every downstream consumer inherits the error. A recording rule that stores a sum and a count costs twice the series but is correct for every consumer, which is usually the right trade: the storage cost is linear and predictable, while the correctness cost of the alternative is unbounded and invisible.

The remaining constraint is that the two components must be aggregated over the same groups. A sum grouped by instance and a count grouped by status cannot be divided, because the grouping is not aligned; the ratio would pair unrelated subsets. Aligning the label sets before the final division is the step where this design most often breaks in practice, and it fails silently: the division produces a number, and the number looks plausible.