Averaging Ratios and the Geometric Mean in Composite Scores

Reported performance is nearly always a dimensionless ratio: this version against that version, this machine against a reference, this build against the previous release. Once several ratios exist, someone averages them into one number, and the choice of average determines whether the number means anything. The arithmetic mean is the reflex and is wrong for this data type; the geometric mean is the correct summary for normalised ratios, and the harmonic mean appears when the quantities being averaged are times that must sum rather than factors that multiply.

Ratios live on a multiplicative scale

A speedup is not an amount of anything. Doubling the speed is a factor of two, and halving it is a factor of one half, and the two are symmetric in the sense that they compose to one: applying one and then the other returns to the starting point. The natural centre of such values is a value whose product with its counterpart is one, which is what the geometric mean computes.

The arithmetic mean of the two, $1.25$, corresponds to no operation that can be performed. It is the result of treating the two factors as distances and taking the midpoint, which is a different model of the quantity. On the log scale the geometric mean is the arithmetic mean of the logs, and the symmetry becomes visible: $\log 2$ and $\log 0.5$ are negatives of each other, and their mean is zero, exponentiating to one.

The consequence for reporting is that percentage changes are asymmetric. A reduction of 50 percent and an increase of 100 percent are the same change of factor two in opposite directions, so a report that averages percentage changes arithmetically weights the two differently depending on which direction it went. Averages of ratios must be computed on the ratio scale.

The reference machine is a choice, and it should not change the answer

The reason the geometric mean is prescribed for normalised benchmark results rather than merely preferred is invariance. A suite measures several workloads on two machines and reports the ratio of each machine’s total time to a reference machine’s time on the same workload. Those normalised ratios are then averaged into a single figure.

If the average is arithmetic, the figure depends on which machine is chosen as the reference. Switching the reference from machine $R$ to machine $S$ multiplies every normalised ratio by the same factor per workload, and the arithmetic mean of the altered ratios is not the original mean times that factor, because the factor differs per workload and the mean of a product of two series is not the product of the means. The reported relative performance of two machines therefore changes if the third, reference machine is swapped, which makes the comparison meaningless as a statement about the two machines.

The geometric mean has the property that scaling every input by a per-workload factor scales the result by the geometric mean of those factors, so the ranking of two candidate machines relative to each other is preserved under a change of reference. That is the property required of a normalised summary, and it is what the standard reporting rules for benchmark suites enforce.

To see the failure concretely, consider two workloads and two machines. Machine A is twice as fast as the reference on the first workload and half as fast on the second. The arithmetic mean of the ratios is $1.25$, which claims A is 25 percent faster overall; the geometric mean is $1$, which claims parity. If the reference is replaced by A, the same two machines compare with ratios that would, under an arithmetic average, give B an apparent advantage, because the workload where B is twice as fast now dominates the mean. The geometric mean is unchanged by the substitution.

When speedup is not the question, rates force the harmonic mean

Ratios of time are not the only aggregate quantity. Throughput, a rate of completed work, is the natural metric for a server, and its aggregation follows different arithmetic.

The relevant identity is that the average rate over a set of intervals is total work divided by total time. If the intervals carry equal amounts of work but take different times, the average rate is the reciprocal of the average of the reciprocals of the individual rates, which is the harmonic mean. Averaging rates arithmetically overstates the total, because the arithmetic mean weights each interval equally in rate while the total weights it by the time it consumed: a period of very high throughput that lasted a millisecond cannot compensate for a period of low throughput that lasted a second.

The relationship between the means is ordered: for positive values the harmonic mean is at most the geometric mean, which is at most the arithmetic mean, with equality only when all values are equal. The size of the gap is a measure of how heterogeneous the data is, so a comparison that reports one mean for a heterogeneous throughput series is reporting a quantity whose value depends on an unstated weighting choice.

The same distinction appears when latency figures are combined across operations of unequal cost. The mean latency of a request mix is a weighted average of the per-operation latencies with weights proportional to how often each operation occurs, and a change that improves an operation composing one percent of traffic shows almost nothing in the aggregate. Reporting the per-operation figures as well is often more useful than reporting the aggregate, because the aggregate’s weights are a property of the workload rather than of the change.

import math
from statistics import fmean, harmonic_mean

ratios = [2.0, 0.5, 1.4, 0.9]
print(fmean(ratios))                       # 1.2  — not a meaningful summary
print(math.exp(fmean([math.log(r) for r in ratios])))   # geometric mean
print(harmonic_mean([1.0, 2.0, 4.0]))     # harmonic mean of rates

Aggregating a score hides the distribution underneath it

A composite score built from per-workload ratios compresses a vector into a scalar, and the compression is not information-preserving. Two machines can share a geometric mean while differing in variance across workloads: one is uniformly a few percent faster, the other is twice as fast on half the workloads and half as fast on the rest. The geometric mean is the same and the engineering consequences are not, because the second machine’s profile makes it a poor choice for a mixed workload and an excellent one for a specialised one.

The reporting convention that addresses this pairs the aggregate with per-workload results and with the reference used. This is why benchmark suites publish both the summary score and the individual ratios, and why the reporting rules constrain how the summary may be computed: a score without its components is a claim about a distribution made from a single number.

There is also a statistical issue with the aggregate. Each workload’s ratio is estimated with error, and the error is multiplicative when expressed as a factor, so the natural analysis is in log space. The average of the logs has an interval that follows from the per-workload standard errors, and the interval on the aggregate is formed by exponentiating the endpoints of that interval. Averaging the ratios first and forming an interval afterwards assumes the per-workload errors are additive and symmetric, which they are not, and produces an interval that understates the uncertainty whenever the workloads differ in repeatability.

The geometric mean estimator has a small bias of its own when the log values are noisy: the estimate is biased low by a factor depending on the variance of the logs. With a handful of workloads whose per-run variation is a few percent, the bias is far below the interval width and can be ignored; with a single randomised measurement per workload, it cannot.

Reporting the aggregate without losing the conditions

Three conditions determine whether a composite figure can be compared with another. The reference set, because normalisation carries it. The aggregation function, because the arithmetic, geometric, and harmonic means give different numbers for the same inputs. And the workload weights, because equal weighting of workloads is a statement about their importance rather than a neutral default.

A figure that omits any of these is comparable only to figures computed the same way, which in practice means it is comparable only to itself. Writing the three conditions next to the number costs one line and removes most of the ways a comparison can be wrong.

The remaining question is whether the workloads in the aggregate resemble the workload the change will meet. A benchmark that measures a distribution server on uniform random keys will not predict the effect of a change on a range-scan-heavy client, no matter how carefully the ratios are averaged. Selecting workloads is a sampling problem, and its errors are not of the kind that a better estimator can fix. External Validity and the Limits of What a Benchmark Shows takes up that question, and the sampling error of the per-workload ratios that feed the aggregate is the finite-sample problem described in Quantiles from Finite Samples and Their Confidence Bands.

Where a speedup larger than one changes the arithmetic

Reported speedups above one and below one are not symmetric in the arithmetic of the summary. A factor of 1.1 and a factor of 0.9 have a geometric mean of approximately $0.995$, which is not one, so a change that is “10 percent faster” on one test and “10 percent slower” on another is, in geometric terms, slightly worse overall. This asymmetry is a feature of the scale rather than a defect of the estimator, and it means that the conventional rounding of small effects to the nearest percent can accumulate into a visible tilt in an aggregate over many workloads. Reporting the aggregate with enough precision to see a fraction of a percent avoids introducing one.