Variance-Stabilising Transforms for Counts and Proportions

A count of events in a window has a variance that grows with its mean. A proportion bounded in the unit interval has a variance that shrinks toward both ends. Both facts are consequences of the generating process rather than of the sample, and both violate the equal-variance assumption that linear models, standard errors, and the residual structure of least squares silently rely on. A variance-stabilising transform is a function chosen so that, under the assumed generator, the transformed variable has approximately constant variance across its range.

The variance-stabilising principle

Let a variable have mean mu and variance g(mu). Apply a monotone transform f and approximate it near the mean by its tangent, with slope f'(mu). The delta method gives the transformed variance as g(mu) * f'(mu)^2. Setting this equal to a constant gives f'(mu) proportional to 1 / sqrt(g(mu)), and integration fixes f up to an affine map. One line of calculus generates the entire family, and each member is a statement about which generating process is believed.

For a Poisson variable g(mu) = mu, so f'(mu) is proportional to mu^(-1/2) and f(mu) = 2 * sqrt(mu). The factor of two is a convention, so the practical form is the square root of the count. The square root stabilises counts whose variance equals their mean, which covers independent event counts in a fixed exposure window, including photon arrivals and the counts of distinct n-grams in documents.

For a binomial proportion, an observed count over n trials, with variance p (1 - p) / n, the derivative condition gives f'(p) proportional to 1 / sqrt(p (1 - p)), and the integral is the arcsine of the square root of p, up to a factor of two. The arcsine square root is the classical transform for proportions, and its stabilising property holds best near the middle of the interval where the normal approximation to the binomial is decent.

For a quantity whose variance scales as the square of its mean, g(mu) = c * mu^2, the condition gives f'(mu) proportional to 1 / mu and f is the logarithm. That is the empirical justification for log-transforming positive measurements: money, durations, and sizes often have variation proportional to level. The one-parameter Box-Cox family, (x^lambda - 1) / lambda for nonzero lambda and log x at lambda = 0, interpolates between the reciprocal, logarithm, square root, and identity, and the parameter is estimated by maximising the profile likelihood of a normal model rather than by reasoning about the generator.

The logarithm and its unit problem

The log transform has three independent costs, and they are usually conflated.

The first is that it is undefined at zero. Adding a constant before taking the logarithm, conventionally log(x + 1) for counts, changes the transform’s curvature in the region of small values, which is exactly the region where the zeros and ones live. The choice of the constant is a modelling decision with consequences: for large x the transform is approximately log x, and at x = 0 the output jumps from 0 to log 2 within one unit of the input, an interval over which the untransformed variable has almost no dynamic range. A related construction, the inverse hyperbolic sine, log(x + sqrt(x^2 + 1)), is defined at zero and at negative values and behaves like log(2x) for large positive x; it is used for money and other signed quantities where the log would be undefined.

The second cost is interpretability. A coefficient in a regression on the logarithm of the response is a proportional effect, and a deviation in the log domain of 0.1 is roughly a 10.5 percent difference in the original units; the difference between log(x + 1) and log x coefficients is not constant across x, so a coefficient estimated on one scale cannot be converted to the other without the data. The back-transform of a mean is not the mean of the back-transform: the exponential of the expected log is not the expected value, and the gap is the exponentiated half-variance. Reporting an untransformed prediction requires the correction.

The third cost is the tail. The logarithm compresses large values, which stabilises variance driven by large values, but it also amplifies the leverage of values just above zero: a count of 1 against a count of 2 becomes a difference of 0.69 in the log domain, larger than the difference between 1,001 and 2,000. In a least squares fit the small counts, where measurement and discretisation error are proportionally largest, acquire the most influence. That inversion of the intended weighting is the reason the log is not automatically safe for sparse counts.

import numpy as np

def stabilise(x, kind, alpha=0.5):
    if kind == "sqrt":
        return np.sqrt(x)                       # Poisson: var = mean
    if kind == "log1p":
        return np.log1p(x)                      # multiplicative spread, zeros allowed
    if kind == "asinh":
        return np.arcsinh(x)                    # signed values, defined at 0
    if kind == "anscombe":
        return 2.0 * np.sqrt(x + 3.0 / 8.0)     # Poisson, better small-count behaviour
    raise ValueError(kind)

The Anscombe variant is worth the extra constant. For small Poisson means the plain square root is noticeably biased and its variance is not flat; adding three eighths before the square root and doubling the result reduces both errors for means down to roughly the single digits. For a column dominated by zeros and ones the improvement is small in absolute terms and large relative to the signal in that region.

Proportions and the edges

The arcsine square root applied to a proportion of zero gives zero and to a proportion of one gives pi/2; both are finite, which looks like an improvement over the log-odds, whose range is unbounded at both ends. But the variance at the edges is not stabilised by it: with n trials, a proportion of 0 has an estimated variance of 0 and the transform cannot invent the sampling error back. The usual remedy is to add a small count to numerator and denominator before forming the ratio, the same smoothing idea that appears in out-of-fold target encoding, and it is a bias-for-finiteness trade rather than a genuine stabilisation.

The logit is the transform that makes proportions linear in a logistic model, and it is a variance-stabilising transform only in a weak sense: under a logistic model the linear predictor enters through the logit, so the model’s own link function handles the mean-variance relation, and applying the logit to the response before fitting a least squares model is a different operation with different properties. Confusing the two produces models that are neither. Where a logistic regression is available, it uses the binomial variance function directly and no transform of the response is needed.

Models that do not care, and ones that do

Trees are invariant under any monotone transform of a single column, so log, square root, and arcsine produce identical splits and identical predictions up to the representation of the threshold. This is not an argument that the transform is neutral, because the transform changes what the model can express about interactions: a split on an untransformed count at threshold t and a split on its logarithm at log t are the same split, but a model that combines two columns multiplicatively is not preserved by transforming one of them. In a boosted ensemble, monotone transforms are absorbed into split points and essentially disappear.

Linear models, nearest-neighbour methods, and kernel machines care. In a linear model the response transform changes the error structure that least squares assumes, so the fit weights errors differently: a Poisson-inverse or a log-link generalised linear model fits the same data with the correct variance function without any transform of the input, and is strictly preferable when the goal is a regression on counts. When the goal is a feature for a downstream model rather than a response, the transform’s job is to make the column’s relationship to the target closer to additive and to stop a handful of large values from dominating the fit, and both effects are real even without a variance argument.

The choice between transforms is then a statement about the believed generator, and the diagnostic is straightforward: bin the column by value, compute the variance of the target within each bin, and plot it against the bin’s mean on a log-log axis. A slope near 1 supports the square root; a slope near 2 supports the logarithm; a slope near 0 means the variance is already flat and the transform buys nothing. The measured relationship is conditional on the model being wrong elsewhere, so the test is suggestive rather than definitive, but it is a great deal more informative than applying the transform that is conventional for the data’s type.