Mean Imputation and the Variance It Removes

Suppose a column is missing in a fraction p of rows and the remaining values are a sample with mean mu and variance s^2. Replacing each missing value with mu is the cheapest imputation that exists: one pass to compute the mean, one pass to substitute. It preserves the sample mean exactly, since p * mu is contributed by the imputed rows and (1 - p) * mu by the observed ones, and the resulting column has mean mu. The variance of the imputed column, with the mean known, is (1 - p) * s^2. The other p * s^2 has been deleted from the data and replaced by a constant that carries no variation at all.

The arithmetic of the collapse

Write the observed fraction as 1 - p. Every second moment in the column is multiplied by 1 - p, not only the variance. The covariance between this column and another complete column is also multiplied by 1 - p if the missingness is independent of that column: the product of the two columns has zero variance on the imputed rows and its mean contribution is p * mu * E[z], which is exactly what a column with no variation would contribute. So mean imputation rescales both the diagonal and the off-diagonal of the covariance matrix by the same factor, and the correlation between the imputed column and any other column is unchanged.

That last fact is the reason the practice persists. If a single column is imputed and the model is a regression whose coefficient on that column is the ratio of a covariance to a variance, the coefficient is unchanged: both numerator and denominator shrink by 1 - p. Where mean imputation is confined to one column with missingness independent of everything, a linear model recovers the same slope it would have had on the complete cases, and this is a genuine invariance rather than an accident. The invariance fails immediately in the presence of several imputed columns, because the covariance between two imputed columns loses a share p_1 + p_2 - p_1 * p_2 when their missing rows are not the same rows, which is different from the shrinkage applied to each. The implied correlation between them rises toward one as more rows have both missing, and a model that relies on the difference between them sees a relationship that does not exist in the observed data.

import numpy as np

def mean_impute(train, other, missing):
    mu = np.nanmean(train, axis=0)
    def _fill(X):
        X = X.copy()
        idx = np.where(np.isnan(X))
        X[idx] = np.take(mu, idx[1])
        return X
    return _fill(train), _fill(other)

Note what is not in the function: a missingness indicator. The two rows whose value was substituted are indistinguishable from rows whose observed value happened to equal the mean, and the information that they were missing is gone. If missingness is related to the target, which is the case that matters, the information was relevant.

When missingness carries signal

The three classical descriptions of missingness are useful here. Data are missing completely at random when the probability of being missing is independent of everything, observed and unobserved. They are missing at random when the probability depends on the observed columns but not on the missing value itself. They are missing not at random when the probability depends on the value that would have been observed.

Mean imputation is an unbiased estimate of a column’s mean under the first two descriptions, and biased under the third. A sensor that saturates and reports nothing above its range is missing not at random: the imputed value is the mean of in-range readings, and it replaces readings that would have been high. A survey question about income has non-response concentrated at both tails, and imputing the mean pulls the distribution inward at both ends simultaneously. The measured mean is then not the population mean, whatever the algebra above says.

Under missing at random the observed values are a biased sample of the full column, conditional on the missingness mechanism. If the probability of missing depends on a complete column, then the mean of the observed values is not the unconditional mean; imputing it puts every missing row at a value that is systematically wrong in the direction of that column’s relationship to the imputed one. Multiple imputation models this dependence explicitly, and it is the reason the procedure exists rather than being a more elaborate form of filling.

What it does to a fitted linear model

For a regression with a single imputed predictor and independent missingness, the slope is preserved and the standard error is not. The residual variance is estimated with (1 - p) * n effective observations of variation, so the standard error of the slope is inflated by roughly 1 / sqrt(1 - p), and the degrees of freedom are counted as if every row were informative. A 40 percent missing column thus yields standard errors about 29 percent larger than the complete-case analysis, in a model that claims to use all n rows. The reported confidence intervals are too wide, which incorrectly suggests a model less precise than it is; more often the problem appears with several imputed columns, where the collapse is uneven and the standard errors can be too narrow as well.

The deeper problem is what happens downstream when the imputed column is used to build another feature. Dividing one column by another whose variance collapsed in a subset of rows produces a ratio with a spike at the value obtained by dividing by the mean. The distribution develops a discrete component at a point that no observed row would take, and tree models split on that spike, spending capacity on the artefact. The same spike affects any transform applied after imputation: standardising an imputed column divides by a standard deviation smaller than the true one, so the imputed rows sit at the centre of a wider standardised scale, and their influence on a model is diluted exactly where the imputation is least informative.

The indicator flag and what it preserves

The cheapest information-preserving change is to add a binary column that marks which rows were imputed, and to keep the imputed value in the original column. The pair separates two effects that the single imputed column conflates: the difference between having any value and having a particular one, and the value itself among those observed. For a model that can use both, the flag recovers the first effect and the imputed column serves as a placeholder for the second.

The flag is not a general solution. It becomes ambiguous when a column is missing as a function of another observed column, since then the flag duplicates that column’s information and inflates the design. It cannot recover the second effect, because the imputed value is still a constant within its group. And it is not free: one flag per imputed column doubles the number of columns carrying missingness, and with a dozen such columns the design grows faster than the information. On a system where 90 percent of one column is missing, the flag is effectively the whole feature and the value column is mostly constant, which is a usable representation only if the missingness is the phenomenon of interest.

Alternatives and their costs

Conditional imputation replaces the column mean with a prediction from the other columns, iteratively, which restores some of the deleted variance but only in the directions the other columns span: the imputed values fall on a lower-dimensional manifold, so the variance that a model can exploit in the imputed rows is exactly the variance already explained elsewhere. Iterative schemes such as those based on chained regressions converge toward a fixed point of the imputation equations, with no guarantee of uniqueness when columns are collinear, and they need a bounded number of rounds.

Adding noise to the imputed values restores the variance in expectation, which makes the single column look right while preserving the deterministic relationship to the other columns; the residual covariance between an imputed column and its predictors stays exactly zero, so the model sees the imputed rows as perfectly predictable from elsewhere. Multiple imputation is the principled version: draw several imputed datasets from a model of the missingness, fit the model on each, and combine the estimates with rules that account for both within- and between-imputation variability. The cost is the number of times the model is fitted and the need to combine parameters, which is straightforward for coefficients and awkward for anything that is not a moment, including tree ensembles and arbitrary metrics.

For prediction accuracy rather than inference, the empirical question is narrower: whether the missingness pattern is informative about the target. If it is, the flag alone is often worth more than any value-level imputation, and a model that handles missingness natively, such as a gradient-boosted tree with a default direction for missing values, learns the pattern from the flag without a separate column. The failure mode there is different and is covered with the variance and covariance mechanics described in scaling and gradient conditioning, since a column whose variance has collapsed is one whose standardised scale no longer reflects its information.

A final note on the mechanism’s provenance. Mean imputation survives because it is unbiased for the first moment and trivially cheap, and those two properties are enough for descriptive summaries. For a model that consumes second moments, it is a transformation that deletes a known share of the signal and leaves no trace of having done so. The bookkeeping required to be honest about it is a missingness indicator, a conditional model, or both.