One-Hot Encoding as Geometry and the Dummy Variable Trap

Encode a categorical column with k distinct values as k indicator columns, one per level, each equal to 1 where the row has that level and 0 elsewhere. The construction looks like a formality. It changes the rank of the design matrix, it changes the interpretation of every coefficient in the model, and it decides whether the fitted parameters are unique.

The indicator columns span k-1 dimensions, not k

Consider one row’s encoding: a vector with a single 1 in the position of its level. The sum of the k indicator columns is the all-ones vector at every row, without exception, because exactly one indicator is 1. So the k columns are linearly dependent: their sum equals the intercept column if one is present. The span of the k indicators has dimension k - 1; the levels contribute k - 1 degrees of freedom to the model, exactly as the counting intuition suggests, since a factor with k levels carries information about k - 1 independent comparisons.

The redundancy is harmless in the population and fatal in the algebra. Write the design matrix as X with n rows and k + 1 columns, consisting of an intercept and k indicators. Its rank is k, so the Gram matrix X^T X is singular and the ordinary least squares solution w = (X^T X)^(-1) X^T y does not exist. What exists is the set of solutions: the fitted values are unique, because the column space does not care about the dependence, but the coefficients are not. Adding a constant c to every level’s coefficient and subtracting k * c from the intercept leaves all predictions unchanged. That is the dummy variable trap, named for the pivot-table practice of summing rows; it is a rank deficiency, not an error in the encoding.

In exact arithmetic, a least squares solver will either pick one particular solution (the minimum-norm one, if it uses a pseudoinverse or a QR factorisation with a rank-revealing pivot) or fail. In floating point, the outcome depends on the LAPACK routine and the data: a singular or near-singular normal-equation solve may return coefficients of magnitude around 10^8 with alternating signs that cancel to produce reasonable predictions. The model is not broken; the coefficient table is meaningless.

Two conventions and what each parameter means

The standard fix is to drop one level, conventionally the first in sorted order, producing k - 1 columns. The dropped level becomes the reference, and each remaining coefficient is the difference in expected response between that level and the reference, holding the other predictors fixed. The intercept is the reference level’s mean. This is the treatment contrast used in every analysis-of-variance table.

The alternative is to keep all k columns and omit the intercept. Now each level’s coefficient is that level’s mean, with no reference. For a single categorical predictor the two parameterisations give identical fitted values and identical residual sums of squares; only the parameterisation differs. With several factors the same holds, and the choice becomes a matter of which contrasts are of interest rather than correctness.

Formally the two are related by a change of basis. Let Z be the k by k - 1 matrix whose column j - 1 is the indicator of level j minus the indicator of level 1, for j from 2 to k; then the treatment-coded design is the full indicator design times Z, combined with the intercept. Because the transformation is invertible on the relevant subspace, everything that is invariant to reparameterisation (predictions, residuals, deviance, standard errors of fitted values) is preserved, and everything that is not (individual coefficients, their standard errors, their t-statistics) is not.

import numpy as np

def treatment_encode(levels, categories=None):
    """Return a (n, k-1) float matrix, dropping the first level as reference."""
    cats = sorted(set(levels)) if categories is None else list(categories)
    ref, rest = cats[0], cats[1:]
    index = {c: i for i, c in enumerate(rest)}
    out = np.zeros((len(levels), len(rest)))
    for row, level in enumerate(levels):
        if level != ref:
            out[row, index[level]] = 1.0
    return out, cats

The reference level is arbitrary but not without consequence: the width of the confidence interval on the intercept depends on how many rows share the reference level, so a reference that is rare produces an imprecise baseline and, through the covariance between the intercept and the contrasts, imprecise differences involving it. Choosing the largest group or the control condition as the reference is a statement about which comparisons matter, not a neutral default.

What the penalty does to a dropped level

For an unpenalised linear model the choice between k and k - 1 columns is irrelevant in the sense above. Under an L2 penalty it is not. Penalised least squares minimises the squared residual norm plus lambda times the squared coefficient norm, and the penalty is not invariant to the reparameterisation, because the two designs use different coefficient vectors for the same fit. With all k indicators, the redundant direction (the constant shift c applied to all levels) carries a penalty proportional to k * c^2, so the optimiser shrinks the level effects toward each other; that shrinkage is what makes the solution unique, and it is a real prior, not an artefact. With k - 1 indicators, no such direction exists and the penalty acts within the k - 1-dimensional subspace. The fitted models differ whenever lambda > 0, and the all-levels variant is equivalent to an extra random-effects-style penalty that pulls all levels toward their common mean.

The general statement is that an L2 penalty restores uniqueness for any rank-deficient design: adding lambda times the identity to X^T X makes it positive definite, so the solution exists. That is convenient, and it hides the problem in the cases where it matters. A penalised model trained on a leaking design will fit fine and report sane coefficients while the rank deficiency silently absorbs one of its degrees of freedom. Diagnostics that inspect the Gram matrix’s condition number are the reliable check, and they need to run on the unpenalised design.

Trees and neural nets read indicators differently

Axis-aligned trees are unaffected by the reference choice in their split structure, because a split on an indicator column is a test of set membership and dropping one level just means that the reference’s rows fall to the left branch by default. The redundancy appears instead as a cost: k binary columns require the split search to scan k features at every node, and for a column with thousands of levels, high-cardinality categories produce splits whose support is a handful of rows. Those splits fit noise and generalise badly, which is the reason tree libraries often treat categorical columns natively rather than accept the expanded form. The combination of one-hot columns with a boosting model is discussed further alongside out-of-fold target encoding, the alternative that trades this split cost for a leakage risk.

Neural networks and other gradient-trained models see the indicators as coordinates in a k-dimensional space and the model’s first layer as a linear map into a lower dimension. Here the redundant direction is genuinely useless, and dropping it costs nothing. Keeping it adds one row to the first weight matrix, and because the redundant direction has non-zero gradient, it doubles the effective step in that direction: the parameter count grows by one while the functional capacity does not. The cleaner treatment for categorical inputs is not one-hot-but-dropped but a learned embedding, which replaces the k indicators with a small dense vector per level and removes the geometry problem entirely.

Cardinality, memory, and the collision alternative

The encoding scales with the number of distinct values. A column with 50,000 levels encodes to a dense matrix with 50,000 columns, which at eight bytes per entry is 400 kilobytes per row. Sparse storage reduces this to one non-zero per row, but many downstream algorithms densify anyway, and the dense design is O(n * k) memory with a rank of at most k. When k approaches n, the model can interpolate the training labels from a single column, and any split or coefficient on that column will not generalise. Rare levels make the situation worse by consuming capacity without repeats.

Three strategies trade this against information. Grouping rare levels into a single “other” bucket caps the width, at the cost of merging distinctions that the data cannot support anyway; the standard practice is to keep levels above a count threshold and merge the rest. Frequency or count encoding replaces the indicator with the level’s observed frequency, giving a single column with no rank issue. Feature hashing maps levels to a fixed number of columns by a hash function, which bounds the width regardless of vocabulary and introduces controlled collisions, with the collision cost analysed in feature hashing collision bounds. Each of these discards the exact identity of the level in exchange for a boundary on the design’s rank and width, and the choice between them is a statement about how much the model can learn from a level that appears a handful of times.