Collapse and Redundancy Reduction in Self-Supervised Training

Any objective that rewards agreement between two views of the same input has a trivial optimum: map everything to the same point. The representations then agree perfectly, the loss is at its minimum, and the encoder carries no information. Avoiding this is the central design problem of self-supervised representation learning, and the solutions fall into two families. One prevents collapse by making the two branches asymmetric. The other adds a term that penalizes a degenerate distribution of the representations themselves.

The two families produce different geometries and different costs. Choosing between them is a decision about what property of the space is being enforced, not just about which loss converges.

Two kinds of collapse

Constant collapse is the easy case to describe: every output is the same vector. The representation matrix over a batch has rank one, and its covariance is a rank-one matrix with a single nonzero eigenvalue.

Dimensional collapse is harder to see because the outputs are not identical. The representations vary, but they vary within a low-dimensional subspace. A batch of a thousand vectors might occupy a ten-dimensional subspace while nominally living in 512 dimensions. Nearest-neighbour retrieval still returns varied results, and a linear probe may still reach reasonable accuracy on tasks whose signal lies in the surviving subspace, so the failure is easy to miss until a task that needs the missing directions is attempted.

Dimensional collapse is what a redundancy-reduction objective is designed to detect. Constant collapse is detectable from the variance of the batch alone. Both show up as a covariance matrix whose spectrum is concentrated, and the difference is only in how many eigenvalues are near zero.

Asymmetric architectures as a structural fix

Methods that use a momentum target with a stop-gradient avoid constant collapse because the target branch is not optimized to agree with the anchor; the anchor is optimized to agree with a target that does not move toward it. The asymmetry is described in Momentum Encoders and Stop-Gradients and it is sufficient to prevent the constant solution from being an attractor.

It does not prevent dimensional collapse. A slow target can converge to a low-rank function just as an online branch can, and the stop-gradient only removes the specific incentive to map everything to one point. Methods in this family typically add a second mechanism to keep the rank up: a centring operation that subtracts a running mean of the target outputs, a sharpening operation that divides by a temperature, or both. Centring prevents one dimension from dominating, because the mean is removed and the remaining variation must be distributed elsewhere. Sharpening counteracts the uniformizing effect of centring.

The combination is delicate. Centring with a coefficient that is too large removes too much of the signal and the branch learns to predict the mean, which is constant collapse by another route. Sharpening with a coefficient that is too small produces a uniform target distribution and the branch learns to match noise. The two coefficients are usually tuned together, and the pair is not transferable across architectures without adjustment.

Redundancy reduction on the cross-correlation matrix

The alternative is to constrain the batch’s statistics directly. Take the representations of two views, normalize them along the batch dimension, and compute the cross-correlation matrix between the two sets of features:

C_ij = Σ_b z^A_{b,i} · z^B_{b,j} / (B · σ_i · σ_j)

where b indexes the batch and i, j index feature dimensions. The matrix has shape d × d and each entry measures how much feature i of one view covaries with feature j of the other.

The objective has two terms. The diagonal of C should be one: each feature of one view should correlate with the same feature of the other view, which enforces invariance. The off-diagonal entries should be zero: distinct features should carry distinct information, which enforces redundancy reduction. The loss is the sum of the squared deviations from that target.

def redundancy_loss(z1, z2, lam=0.005):
    B, d = z1.shape
    z1 = (z1 - z1.mean(0)) / z1.std(0)
    z2 = (z2 - z2.mean(0)) / z2.std(0)
    c = z1.T @ z2 / B                      # (d, d) cross-correlation
    on_diag = (c.diagonal() - 1).pow(2).sum()
    off_diag = c.pow(2).sum() - c.diagonal().pow(2).sum()
    return on_diag + lam * off_diag

The diagonal term is what prevents collapse: if all features were constant, the standard deviation would be zero and the normalization undefined, and if all features were copies of one another, the diagonal could still be one but the off-diagonal would be large. The off-diagonal term is what prevents dimensional collapse: it penalizes any two feature dimensions for carrying the same information.

The weight lam sets the balance. A small value lets the encoder satisfy invariance while tolerating correlated features; a large value forces near-diagonal correlation and pushes the representation toward an orthogonal basis. The published default for this family is small, on the order of a few thousandths, because the off-diagonal term sums over d² entries while the diagonal sums over d, so an unweighted sum would be dominated by the off-diagonal.

Whitening as an explicit operation

A third approach applies whitening to the batch of representations, transforming them so that their covariance is the identity. The transform is a matrix square root of the inverse covariance, computed from the batch or maintained as a running estimate. After whitening, the features are decorrelated by construction and have unit variance, so both collapse modes are excluded.

The cost is the eigendecomposition of a d × d covariance matrix. For d = 2048, that is a few million floating-point operations per batch, small relative to the encoder’s forward pass but not negligible, and the operation is not well supported in reduced precision. Distributed training complicates it further: the covariance must be computed over the global batch, which requires an all-reduce of the d × d matrix, so the communication volume is quadratic in the feature width.

Whitening also changes the gradient path. Because the transform depends on the batch’s statistics, the gradient of the loss with respect to a representation includes a term from its effect on the covariance. That term couples all items in the batch, which makes per-sample gradient computation impossible and rules out gradient accumulation schemes that assume independence.

What each approach costs

The asymmetric approach costs a second copy of the model and a second forward pass, plus whatever statistics the centring operation maintains. It has no d × d matrix and no eigendecomposition, so it scales to wide representations without extra cost in the width. Its weakness is that the collapse prevention is indirect: it excludes one degenerate solution and relies on additional terms to exclude others.

The correlation approach costs a d × d matrix multiply per step, which is B · d² operations, and for a batch of 1024 and a width of 2048 that is about 4.3 billion multiply-adds, comparable to a fraction of a forward pass through a moderate encoder. It also materializes a d × d matrix, which at 32 bits is 16 MB for that width. The objective is transparent about what it enforces and does not need a momentum branch.

The whitening approach costs an eigendecomposition and a global reduction, and it makes the loss depend on the whole batch in a way that complicates distributed execution. Its advantage is that the resulting space has a known spectrum, which simplifies reasoning about downstream linear separability.

Degeneracies the statistics do not exclude

A redundancy objective can be satisfied by making the features noisy. If each dimension carries independent noise and no signal, the cross-correlation is diagonal and the off-diagonal is zero, so the loss is low while the representation is useless. The diagonal term is supposed to prevent this by requiring correlation between views, but a weak diagonal weight allows the noise solution to dominate.

Whitening can amplify directions that carry almost no variance. The inverse square root of a covariance with a tiny eigenvalue is large, so a direction with negligible signal is scaled up to unit variance. The result is a representation in which noise directions have the same magnitude as signal directions. Regularizing the covariance toward the identity, which bounds the inverse, is the standard mitigation.

Both approaches interact badly with batch normalization. If the encoder contains batch-norm layers, the statistics used for normalization and the statistics used for the decorrelation term are computed over the same batch, and the two can fight. Removing batch norm from the projection head, or replacing it with layer norm, is common in these designs.

Finally, the two collapse modes are not the only degeneracies. A space can be full-rank and still uninformative if the rank is spread over directions that are functions of the input’s nuisance attributes. Rank is a necessary condition for a useful representation, not a sufficient one. What the space actually encodes is set by the augmentations and the negative distribution, as Data Augmentation as an Invariance Specification and False Negatives and Batch Composition describe, and the spread of the resulting directions is what Alignment and Uniformity as Separate Objectives measures.