Data Augmentation as an Invariance Specification

A contrastive objective needs two representations of the same input that differ enough to make the comparison non-trivial, and it needs the difference to be one the model should ignore. That second requirement is where augmentation enters. An augmentation pipeline is a claim about the data: it asserts that a crop, a colour shift, or a small rotation does not change the identity of the thing being represented. The training procedure then enforces the claim by penalizing any representation difference between the augmented pair.

The claim is rarely stated explicitly, which is why augmentation choices cause so much trouble downstream. A pipeline built for object recognition learns to discard colour; a pipeline built for scene classification may need colour. The same encoder architecture, trained with different augmentations, produces spaces that answer different questions.

The augmentation set defines an equivalence class

Formally, the training objective treats every pair produced by the augmentation distribution as the same point. The representation is therefore constrained to be constant on each orbit of the augmentation group, at least approximately. If the group includes horizontal flips, the representation of a left-facing object and a mirrored right-facing one must coincide. If it includes colour jitter, luminance and hue are removed from the representation.

This is a strong statement about what the downstream task can be. A linear classifier reading the representation cannot recover information the encoder was trained to discard, because the discarded information is precisely the direction the objective minimized. A representation trained with aggressive colour jitter cannot support a task that depends on distinguishing a ripe fruit from an unripe one, no matter how much labelled data the classifier sees.

The converse also holds, and it is the useful direction. An augmentation that is not in the set leaves the corresponding variation free, and the encoder will often preserve it because preserving it costs nothing. A pipeline with only random crops and no flips produces a representation in which orientation is still encoded. This means augmentation design is a way of allocating capacity: the encoder spends its dimensions on variation that the augmentations do not remove.

Composition matters more than any single transform. A crop followed by a colour jitter followed by a blur produces a pair whose differences span several factors at once. The model cannot tell which factor was responsible for a given difference, so it learns to be invariant to the combination. Augmentations that are individually mild can compose into a pair that no longer looks like the same object, at which point the positive pair is wrong and the loss pushes apart things that should be close. Cropping two different regions of a large image can produce two views with no shared content; the standard mitigation is to constrain the crop scale so that the overlap is guaranteed.

Which transforms are safe depends on the label space

For photographic object recognition, the standard set is well established: random resized crop, horizontal flip, colour jitter, grayscale conversion, Gaussian blur, and sometimes solarization. Each of these leaves the object category unchanged for the categories in typical benchmarks. The set is not universal. Vertical flips are excluded because gravity gives images a canonical orientation and an upside-down dog is not a typical dog photograph, but for aerial imagery or microscopy the argument reverses and vertical flips become safe.

import torch
import torchvision.transforms as T

# Invariance claim: object identity survives crop, flip, and colour change.
# It does not survive a 180-degree rotation, so rotation is excluded.
augment = T.Compose([
    T.RandomResizedCrop(224, scale=(0.2, 1.0)),   # scale floor bounds overlap
    T.RandomHorizontalFlip(p=0.5),
    T.RandomApply([T.ColorJitter(0.4, 0.4, 0.4, 0.1)], p=0.8),
    T.RandomGrayscale(p=0.2),
    T.RandomApply([T.GaussianBlur(kernel_size=23)], p=0.5),
])

The same reasoning applied to other domains produces very different pipelines. For audio, the safe transforms are time masking, frequency masking, and small time shifts, because a sound event is roughly stationary and its identity does not depend on a few masked frames. For text, the safe transforms are back-translation, token deletion, and span substitution, and even these are contested because deleting a negation token changes meaning. For molecular graphs, the safe transforms are node order permutation and small bond perturbations, because the graph’s identity is invariant to the ordering of its adjacency list.

The pattern is that safety is a property of the label, not of the data type. A transform is safe when the label is provably unchanged under it for the population the model will see. When that cannot be argued, the transform introduces label noise into the positive pairs.

Asymmetric and information-asymmetric views

Two views need not be augmented the same way. Several designs use a stronger pipeline for one branch and a weaker one for the other, so that one view is closer to the original and the other is heavily distorted. The pairing then asks the model to map a distorted view onto a cleaner one, which is a harder and sometimes more useful task than matching two equally distorted views.

The information asymmetry view formalizes this. If one view retains more task-relevant information than the other, the shared information between them is bounded by the poorer view, and the representation is forced to capture only what survives the distortion. Making both views equally aggressive lowers that bound; making one view mild raises it but risks letting the model solve the task by copying rather than abstracting. The useful regime is the one where the aggressive view still contains the label but little else.

Multi-crop designs extend this by producing several small crops and one or two large ones per image, then computing the loss between the large views and each of the small ones. The small crops cover a fraction of the image, so their overlap with each other may be zero, but each overlaps the large view. This increases the number of positive pairs per image without increasing the batch of distinct images, which raises the effective number of comparisons at a fixed memory cost.

Augmentation strength is a schedule, not a constant

The magnitude of a transform changes the difficulty of the positive pair. A crop covering 80 percent of the image produces a view almost identical to the original; a crop covering 10 percent produces a view that may show only a fragment. Training with the aggressive setting from the start presents pairs the encoder cannot yet match, and the gradient is dominated by the impossibility rather than by the learnable structure. Several training recipes therefore ramp the augmentation strength over the first portion of training, starting with mild distortions and increasing them.

The ramp also interacts with the temperature. A stronger augmentation produces a larger expected similarity gap between positives and negatives, which changes the appropriate softmax sharpness; the two parameters are usually tuned together, and a recipe that changes one without the other often performs worse than either setting alone. The reason the softmax sharpness is coupled to the pair distribution is set out in InfoNCE and the Temperature Parameter.

When the invariance claim is false

The most common failure is a positive pair that is not actually the same object. Aggressive cropping of images with small objects, or of images containing several objects, produces views that show different things. The loss treats them as positives, so the encoder is pushed to make unrelated content similar, which blurs the space. Detecting this requires looking at the pairs, not at the loss.

The second is shortcut learning through a transform that leaves a trace. If the augmentation is implemented with a fixed border fill or a fixed interpolation kernel, the encoder can identify which view was augmented and how, and use that as a feature. Pairs generated by the same code path share implementation artefacts, and the model can learn to match on the artefact rather than on content. Randomizing the fill value and the kernel, and mixing augmentation orders, reduces the opportunity.

The third is the mismatch between pretraining and deployment. A representation trained to be invariant to crops has no notion of absolute position; a downstream task that needs to localize an object must relearn position from the features, and the crop invariance actively fights that. This is not a defect of the pretraining, but it is a constraint on which downstream tasks the representation suits, and it is the reason the choice of evaluation protocol in Linear Probing and Fine-Tuning as Different Measurements changes the ranking of pretrained models.

Where augmentations cannot be defined, because the domain has no safe transformations, the alternative is to construct views by other means: different modalities of the same event, different sensors, or different segments of a time series. Those constructions carry the same obligation, which is to argue that the shared information contains the target and that the differences are irrelevant to it.