False Negatives and Batch Composition

The contrastive loss is built on an assumption that is never checked during training: that the positive pair is the only semantically related pair in the batch. For a batch of images drawn independently from a dataset, this is false often enough to matter. Two photographs of the same breed of dog, two frames from the same video, two paraphrases of the same sentence, or two patches from the same tissue slide are all pairs that the loss will push apart because they occupy different rows.

The magnitude of the problem scales with how semantically clustered the data is. A dataset with a thousand fine-grained classes and a batch of 512 draws many batches in which several items share a class. The loss then penalizes the encoder for placing them close, which is the opposite of what the downstream task wants.

What the pressure does to the space

Consider a batch containing two images of the same object, one as the positive for its own anchor and the other as a negative for every other anchor. The gradient on the second image, as a negative, pushes it away from all anchors except its own. If the two images are the positive pair of each other, the gradients conflict: the positive term pulls them together while the negative terms, evaluated on other rows, pull them apart.

The net effect depends on the relative weight of the terms. Since each row contributes one positive term and N-1 negative terms, the negative pressure dominates by a factor of N-1. The encoder resolves the conflict by placing the two similar items at a moderate distance, close enough to satisfy the positive term partially and far enough to reduce the negative loss. The result is a space in which semantically identical items are not adjacent, and a nearest-neighbour query returns an item from a different class that happened to be in the same batch less often.

The conflict also injects variance into the gradient. Which items co-occur in a batch is random, so the amount of false-negative pressure on any given item varies from step to step. This is one source of the noisy loss curves typical of contrastive training, and it is why batch composition affects the outcome more than batch size alone.

Recognizing positives without labels

The simplest fix is to use labels when they exist. A supervised contrastive loss treats every item with the same label as a positive for the anchor and averages the positive terms, so the loss becomes a softmax over the union of one positive class and the negatives. The formulation handles any number of positives per anchor, including zero, and the gradient pulls the whole class together while pushing other classes away. The resulting space is more clustered by class than the self-supervised version, which helps linear evaluation but can hurt tasks that need to distinguish within a class.

The label-based fix requires labels, which defeats the purpose of self-supervised pretraining. The alternatives estimate positives without them.

One approach uses similarity in an independent space. If a second representation, produced by a different encoder or a different modality, places two items close, they are probably related, and the loss can mask the pair as a non-negative. The estimate is only as good as the second space, and the two encoders can converge to the same mistakes.

Another approach uses the model’s own recent outputs. An item whose representation is very close to the anchor, under a threshold, is likely a positive, and the loss can either drop it or treat it as a positive. The threshold has to be set relative to the similarity distribution, which shifts during training; a fixed threshold either masks nothing early or masks too much late.

A third approach corrects the loss rather than the pairs. If the fraction of false negatives in the batch can be estimated, the negative term can be reweighted so that the expected contribution of a false negative is cancelled. The estimator requires an assumption about the class distribution, and the correction is unbiased only under that assumption, but it avoids the need to identify individual pairs.

def mask_similar(sims, threshold=0.9):
    # sims: (n, n) normalized similarities with positives on the diagonal
    off = sims.clone()
    off.fill_diagonal_(-1.0)
    # entries above threshold are assumed to be the same class as the anchor
    mask = off > threshold
    return mask          # True where a negative should be dropped or relabelled

Masking is not free. If the threshold is too low, genuine hard negatives are dropped and the model stops learning the fine distinctions that make the space useful. The right operating point depends on the desired granularity: a space for coarse retrieval can tolerate aggressive masking, while a space for fine-grained recognition cannot.

Batch construction as a sampling decision

If the batch is a random sample, the expected number of same-class pairs is determined by the class distribution. Deliberately constructing batches changes that number in either direction.

Clustered batches, where items are grouped by a cheap proxy for similarity, increase the number of hard negatives and the number of false negatives together. The hard negatives are valuable because they produce large gradients, and the false negatives are harmful because they produce wrong gradients. The net effect is usually positive for coarse features and negative for fine ones.

Stratified batches, where each batch contains items from as many distinct groups as possible, reduce both. They lower the fraction of false negatives and also lower the fraction of hard negatives, which makes the learning signal weaker per step but cleaner.

The interaction with batch size is not monotone. Doubling the batch doubles the number of negatives per anchor and therefore the chance that at least one is a false negative. In a dataset with heavy class imbalance, a large random batch is almost guaranteed to contain duplicates of the head classes, so the effective negative set for a head-class anchor is much smaller than the nominal one. This is a mechanism by which contrastive pretraining under-represents rare classes even without any explicit sampling bias.

Duplicate detection before the loss

Exact duplicates are the easy case. Datasets assembled from web crawls contain repeated images and repeated text passages, and a batch can contain the same item twice, in which case the two copies are positives of each other by construction but are not marked as such. Deduplicating by content hash before training removes this class of error entirely and is cheaper than any loss-level correction.

Near-duplicates are harder. Perceptual hashing, embedding-based clustering, and shingle-based text comparison all provide a cheap signal, and the usual practice is to remove pairs above a similarity threshold from the training set rather than to handle them in the loss. Removing data is a loss of signal, but training on a mislabelled pair is worse.

The same logic applies across the train and evaluation split. If the same underlying item appears in both, evaluation measures memorization rather than representation quality, and the effect is largest exactly for the near-duplicate pairs that the loss mishandles.

Interaction with other design choices

False negatives are not the only source of conflicting gradients, and the mitigations interact. A momentum encoder, described in Momentum Encoders and Stop-Gradients, changes the target side of the loss, so a false negative’s target representation is computed by the slow branch and its gradient contribution is asymmetric. The conflict is still present but its amplitude differs.

The temperature interacts as well. A sharper softmax concentrates the gradient on the highest-scoring negatives, and a false negative is likely to be among them because it is genuinely similar. Raising the temperature spreads the gradient and reduces the damage from any single mislabelled pair, at the cost of weaker pressure from the hard negatives that are correct. The full set of temperature effects is covered in InfoNCE and the Temperature Parameter. This is one reason temperature is tuned jointly with the data pipeline rather than in isolation.

Finally, the composition of the evaluation set determines whether the problem is visible. A retrieval benchmark built from random pairs will show good numbers even when the space has been distorted by false negatives, because the benchmark’s queries have no near-duplicates either. A benchmark built from within-class pairs, where the query and the target are known to be the same category, exposes the distortion directly, and the gap between the two benchmarks is a rough measure of how much the loss has been fighting the data.