Cross-Modal Alignment and the Modality Gap

A dual-encoder model trained on paired images and text produces one space in which both modalities live. The training signal is a contrastive loss over matched and unmatched pairs, and after training a photograph of a dog and the sentence “a dog on a lawn” have high cosine similarity. That much is expected. What is less expected is that the image vectors and the text vectors occupy different regions of the sphere: the average image embedding and the average text embedding are far apart, and a text query is closer to other text than to the image it describes, even when the matched pair has the highest cross-modal similarity in the set.

The phenomenon is called the modality gap, and it is not a bug in the sense of a broken objective. The loss never asks the two modalities to share a region; it asks matched pairs to be closer than unmatched pairs, which is satisfied by two parallel cones as well as by one shared cluster.

What the objective actually constrains

For a batch of N image-text pairs, the loss is computed in both directions: each image must pick its text out of N candidates, and each text must pick its image. The gradient acts on the relative ordering of similarities within each row and column of the N × N score matrix.

The objective is invariant to any transformation that preserves those orderings. Adding a constant vector to every image embedding and a different constant to every text embedding leaves all cross-modal similarities unchanged up to a shared offset, which the softmax absorbs. The loss is therefore blind to the relative placement of the two modality clouds. It constrains the correspondence between them but not their absolute positions.

In practice the two clouds separate because of initialization and the structure of the data. The image encoder and the text encoder start as independent random functions, so their outputs occupy different regions from the first step. Gradient descent reduces the loss by moving each modality’s outputs to satisfy the pairwise constraints, and the cheapest way to do that is to move them within their existing regions rather than to collapse the regions together. There is no term that would pay for the collapse.

Measuring the gap

The gap is visible in the difference between the mean embedding of each modality. If μ_I is the average of image embeddings over a dataset and μ_T the average of text embeddings, the gap vector is μ_I - μ_T, and its norm is a direct measure. Because embeddings are normalized, the norm of the difference of two unit vectors is between zero and two, and the gap is often a substantial fraction of that range.

A second measure is the modality gap in retrieval. For a text query, compute its similarity to all images and to all texts. If the highest text-text similarity exceeds the highest text-image similarity for the matched item, the query is closer to unrelated text than to its own image, which is a clear signature.

A third is the anisotropy of each modality’s cloud. Text embeddings typically occupy a narrower cone than image embeddings, because the text encoder sees a much smaller input space and its outputs are more correlated. The narrower cone means text-text similarities are systematically higher than text-image similarities, which biases any ranking that mixes the two.

import torch
import torch.nn.functional as F

def gap_stats(img, txt):
    img = F.normalize(img, dim=1)
    txt = F.normalize(txt, dim=1)
    gap = (img.mean(0) - txt.mean(0)).norm()
    cross = (img * txt).sum(1).mean()          # matched pair similarity
    within_i = (img @ img.T).mean()
    within_t = (txt @ txt.T).mean()
    return gap.item(), cross.item(), within_i.item(), within_t.item()

Why the gap is not harmless

If the two clouds are separated, the cross-modal similarity between a matched pair is reduced by a constant that depends on the gap. A larger gap means matched pairs score lower in absolute terms, and the margin over unmatched pairs shrinks if the unmatched pairs are within the same cone. Retrieval still works if the ranking is preserved, but threshold-based decisions, such as “accept this match if similarity exceeds a value”, become miscalibrated: the threshold that works for image-to-text does not work for text-to-image.

The gap also breaks arithmetic in the joint space. The commonly described operation of combining a text vector with an image vector, or of subtracting a concept direction, assumes the two modalities are commensurate. If they occupy separate cones, a sum of an image vector and a text vector lands in a region that belongs to neither, and its nearest neighbours are arbitrary. The composition works only when the offset between the cones is removed first.

Zero-shot classification is affected through the same mechanism. Classifier weights are text embeddings of class names, and the query is an image embedding. The ranking of class similarities is a comparison of the query against a set of vectors that all share the same cone offset, so the offset cancels in the ranking as long as the class texts are similar in style. It does not cancel when the class descriptions vary in length or specificity, because those variations move the text embeddings within their cone by different amounts.

Removing the offset after training

The direct intervention is to remove the mean of each modality after training. Subtracting μ_I from all image embeddings and μ_T from all text embeddings, then renormalizing, centres both clouds at the origin and removes the first-order gap. The operation is cheap, requires only the means over a sample, and preserves the within-modality geometry up to a translation. It does not preserve cross-modal similarity exactly, because the translation changes the angle between a matched pair, but the change is usually small relative to the gap.

The stronger intervention is to whiten each modality separately before comparing, which equalizes the anisotropy as well as the mean. Whitening a modality’s embeddings makes its covariance the identity, so its cone becomes the full sphere. Applying it to both modalities puts them on equal footing and removes the systematic difference in within-modality similarity. The cost is an eigendecomposition per modality and a loss of the norm information that the encoder may have used.

Training-time interventions aim to prevent the gap rather than remove it. Adding a term that penalizes the distance between the modality means, or that requires the batch’s cross-modal similarity matrix to be centred, gives the loss a signal about the relative placement. Some designs regularize each modality’s embeddings toward a shared prior, which has the same effect. The trade is that the additional term competes with the contrastive objective and can reduce retrieval accuracy if it is weighted too heavily.

The gap and the temperature

The temperature in the contrastive loss affects the gap indirectly. A small temperature sharpens the softmax and concentrates gradient on the hardest negatives, which are typically within the same modality because same-modality pairs are more similar. The sharpening therefore pushes the two cones apart, since separating them is a cheap way to reduce same-modality confusion. A larger temperature spreads the gradient and reduces this pressure.

This is a mechanism by which the gap grows with the sharpness of the objective, and it explains why some training recipes with very small temperatures produce larger gaps. The temperature’s other effects on the score distribution are set out in InfoNCE and the Temperature Parameter. The gap’s size also depends on the initial separation of the two encoders and on how much the data allows the clouds to interpenetrate.

Where cross-modal alignment stops

Alignment between modalities is a claim about correspondence, not about shared semantics. A model that aligns images and text can retrieve the right caption for an image without representing the scene’s spatial layout, because the caption and the image are matched as wholes. Tasks that require composition, such as “the dog to the left of the tree”, depend on structure that the pooled embedding does not preserve. Region-level alignment, where parts of an image are matched to phrases, requires an objective that operates on local features rather than on a single vector per input.

The same limitation appears within a single modality. A pooled embedding of a long document loses the order and the local detail that a question about a specific sentence needs. The representation is a summary at a chosen granularity, and the granularity is fixed by the training objective, not by the downstream task.

Finally, cross-modal training inherits the false-negative problem in a sharper form. A batch of image-text pairs from a web crawl contains many texts that describe images other than their own, and many images that match multiple texts. The loss treats every off-diagonal entry as a negative, so the encoder is pushed to separate pairs that are genuinely related. The mechanisms for handling this are described in False Negatives and Batch Composition, and they apply with more force here because the pairing is noisier than in curated single-modality data.

The practical consequence for a system that uses a joint space is that the gap should be measured before the space is used, not assumed away. Two encoders trained together, with the same loss and the same data, can produce spaces with materially different gaps depending on the initial separation and the temperature, and downstream thresholds, arithmetic, and classifier weights all inherit that difference. The measurement is a few lines over a sample of paired data, and it is the only way to know whether a similarity value means the same thing for an image query as for a text query.