Similarity Metrics and the Normalisation They Require

An embedding model returns a point in R^d. Turning that point into a ranking requires a choice of comparison, and the three common choices disagree in ways that matter: cosine similarity, inner product, and Euclidean distance. On unit-length vectors the first two are identical and the third is a monotone function of them, so the three collapse into one ordering. On arbitrary vectors they do not, and the divergence is not a rounding difference but a reordering of results. The single most common source of unexplained retrieval degradation is a pipeline where the encoder, the index, and the query path do not agree on which convention is in force.

The arithmetic is small enough to state exactly. Inner product is the sum over dimensions of the product of corresponding coordinates. Cosine similarity divides that sum by the product of the two vector norms, so it measures angle and ignores magnitude. Euclidean distance is the square root of the sum of squared coordinate differences, and its square expands to the sum of the squared norms minus twice the inner product. For a fixed query, ranking by squared Euclidean distance is therefore equivalent to ranking by inner product minus the squared norm of the candidate. When every candidate has the same norm, the two orderings coincide. When they do not, the norm term decides ties, and a long document with a large-norm embedding gets an advantage in inner product that it does not get under cosine.

Why embeddings have unequal norms at all

A dual-encoder trained with an inner-product objective has no incentive to keep output norms constant. The contrastive loss it minimises rewards large projections for vectors that should be close and pushes others away, and the cheapest way to increase a dot product is to increase both norms. Training therefore tends to inflate norms, and it inflates them unevenly: frequent, central concepts accumulate larger norms than rare ones, because they appear in more positive pairs.

The consequence is that inner-product search over unnormalised vectors is not a pure semantic search. It is a search weighted by a magnitude the model learned incidentally, which acts as a prior toward common material. For a corpus where the same boilerplate sentence appears in a thousand documents, that prior is actively harmful: the boilerplate has a large norm and dominates results for queries that merely share its vocabulary.

Normalisation removes the prior. Dividing every vector by its norm projects the corpus onto the unit hypersphere, after which inner product equals cosine and ranking depends only on direction. Encoders trained with a cosine objective, or trained with an inner-product objective and then normalised at encode time, produce embeddings whose geometry matches this treatment, and the two metrics then give identical orderings.

The cost is that the norm information is discarded permanently. It cannot be recovered at query time, so if a downstream stage wanted to use confidence or specificity as a signal, that signal is gone. In practice the discarded information is rarely useful and the discarded prior is usually harmful, which is why normalisation is the default rather than the exception.

import math

def encode_normalised(text, model, dim):
    v = model.encode(text)                      # returns a list of dim floats
    n = math.sqrt(sum(x * x for x in v))
    if n == 0.0:                                # degenerate: empty or all-zero output
        return [0.0] * dim
    return [x / n for x in v]

def cosine(a, b):
    # valid only because both inputs are unit length; no norms recomputed here
    return sum(x * y for x, y in zip(a, b))

The zero-norm case is not hypothetical. An encoder given an empty string, or a string it tokenises to nothing, may return an all-zero vector, and dividing by its norm raises or produces NaN. A NaN propagated into a distance computation poisons the index: some implementations will silently sort it last, others will crash, and a few will return it first because the comparison is false in both directions. The guard is one line and belongs at encode time, not at query time.

What normalisation does to an index

For inner-product indexes, normalising at index time is what makes the index correct. A graph index built with Euclidean distance on unit vectors is also usable, because on the unit sphere squared Euclidean distance is 2 minus twice the inner product, a strictly decreasing function of similarity. So a normalised corpus can be served by any of the three metrics, and the choice becomes one of implementation convenience rather than of semantics, which is why the recall a graph index delivers does not depend on the metric once the corpus is normalised.

For unnormalised corpora, the choice is binding. An index built for inner product stores adjacency chosen by dot-product proximity, which is not symmetric in the way Euclidean proximity is, and the graph’s greedy walk assumes a metric structure it does not have. Implementations handle this by converting the inner-product problem into a Euclidean one through an augmented dimension, or by restricting the index to a family that supports it directly. Either way, the index and the metric must be decided together; swapping the metric on an existing index produces results that are wrong in a way no amount of parameter tuning fixes.

Quantization interacts with normalisation in a way that surprises people. Compressing a unit vector into 8-bit codes and reconstructing it does not return a unit vector; the reconstruction has a norm slightly below one, and the error is systematic rather than random. Rescoring candidates against full-precision vectors corrects the ranking, but if the rescoring step compares a compressed candidate against a normalised query using cosine, the reconstruction error appears as a spurious angle change. Keeping the same metric at both stages, and rescoring only the shortlist, avoids the compounding.

Scale and offset: the failure that looks like bad luck

The most damaging convention mismatch is not between metrics but between the two encoders in a pipeline. A query encoder that applies normalisation and a document encoder that does not produces two vector sets in the same space with different scales. Cosine similarity still works, because it divides by both norms. Inner product does not, because the query’s unit norm and the document’s inflated norm multiply into a score that is dominated by document length.

A subtler variant involves models that were trained with a fixed scale factor. Some contrastive setups learn a temperature that multiplies the logits, and some embedding APIs return vectors that have been scaled by a constant before normalisation. If the constant differs between the indexing job and the query path, for example because one was pinned to an older model revision, every score shifts by a factor. Ranking within a single query is unaffected by a uniform scale, which is why the bug survives testing: the top result is still plausible. It becomes visible only when scores are thresholded, because a fixed cutoff calibrated on one scale silently drops or admits everything on another.

Two operational rules follow. The encoder revision used for indexing must be recorded alongside the index, and a query encoded by a different revision must be rejected rather than served, since embedding spaces are not compatible across model versions even when the architecture is unchanged. And scores must never be thresholded against a constant that was not measured on the same encoder and the same metric; a threshold is a property of the triple of encoder, metric, and corpus, not of the score alone.

When the metric should not be cosine

Cosine is the default because it is scale-invariant and because most text encoders are trained for it. Three situations argue otherwise.

The first is asymmetric retrieval, where queries are short and documents are long and the model was trained to predict relevance rather than to align directions. Some of these models are trained with inner product over unnormalised outputs, and the norm carries a genuine signal about document length and specificity. Forcing normalisation discards it and can lower ranking quality. The encoder’s documentation is the authority here, not the general default.

The second is when the vector is not an embedding of text but a feature vector assembled from counts or ratings. Cosine on a term-frequency vector ignores document length by design, which is sometimes desired and sometimes not; the length normalisation that BM25 performs is a deliberate alternative, and the two produce different rankings on the same corpus, which is the starting point for Hybrid Lexical and Dense Retrieval.

The third is a corpus where the relevant documents are near-duplicates of each other. Under cosine, near-duplicates occupy almost the same direction, so the top twenty results may all be the same paragraph restated. Inner product with a mild norm term, or an explicit diversity step after retrieval, changes which duplicates surface. This is a ranking-objective problem rather than a metric problem, but the metric is where it first appears.

The check that catches most of it

A single diagnostic distinguishes most convention errors from genuine retrieval failure: compute the norm distribution of the indexed vectors and of the query vectors. Unit-normalised sets have norms within a small band around one. A corpus with a long tail of norms, or a query set whose norms sit at a different scale from the document set, indicates that normalisation is inconsistent. The second diagnostic is to compare the ordering produced by cosine and by inner product on a sample of queries against a hand-checked set of relevant chunks. If the two orderings differ substantially and the cosine ordering matches the checked set, the index is serving the wrong metric, and the fix is at the boundary between the encoder and the index rather than anywhere in the search itself.

Both checks are cheap, both are independent of the index implementation, and neither requires labelled data beyond a few dozen queries. They are worth running before tuning any search parameter, because parameter tuning on a metric mismatch optimises the wrong objective and produces configurations that do not transfer.