Cosine Similarity and the Geometry of Normalized Embedding Vectors

A vector index returns a ranked list, and the ranking is produced by a scoring function applied to pairs of vectors. Before any tree, graph, or hash table exists, the choice of that function decides what “near” means, which candidates are worth exploring, and how the index must be built. The cosine score is the most common choice, and it carries a set of geometric commitments that are easy to inherit without noticing.

What the cosine score keeps and what it throws away

For two vectors $a$ and $b$ in $\mathbb{R}^d$, the cosine is the dot product divided by both Euclidean lengths:

$$\cos(a,b) = \frac{a \cdot b}{\lVert a \rVert , \lVert b \rVert}$$

The value lies in $[-1, 1]$ for real vectors, and it depends only on the angle between the two directions. Scaling $a$ by any positive constant leaves the score unchanged. That invariance is the entire point: in a text embedding, the length of the vector often tracks something orthogonal to topical similarity, such as document verbosity or the number of tokens averaged into a pooled representation, so dividing it out removes an unwanted nuisance variable.

The cost is that the discarded magnitude cannot be recovered. Some embedding models deliberately place information in the norm. When a model is trained with a contrastive objective that includes a norm-related term, or when vectors are produced by summing token vectors without averaging, a long vector may genuinely be a stronger or more specific signal. Normalizing such vectors conflates a short, confident vector with a long, diffuse one pointing the same way. The failure is silent: scores remain in range, rankings remain plausible, and nothing errors.

The other commitment is that cosine ignores the origin. Two vectors that both point in nearly the same direction score high even if one is a hundred times longer. For retrieval over normalized embeddings this is desirable. For a representation where position in space matters, such as a coordinate encoding or a vector whose components are counts of distinct features, the angle is the wrong summary and Euclidean distance is the one that preserves the intended neighborhood structure.

Why normalization reduces the index to a dot product

Dividing every stored vector by its own norm once, at write time, makes all lengths exactly one. Then $\lVert a \rVert = \lVert b \rVert = 1$ and the cosine equals $a \cdot b$. The per-query division disappears entirely, and the inner loop of every distance computation becomes a multiply-accumulate chain with no square root and no division.

The saving is not marginal. For a $d$-dimensional vector, a naive cosine computation performs $d$ multiplies, $d$ adds, two length calculations of $d$ operations each, one square root per vector, and one division. Pre-normalization reduces this to $d$ multiplies and $d$ adds. On the query side one normalization remains, amortized over every comparison the query makes. Since the comparison is the innermost operation of the whole system, at perhaps billions of comparisons per second in a large index, removing a square root and two divisions from it is the difference between a compute-bound and a memory-bound loop.

Normalization is also what makes the common shortcut safe. Many indexes store vectors as float32, and a dot product accumulated in float32 over a few hundred dimensions loses low-order bits as the running sum grows. Because the normalized dot product is bounded by one in magnitude, the absolute error of the sum stays proportional to the vector length in components rather than to the magnitudes of the inputs. With unnormalized vectors whose components are in the hundreds, the same absolute error is a far smaller relative error but a far larger absolute one, and thresholds expressed as raw scores stop being portable across models.

The practical rule that follows: normalize once, store the normalized form, and treat the magnitude as metadata if it matters at all. Storing both the original norm and the unit vector costs four extra bytes per record and allows the magnitude to be reintroduced into a final reranking stage without recomputing anything.

The metric question, and why maximum inner product is the odd one out

Cosine distance, $1 - \cos(a,b)$, is a metric on the unit sphere. It is non-negative, symmetric, zero only for identical directions, and it satisfies the triangle inequality because it equals half the squared Euclidean distance between the corresponding unit vectors. That last identity is worth writing down:

$$\lVert \hat{a} - \hat{b} \rVert^2 = 2 - 2,\hat{a}\cdot\hat{b} = 2,(1 - \cos(a,b))$$

Ranking by cosine distance and ranking by Euclidean distance on normalized vectors produce identical orders. A system that advertises Euclidean search over unit vectors is performing cosine search, and a system that advertises cosine search can use any Euclidean index by normalizing first. The equivalence breaks the moment vectors are not normalized, because the expansion of the squared distance carries the two norm terms:

$$\lVert a - b \rVert^2 = \lVert a \rVert^2 + \lVert b \rVert^2 - 2,a\cdot b$$

The norms are not constant across records unless normalization forced them to be, so distance and inner product order documents differently.

Raw inner product is not a metric at all. It is not non-negative, it is not zero on identical vectors, and it violates the triangle inequality; the maximum inner product search problem is structurally different from nearest neighbor search in a metric space. The standard reduction appends one extra coordinate derived from the largest norm in the collection, transforming the vectors so that maximum inner product becomes nearest neighbor in a space one dimension wider. This is the transformation behind the observation that some recommendation models, trained to score by dot product, cannot be served by a plain Euclidean index without it.

The distinction matters when the index is approximate. Metric indexes exploit the triangle inequality to prune; the pruning argument assumes a metric. An index built for Euclidean distance and then fed unnormalized vectors for an inner-product query does not merely return a different ranking, it returns candidates selected under the wrong geometry. The over-fetch and reranking pattern described in Over-Fetch and Exact Reranking in Two-Stage Vector Retrieval exists partly to absorb this class of mismatch.

Bytes per vector and the arithmetic of a full scan

Embedding widths cluster in the low hundreds to about fifteen hundred dimensions. At float32, four bytes per component, a single 768-dimensional vector occupies 3072 bytes, exactly three kibibytes. A million vectors occupy about three gigabytes, 3.072 billion bytes, before any index overhead, and the index structures themselves add a pointer or two per vector per graph edge or posting entry.

That arithmetic sets the baseline that every approximate method is measured against. A brute-force scan of one million 768-dimensional float32 vectors reads those three gigabytes and performs roughly $7.7 \times 10^8$ multiply-accumulate operations per query. Modern hardware can execute the arithmetic far faster than it can deliver three gigabytes from memory, so the exact scan is bound by memory bandwidth, not by floating-point throughput. This is why the first line of attack against brute force is not a better algorithm but a smaller representation: halving bytes per component, or replacing a vector with a short code as Product Quantization and the Asymmetric Distance Computation does, cuts the scan cost proportionally and is often a larger win than a more elaborate index.

import numpy as np

def prepare(matrix: np.ndarray) -> np.ndarray:
    """Row-normalize once, at write time. Zero vectors get an explicit norm."""
    norms = np.linalg.norm(matrix, axis=1, keepdims=True)
    # A zero vector has no direction; substituting an arbitrary unit vector
    # keeps the row finite and makes the anomaly visible downstream.
    norms[norms == 0] = 1.0
    return (matrix / norms).astype(np.float32)

def top_k(query: np.ndarray, unit_rows: np.ndarray, k: int = 10):
    q = query / max(np.linalg.norm(query), 1e-12)
    scores = unit_rows @ q                      # cosine, since both sides are unit length
    part = np.argpartition(-scores, k - 1)[:k]
    return part[np.argsort(-scores[part])]

The loop is a single matrix-vector product, which is the shape every exact index eventually reduces to. In a batched form the same matrix-vector product becomes a matrix-matrix product against a block of queries, and the block amortizes the cost of streaming the stored vectors once for several queries.

Where the geometry stops matching the model

Four cases account for most cosine surprises.

Zero-length vectors appear when a document has no extractable content, when a pooling step averages an empty set, or when a model emits a masked-out representation. The norm is zero, the division is undefined, and a naive implementation produces a NaN that propagates through the ranking and silently sorts to one end. Explicit handling, either rejecting the record at write time or substituting a canonical direction, is the only reliable fix.

Near-zero norms are worse than exact zeros because they do not announce themselves. A vector with norm around $10^{-7}$, produced by cancellation between large components, divides into a unit vector whose direction is dominated by rounding error. Two such records may score as near-duplicates. The acceptable floor for the norm depends on the float precision and the accumulation order, but a threshold well above the epsilon of the storage type is prudent.

Asymmetric models are a third source. Retrieval systems frequently embed queries and documents with different input templates or different encoder towers, and the two embedding spaces are only comparable if the training procedure aligned them. Cosine similarity between a query vector and a document vector from unaligned spaces produces numbers in range with no meaning. The check is empirical and cheap: a handful of known relevant pairs should score above a handful of known irrelevant ones before any index is built. Even with aligned models, the spread of scores shrinks as the width grows, for the reasons given in Distance Concentration and Why Tree Indexes Collapse in High Dimensions.

Finally, score thresholds do not transfer. A cosine of 0.75 means one thing for one model and something entirely different for another, and the distribution of scores shifts with the corpus, since a corpus of near-duplicate documents concentrates every score near one. Absolute cutoffs must be calibrated per collection, and a relative cutoff, such as a fixed fraction of the top score, is usually more stable. The same reasoning applies to the distance metrics an index exposes, and it is the reason index parameters are tuned per collection rather than inherited from a default.

The geometry is exact and the arithmetic is cheap; what requires care is the assumption that the model’s notion of similarity survives the projection onto the unit sphere. Where it does, cosine is the right score and normalization is a free speedup. Where it does not, the loss is invisible in every metric the system reports.