Rotary Position Embeddings and Why the Score Depends Only on the Offset

Adding a position vector to the token embedding works, but it does so by making the content of a token’s representation depend on where it sits. A token at index 40 and the same token at index 41 carry different vectors even before the layer sees them, and the model must learn to subtract that difference away whenever it wants to reason about distance rather than about absolute place. Rotary embeddings take a different route: they leave the vectors alone and rotate them, in a way that makes the inner product between a query and a key depend on the difference of their indices and on nothing else.

The rotation, written out

Partition a query vector $q$ and a key vector $k$, both of width $d_k$, into $d_k/2$ pairs of adjacent coordinates. For each pair index $j \in {0, \dots, d_k/2 - 1}$, define a frequency

$$\theta_j = b^{-2j/d_k}$$

with a base $b$ conventionally set to 10,000. Position $m$ rotates the pair by angle $m\theta_j$:

$$R(m) = \begin{pmatrix} \cos m\theta_0 & -\sin m\theta_0 & & & \ \sin m\theta_0 & \cos m\theta_0 & & & \ & & \ddots & & \ & & & \cos m\theta_{d/2-1} & -\sin m\theta_{d/2-1} \ & & & \sin m\theta_{d/2-1} & \cos m\theta_{d/2-1} \end{pmatrix}$$

This is a block-diagonal rotation matrix: orthogonal, with determinant 1, and with the useful property that $R(m)^\top R(n) = R(n - m)$. The layer computes $q_m’ = R(m)q_m$ and $k_n’ = R(n)k_n$, and then the usual scaled dot product:

$$q_m’ \cdot k_n’ = q_m^\top R(m)^\top R(n) k_n = q_m^\top R(n-m) k_n$$

The score depends on $m$ and $n$ only through their difference. No absolute index survives in the expression, and no learned parameter is introduced: the rotation is fixed by the frequency schedule and the index.

Because $R$ is orthogonal, it preserves the norm of the vector it acts on. That is worth stating explicitly because it separates rotary embeddings from every additive scheme: adding a position vector changes the norm of the representation, and the change is larger for some positions than others when the encoding’s coordinates happen to align with the content. Rotation never changes a norm, so the scale statistics that the rest of the block relies on are untouched. The variance argument behind the attention divisor is unaffected.

Why this is translation equivariance and not just relative bias

A relative-bias scheme adds a learned scalar or vector to the score as a function of $j - i$. The model has to learn the table, the table has a finite range, and offsets beyond that range are clipped. RoPE achieves a relative dependence with no table and no clipping, because the rotation is exact for any pair of integers.

The consequence is a symmetry the additive schemes do not have: if every token in the sequence is shifted by a constant offset $c$, then every rotation angle shifts by $c\theta_j$ in each pair, and every score $q^\top R(n-m)k$ is unchanged. The outputs are identical, not merely similar. This is why a rotary model trained on sequences of one length can in principle process a sequence whose absolute indices start anywhere.

The symmetry has a limit, and it is the period of the rotation. Each pair rotates with period $2\pi/\theta_j$. For the lowest-frequency pair at $d_k = 64$ and base 10,000, $\theta_{31} = 10000^{-62/64} \approx 1.3 \times 10^{-4}$, so the period is $2\pi/\theta_{31} \approx 47{,}000$ positions before the angle wraps. For the highest-frequency pair, $\theta_0 = 1$, so the period is $2\pi \approx 6.28$ positions. Adjacent indices are distinguished by a rotation of about one radian in that fastest pair, which is a large, easily separable change; indices 100 apart differ by 100 radians, which modulo $2\pi$ is indistinguishable from some much smaller offset. The fast coordinates provide fine resolution over short distances and alias over long ones; the slow coordinates provide coarse resolution over long distances and are what allows a model to represent a 4,000-token offset at all.

That division of labour is the design. It is also the source of the difficulty in extending context, since the slow coordinates were chosen so that the largest trained offset corresponds to a modest fraction of their period. Push the index well past the trained range and the slowest coordinates wrap too, and the model has no coordinate left that changes monotonically over the new range.

What changes when the context is extended

Three families of adaptation appear in the literature, and they differ in which quantity they rescale.

Position interpolation divides the index by a factor $s$ before applying the rotation, so index $m$ is treated as $m/s$. The effect is that a sequence of length $s \cdot L$ occupies the same range of angles that a length-$L$ sequence did during training. Every offset is compressed by $s$, which preserves the ordering and keeps every coordinate inside its trained range, at the cost of reducing the angular separation between adjacent tokens in the fast coordinates. At $s = 8$ the fastest pair’s per-step rotation falls from 1 radian to 0.125, which is still separable in 16-bit precision but is a much finer distinction than the model saw during training.

Frequency-base adjustment changes $b$ instead. Raising the base stretches every period by the same factor while leaving the fastest coordinate’s period nearly unchanged, because $\theta_0 = b^0 = 1$ regardless of $b$. The result is that short offsets keep their original resolution and long offsets gain range, a non-uniform stretch that is better matched to the non-uniform periods than a uniform index division. The trade-off is that no single base is optimal for all coordinates, and the choice interacts with the model’s trained frequency distribution.

The third family applies a smooth ramp, leaving low indices unscaled and compressing only the range beyond the trained length. Implementations differ in the exact ramp, and the details are where the reported quality differences live; the mechanism is the same in all of them, which is a per-coordinate modification of the effective angle.

None of these change the parameter count, since RoPE has none. They change the function applied to indices, and because the function is applied at inference time to indices the model never saw, they change behaviour without retraining, which is why they are attractive and also why their effect is hard to predict from first principles.

import numpy as np

def rope_tables(n, d_k, base=10000.0):
    j = np.arange(0, d_k, 2)
    theta = base ** (-j / d_k)          # d_k/2 frequencies
    m = np.arange(n)[:, None]
    ang = m * theta[None, :]            # (n, d_k/2)
    return np.cos(ang), np.sin(ang)

def apply_rope(x, cos, sin):
    # x: (n, d_k) laid out as interleaved pairs (x0,x1), (x2,x3), ...
    x1, x2 = x[:, 0::2], x[:, 1::2]
    out = np.empty_like(x)
    out[:, 0::2] = x1 * cos - x2 * sin
    out[:, 1::2] = x1 * sin + x2 * cos
    return out

The interleaved layout in that snippet is one of two conventions. The other splits the vector into two halves and pairs coordinate $i$ with coordinate $i + d_k/2$, which is equivalent up to a permutation of the feature axis and gives the same scores. The two conventions are not interchangeable across checkpoints: applying the wrong one to a pretrained model produces a rotation that mixes unrelated coordinates, and the resulting model degrades in a way that looks like a corrupted checkpoint rather than like a bug in the encoding.

Interaction with the rest of the block

RoPE is applied to queries and keys after the projection and before the score computation, per head, and it is not applied to values. The value path carries content only; rotating values would add position information to the output in a way that the output projection cannot undo, and the relative property would be lost because the values are summed with weights that already depend on position.

Because the rotation is per-head and depends only on $d_k$, heads with different widths need different frequency tables, which is one reason per-head width is usually uniform. The tables are also cheap: for a context of 8,192 and $d_k = 128$, the cosine and sine tables hold $8192 \times 64$ entries each, about half a million values apiece, and they are computed once and reused across layers and batch elements. Some implementations compute the rotation on the fly from the angle to avoid storing the tables at long context, trading a pair of transcendental functions per element against the memory.

The interaction with causal masking is neutral: the mask decides which pairs are compared, and the rotation applies to whichever pairs survive. The interaction with grouped-query attention is not neutral, and it is a common source of confusion. When several query heads share one key head, each query head has its own rotation applied at its own index, and the shared key head is rotated once. Since the relative property holds per pair, the score between query head $i$ at position $m$ and the shared key at position $n$ still depends only on $n - m$, so sharing keys does not break the encoding. It does mean the key rotation must be computed once and reused, not once per query head, which is a small but real saving described in grouped-query attention and KV cache sharing.

A subtler point concerns the aliasing described above and the effective rank of the score matrix. Because the fast coordinates alias at short periods, two keys separated by an offset that is an exact multiple of $2\pi/\theta_0$ in every coordinate would be indistinguishable, but that requires the offset to be a common multiple of all periods, and the frequencies are irrational multiples of one another, so no such offset exists in practice. The degeneracy is instead gradual: as the offset grows, more coordinates wrap, and the score becomes progressively less sensitive to the exact distance. This is the mechanism behind the observation that rotary models attend accurately to nearby tokens and increasingly coarsely to distant ones, and it is a property of the encoding rather than a defect of training.

The choice between rotary and the additive schemes is not a matter of one being correct. Additive absolute encodings give the model a fixed reference frame, which is useful when absolute position carries meaning, as with a document’s first token, a header, or a fixed-format field. Rotary encodings give exact translation equivariance and no parameter growth with context, which is useful when distance is what matters. The symmetry each provides is developed further in permutation equivariance and position encodings, and the masking that decides which pairs are scored at all is in causal masking and mask placement.