Tag
position-encoding
- Jul 7, 2025 Permutation Equivariance in Self-Attention and What Position Encodings Restore Why an attention layer treats a sequence as a set, the exact symmetry group it is invariant to, and how absolute, relative, and rotary encodings break that symmetry differently. deep learning
- Dec 26, 2024 Rotary Position Embeddings and Why the Score Depends Only on the Offset How rotating query and key pairs by an angle proportional to index makes attention a function of relative distance, and what the rotation periods cost when context grows. deep learning