Deep Learning · Cluster
Transformers
Attention mechanics and scaling behavior.
- 01 Why Scaled Dot-Product Attention Divides by the Square Root of the Key Dimension Where the 1/sqrt(d_k) factor comes from, what quantity it holds constant as width changes, and how a saturated softmax destroys the gradient reaching queries and keys.
- 02 Multi-Head Attention as Independent Subspace Projections What the concatenation of heads buys over a single wide head, how the output rank is bounded, and why pruned heads can be removed without changing behaviour much.
- 03 Permutation Equivariance in Self-Attention and What Position Encodings Restore Why an attention layer treats a sequence as a set, the exact symmetry group it is invariant to, and how absolute, relative, and rotary encodings break that symmetry differently.
- 04 Causal Masking, Additive Mask Terms, and Where the Mask Enters the Softmax Why a causal constraint is expressed as negative infinity added to the logits, how padding and causality combine in one layer, and which arithmetic the mask must precede.
- 05 Rotary Position Embeddings and Why the Score Depends Only on the Offset How rotating query and key pairs by an angle proportional to index makes attention a function of relative distance, and what the rotation periods cost when context grows.
- 06 KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling How the decode-time key and value cache is sized, why it scales with every layer rather than only the last, and how its growth sets the maximum batch on a fixed device.
- 07 IO-Aware Attention: Tiling the Score Matrix Without Materializing It Why the intermediate attention tensors cost more than the arithmetic, how blockwise softmax stays exact, and what the online rescaling step has to preserve.
- 08 Grouped-Query Attention and What Sharing Key Heads Costs How mapping several query heads onto one key head shrinks the cache by an exact factor, what the shared head cannot represent, and why the loss falls harder on keys than on values.
- 09 Sliding-Window Attention and the Reach of a Stacked Local Window How a fixed-width window bounds per-layer cost while the stack multiplies its reach, what the reach formula predicts, and where the prediction fails because information cannot travel that far.
- 10 Compute-Optimal Allocation Between Attention and Feed-Forward Width How parameter and flop budgets split between attention and MLP blocks, why the split shifts with sequence length, and which allocations survive a change in context size.