Tag
attention
- Jul 6, 2026 Attention as Content-Based Alignment Over Encoder States How a decoder selects among stored states with a learned weight distribution, why that gives a short gradient path to any position, and what the attention weights cannot represent. deep learning
- May 21, 2026 Why Scaled Dot-Product Attention Divides by the Square Root of the Key Dimension Where the 1/sqrt(d_k) factor comes from, what quantity it holds constant as width changes, and how a saturated softmax destroys the gradient reaching queries and keys. deep learning
- Sep 19, 2025 Multi-Head Attention as Independent Subspace Projections What the concatenation of heads buys over a single wide head, how the output rank is bounded, and why pruned heads can be removed without changing behaviour much. deep learning
- Jan 28, 2025 Context Dilution: When More Retrieved Passages Lower Answer Quality Why irrelevant context degrades generation rather than merely failing to help, how position in the prompt changes attention, and what to cut before raising k. inference systems
- Sep 6, 2024 KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling How the decode-time key and value cache is sized, why it scales with every layer rather than only the last, and how its growth sets the maximum batch on a fixed device. deep learning
- Nov 4, 2021 IO-Aware Attention: Tiling the Score Matrix Without Materializing It Why the intermediate attention tensors cost more than the arithmetic, how blockwise softmax stays exact, and what the online rescaling step has to preserve. deep learning