Tag
multi-head
- Sep 19, 2025 Multi-Head Attention as Independent Subspace Projections What the concatenation of heads buys over a single wide head, how the output rank is bounded, and why pruned heads can be removed without changing behaviour much. deep learning
- Jun 11, 2025 Grouped-Query Attention and What Sharing Key Heads Costs How mapping several query heads onto one key head shrinks the cache by an exact factor, what the shared head cannot represent, and why the loss falls harder on keys than on values. deep learning