Tag
inference-memory
- Jun 11, 2025 Grouped-Query Attention and What Sharing Key Heads Costs How mapping several query heads onto one key head shrinks the cache by an exact factor, what the shared head cannot represent, and why the loss falls harder on keys than on values. deep learning
- Sep 6, 2024 KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling How the decode-time key and value cache is sized, why it scales with every layer rather than only the last, and how its growth sets the maximum batch on a fixed device. deep learning