Tag
context-length
- Dec 26, 2024 Rotary Position Embeddings and Why the Score Depends Only on the Offset How rotating query and key pairs by an angle proportional to index makes attention a function of relative distance, and what the rotation periods cost when context grows. deep learning
- May 17, 2024 Sliding-Window Attention and the Reach of a Stacked Local Window How a fixed-width window bounds per-layer cost while the stack multiplies its reach, what the reach formula predicts, and where the prediction fails because information cannot travel that far. deep learning
- Feb 20, 2024 The KV Cache Memory Budget and What It Costs per Token Deriving bytes per cached token from attention shapes, then turning the freed capacity into a hard ceiling on concurrency and a floor on cost per token. inference systems