Tag
kv-cache
- Aug 8, 2026 Iteration-Level Batching: Reforming the Batch at Every Token How a serving loop rebuilds its batch before every forward pass, why request-level batching wastes half the machine, and which budget actually limits concurrency. inference systems
- Jul 21, 2026 Autoscaling Signals for Token Servers CPU utilization and request rate both mislead when capacity is pinned by KV cache and iteration time. Queue depth in tokens, cache occupancy, and time-to-first-token measure what matters. inference systems
- Jun 16, 2026 KV Cache Quantization and Per-Token Scale Placement Why the key and value caches need different quantization axes, how per-token scales interact with the attention kernel, and what happens to the softmax when keys lose precision. inference systems
- Feb 10, 2026 Paged KV Cache Allocation and the Block Table Why reserving contiguous cache for a worst-case output length caps batch size, and how fixed-size blocks with an indirection table remove most of that waste. inference systems
- Jan 5, 2026 Preemption by Swap Versus Recomputation When the cache pool cannot serve a running sequence, the scheduler must take memory back; the two recovery strategies have different costs that depend on sequence length. inference systems
- Dec 5, 2025 Paged Key-Value Residency and Block Tables How fixed-size cache blocks and per-sequence block tables remove the allocation and fragmentation penalties of contiguous cache extents. inference systems
- Nov 11, 2025 How Key-Value Cache Bytes Set the Batch Ceiling The byte arithmetic that decides how many sequences fit beside the weights, why the ceiling moves with context length, and what remains after it is reached. inference systems
- Sep 2, 2025 Prefix Caching and Shared Residency Deduplicating key-value blocks across requests that begin with the same tokens, how blocks are matched by hash, and what eviction must preserve. inference systems
- Jun 11, 2025 Grouped-Query Attention and What Sharing Key Heads Costs How mapping several query heads onto one key head shrinks the cache by an exact factor, what the shared head cannot represent, and why the loss falls harder on keys than on values. deep learning
- May 29, 2025 Tensor-Parallel Residency and Interconnect Pressure Sharding a model across devices splits weights and key-value heads, but every decode step becomes a collective; how that changes the memory and latency balance. inference systems
- Sep 6, 2024 KV Cache Arithmetic: Bytes per Token, Layers, and the Batch Size Ceiling How the decode-time key and value cache is sized, why it scales with every layer rather than only the last, and how its growth sets the maximum batch on a fixed device. deep learning
- Feb 20, 2024 The KV Cache Memory Budget and What It Costs per Token Deriving bytes per cached token from attention shapes, then turning the freed capacity into a hard ceiling on concurrency and a floor on cost per token. inference systems