Tag
prefill
- Apr 22, 2026 Chunked Prefill and the Interference It Removes Prefill is compute-bound and decode is bandwidth-bound, so running them in the same iteration makes each wait on the other. Splitting prompts into chunks is the standard remedy. inference systems
- Apr 6, 2026 Disaggregated Prefill and Decode Pools Separating prompt processing from token generation into distinct pools, the cache handoff that joins them, and how the capacity ratio between pools is set. inference systems
- Feb 4, 2026 Prefill and Decode: The Two Cost Curves Behind One Token Price Why the same model spends prefill time on arithmetic and decode time on memory bandwidth, and how that split decides what a token costs to serve. inference systems
- Jun 23, 2025 Prefill and Decode Disaggregation Prefill saturates arithmetic and decode saturates bandwidth. Putting them on separate machines lets each be sized for its own bottleneck, at the cost of moving the KV cache over a network. inference systems
- Apr 13, 2025 Splitting Prefill From Decode Inside a Step Prompt processing is compute-bound and token generation is bandwidth-bound; how mixing them in one step costs latency, and how chunked prefill bounds the interference. inference systems