Tag
decode
- Jun 18, 2026 Iteration-Level Batching and the Cost of a Fixed Batch Why a batch fixed at admission time strands accelerator capacity, and how re-deciding the batch at every decode step changes the cost of a token. inference systems
- Apr 22, 2026 Chunked Prefill and the Interference It Removes Prefill is compute-bound and decode is bandwidth-bound, so running them in the same iteration makes each wait on the other. Splitting prompts into chunks is the standard remedy. inference systems
- Apr 6, 2026 Disaggregated Prefill and Decode Pools Separating prompt processing from token generation into distinct pools, the cache handoff that joins them, and how the capacity ratio between pools is set. inference systems
- Feb 4, 2026 Prefill and Decode: The Two Cost Curves Behind One Token Price Why the same model spends prefill time on arithmetic and decode time on memory bandwidth, and how that split decides what a token costs to serve. inference systems
- Jun 23, 2025 Prefill and Decode Disaggregation Prefill saturates arithmetic and decode saturates bandwidth. Putting them on separate machines lets each be sized for its own bottleneck, at the cost of moving the KV cache over a network. inference systems
- Apr 13, 2025 Splitting Prefill From Decode Inside a Step Prompt processing is compute-bound and token generation is bandwidth-bound; how mixing them in one step costs latency, and how chunked prefill bounds the interference. inference systems
- Feb 24, 2025 Decode Bandwidth and the Cost of Variable Instruction Length Why fixed-width instructions let a front end decode several per cycle, how variable-length encodings force serial scanning, and where the two approaches land on fetch and decode throughput. computer architecture