Tag
continuous-batching
- Aug 8, 2026 Iteration-Level Batching: Reforming the Batch at Every Token How a serving loop rebuilds its batch before every forward pass, why request-level batching wastes half the machine, and which budget actually limits concurrency. inference systems
- Jun 18, 2026 Iteration-Level Batching and the Cost of a Fixed Batch Why a batch fixed at admission time strands accelerator capacity, and how re-deciding the batch at every decode step changes the cost of a token. inference systems