Tag
throughput-latency
- Aug 8, 2026 Iteration-Level Batching: Reforming the Batch at Every Token How a serving loop rebuilds its batch before every forward pass, why request-level batching wastes half the machine, and which budget actually limits concurrency. inference systems
- May 13, 2025 Batch Size, Occupancy, and the Marginal Cost of a Token How the cost of one more concurrent request falls as the batch grows, where it stops falling, and why the knee of that curve sets the price floor. inference systems