Inference Systems · Cluster
Accelerator Scheduling
Memory residency and utilization.
- 01 Iteration-Level Batching and the Cost of a Fixed Batch Why a batch fixed at admission time strands accelerator capacity, and how re-deciding the batch at every decode step changes the cost of a token.
- 02 How Key-Value Cache Bytes Set the Batch Ceiling The byte arithmetic that decides how many sequences fit beside the weights, why the ceiling moves with context length, and what remains after it is reached.
- 03 Splitting Prefill From Decode Inside a Step Prompt processing is compute-bound and token generation is bandwidth-bound; how mixing them in one step costs latency, and how chunked prefill bounds the interference.
- 04 Paged Key-Value Residency and Block Tables How fixed-size cache blocks and per-sequence block tables remove the allocation and fragmentation penalties of contiguous cache extents.
- 05 Admission Control and the Memory Watermark Reserving headroom so that growth does not become preemption: how watermarks, reservation bounds, and head-of-line blocking interact in a cache pool.
- 06 Preemption by Swap Versus Recomputation When the cache pool cannot serve a running sequence, the scheduler must take memory back; the two recovery strategies have different costs that depend on sequence length.
- 07 Prefix Caching and Shared Residency Deduplicating key-value blocks across requests that begin with the same tokens, how blocks are matched by hash, and what eviction must preserve.
- 08 Tensor-Parallel Residency and Interconnect Pressure Sharding a model across devices splits weights and key-value heads, but every decode step becomes a collective; how that changes the memory and latency balance.
- 09 Deadline-Aware Batch Assembly Turning latency targets into per-step scheduling decisions: slack, remaining-work estimates, and why maximising throughput and meeting deadlines diverge.
- 10 Disaggregated Prefill and Decode Pools Separating prompt processing from token generation into distinct pools, the cache handoff that joins them, and how the capacity ratio between pools is set.