Inference Systems · Cluster
Serving Architectures
Batching, routing, and model placement.
- 01 Iteration-Level Batching: Reforming the Batch at Every Token How a serving loop rebuilds its batch before every forward pass, why request-level batching wastes half the machine, and which budget actually limits concurrency.
- 02 Paged KV Cache Allocation and the Block Table Why reserving contiguous cache for a worst-case output length caps batch size, and how fixed-size blocks with an indirection table remove most of that waste.
- 03 Chunked Prefill and the Interference It Removes Prefill is compute-bound and decode is bandwidth-bound, so running them in the same iteration makes each wait on the other. Splitting prompts into chunks is the standard remedy.
- 04 Least-Outstanding Routing When Request Costs Vary Round robin assumes equal service cost and equal replica capacity. Token generation breaks both assumptions, so routers score replicas by work in flight instead of counting requests.
- 05 Prefix-Cache-Aware Routing and Cache Locality A shared system prompt makes the same prefill work repeat on every replica. Routing by longest cached prefix trades load balance for a large reduction in prefill cost.
- 06 Prefill and Decode Disaggregation Prefill saturates arithmetic and decode saturates bandwidth. Putting them on separate machines lets each be sized for its own bottleneck, at the cost of moving the KV cache over a network.
- 07 Deadline-Aware Batching and Tenant Fairness When a batch is full, admission becomes an arbitration problem. Deadline slack, tenant shares, and starvation bounds each answer a different question about who gets the next slot.
- 08 Autoscaling Signals for Token Servers CPU utilization and request rate both mislead when capacity is pinned by KV cache and iteration time. Queue depth in tokens, cache occupancy, and time-to-first-token measure what matters.
- 09 Replica Packing and Topology Constraints Tensor parallelism turns a model into a group of devices that must be co-located and upgraded together. That group, not the individual replica, is the unit a scheduler can place.
- 10 Adapter Residency When One Replica Serves Many Fine-Tunes Low-rank adapters let dozens of fine-tunes share a base model, but each token still needs one adapter applied. Residency, batch grouping, and routing decide whether that sharing pays.