Prefill and Decode Disaggregation

Chunked prefill bounds the interference between prompt processing and token generation by interleaving them. Disaggregation takes the more radical option: it stops interleaving entirely, puts prefill on one set of machines and decode on another, and hands the intermediate state across the boundary. The motivation is that the two phases have different bottlenecks and therefore different optimal configurations, and a machine can only be configured one way.

Why one configuration cannot serve both phases

Prefill over L positions performs L rows of every matrix multiplication. The arithmetic intensity is high, so the phase is limited by tensor-core throughput and benefits from large tiles, high parallelism within a pass, and as much compute as the accelerator offers. Decode performs one row per sequence. The arithmetic intensity is low, so the phase is limited by the rate at which weights can be read from memory, and additional compute changes nothing. What helps decode is more memory bandwidth and more room for cache, not more flops.

Tensor parallelism sits on opposite sides of this divide. Splitting a model across k accelerators divides the weight read by k, which is close to a k-fold speedup for decode until interconnect latency dominates. For prefill it divides the arithmetic too, but adds an all-reduce per layer whose cost grows with the number of participants and with message size. Prefill therefore prefers fewer, larger devices; decode prefers more devices with high-bandwidth links between them. A single deployment must pick a compromise, and the compromise is wrong for one of the phases.

The scheduling objectives diverge as well. A prefill request is served once and leaves; its latency contribution is time-to-first-token, and it holds no cache afterward. A decode sequence persists for hundreds of iterations and its latency contribution is inter-token latency, which is sensitive to how many sequences share the batch. Mixing them in one queue means one admission policy must arbitrate between two incompatible notions of cost, which is what Chunked Prefill does at the cost of a compromise chunk size.

The transfer that makes it possible

The interface between the phases is the key-value cache. Prefill produces, for a prompt of L tokens, the keys and values of all L positions at every layer. Decode needs exactly that to generate token L+1. If the two run on different machines, the cache must cross the interconnect.

Its size is fixed by model geometry. For a 32-layer model with 4,096-wide multi-head attention in 16-bit precision, one token costs 2 x 32 x 4,096 x 2 bytes, or 512 KiB. A 4,000-token prompt therefore produces 2 GiB of state to transfer. Grouped-query attention with eight key-value heads of width 128 reduces the per-token figure to 128 KiB, so the same prompt costs 512 MiB. That reduction, originally made for cache capacity, is what makes disaggregation practical at all.

The bandwidth arithmetic is unforgiving. A 100 Gb/s link delivers 12.5 GB/s at line rate, so 512 MiB takes roughly 40 ms, and 2 GiB takes roughly 160 ms. Compare with the decode step for a 7-billion-parameter model at 16-bit precision: 14 GB of weights read at several terabytes per second is a few milliseconds. A transfer that takes forty times a decode step is a serious addition to time-to-first-token, and it is why disaggregation is usually paired with high-bandwidth interconnect, with prompt lengths long enough that prefill dominates the transfer, or with both.

Transfer time can be overlapped with computation. A decode machine can begin generating as soon as the first layer’s cache arrives, provided the layers are streamed in order and the attention kernel for layer i waits only for layer i’s blocks. Pipelining the transfer against the forward pass hides part of the cost, at the price of a more complex protocol.

Placement and the pool

Two arrangements are common, and they differ in where the boundary sits.

The first keeps prefill and decode in separate pools with a router in front. A request arrives, is routed to a prefill worker, waits for its cache to arrive at a decode worker, and then streams tokens back. The router must choose both workers, which is a two-sided placement problem: the prefill choice is governed by prompt length and compute availability, the decode choice by cache capacity and the arrival’s expected output length. Decode workers must reserve capacity before the cache lands, so the router’s estimate of output length becomes an admission decision.

The second runs both phases on the same machines but time-shares them by configuration, switching a device between prefill and decode roles. This avoids the network transfer entirely and captures most of the configuration benefit, at the cost of a reconfiguration pause and of coarse-grained switching.

The pool arrangement wins when traffic has a wide range of prompt lengths and when interconnect is fast; the time-shared arrangement wins when the model is small enough that transfer cost is a large fraction of total work.

client -> router
           |-- prefill worker (compute-heavy, few devices)
           |      produces KV blocks for prompt
           |-- transfer over RDMA or high-bandwidth link
           `-- decode worker (bandwidth-heavy, many devices)
                  holds cache, emits one token per step

Failure modes specific to the split

The first is stranded state. Cache that has been produced by prefill but not yet accepted by decode occupies memory on the prefill machine, whose capacity was sized for computation rather than for storage. If the decode pool is saturated, prefill workers accumulate completed prompts they cannot release, and their throughput collapses. Backpressure from decode to prefill must be explicit, and the prefill worker must be able to refuse work before it computes a cache it cannot hand off.

The second is failure during transfer. A prefill worker that dies after computing cache but before the transfer completes leaves a request with no recoverable state; the work must be redone. Because prefill is the expensive part, this failure is worse than a decode failure, where only a suffix of tokens is lost. Checkpointing cache is not economical, so the practical answer is to restart the request and count it against a retry budget.

The third is ordering and correctness of the handoff. The receiving sequence’s block table must reference blocks in the same logical order the prefill produced them, and the sequence’s position counter must equal the prompt length. A mismatch produces attention over the wrong positions, which degrades output without raising an error. Protocol versions between the two pools must therefore be tied together, and a rolling upgrade that changes the cache layout cannot mix versions across the boundary.

The fourth is that decode-only machines still pay for the model’s full weights. Disaggregation does not reduce the parameter memory each device must hold; it only changes how many devices serve which phase. If the model barely fits on a decode device, there is no room for the cache that disaggregation exists to manage.

What it does and does not buy

Disaggregation improves goodput when the two phases have genuinely different optima, which happens for large models, long prompts, and high concurrency. It costs a network hop, a new failure mode, and a more complex control plane. For small models or short prompts the transfer is a large fraction of the work and the split is a loss.

It also composes with the rest of the architecture rather than replacing it. The decode pool still runs an iteration-level batching loop, still allocates cache in blocks, and still needs a router that accounts for outstanding work. What changes is that the prefill pool becomes a separate queueing system with its own scaling signal, since its load is measured in prompt tokens rather than in resident sequences. That distinction is the reason Autoscaling Signals for Token Servers treats prefill and decode capacity as separate quantities, and why placement decisions in Replica Packing and Topology Constraints must account for the fact that the two pools no longer share a shape.