Disaggregated Prefill and Decode Pools

Chunked prefill keeps both phases on one device and bounds the interference between them by rationing prompt tokens per step. That bound is a compromise: the prompt is processed at a fraction of the device’s width, and the decode batch is protected by giving up throughput it could otherwise have had. A different answer is to stop compromising and give each phase its own hardware. The prompt runs on devices configured for arithmetic throughput; generation runs on devices configured for memory capacity and bandwidth; the cache produced by the first is moved to the second.

The mechanism is simple to state and hard to make pay, because the handoff moves exactly the data that the decode pool was short of: the key-value cache. Whether disaggregation wins depends on whether the transfer costs less than the interference it removes.

Why the two phases want different machines

Prefill is compute-bound and its arithmetic intensity rises with prompt length. Its cost per token is dominated by matrix multiplication, so a device with high floating-point throughput and modest memory capacity is a good fit. The cache it produces must be held somewhere until the sequence is handed off, but only briefly.

Decode is bandwidth-bound and its cost per token is dominated by reading weights and cache. Its arithmetic intensity is low, so a device with large memory capacity and high bandwidth, and with whatever arithmetic comes with that, is the right fit. The model weights must be resident in full on every decode device, which is the dominant consumer of capacity.

A homogeneous pool must be sized for both requirements at once, which means every device carries full weights and enough arithmetic for prompt work that arrives unevenly. Under a workload where prompts are long and outputs are short, the decode devices are mostly idle; under the opposite, prefill devices are. The phases’ resource demands are not merely different in magnitude, they are different in kind, and a single device serves both at the cost of serving each below its own optimum.

There is also a batching argument. Prefill wants large batches of tokens with no requirement for low latency per token, since a prompt has one deadline rather than one per token. Decode wants a batch large enough to amortise weight reads and small enough to keep step time under the inter-token budget. Those two batch-size objectives are set by different constraints, and separating them lets each pool choose its own.

The handoff

When a prompt finishes on a prefill device, its cache blocks must reach the decode device that will continue the sequence. The transfer volume is the prompt’s full cache, which at the per-token figures used throughout this cluster is hundreds of megabytes for a long prompt. Moving that over a general-purpose network is the cost that disaggregation introduces.

Three transfer mechanisms are available, and they differ by roughly an order of magnitude in bandwidth.

Intra-node interconnect. Devices in one chassis connected by a high-bandwidth link can exchange cache blocks at rates comparable to device memory bandwidth, though not equal to it. A transfer of a few hundred megabytes takes a few milliseconds.

Remote direct memory access over a fast network. Between nodes, the cache moves over the network fabric. The bandwidth is lower and the latency is higher, but the transfer is one-sided and can be pipelined with computation on both ends.

Through host memory. The cache is staged through the host, which adds a copy in each direction and is the slowest of the three. It is the fallback when no direct path exists.

Layer-wise transfer changes the arithmetic. Rather than waiting for the entire prompt to be processed and then sending the whole cache, the prefill device can send each layer’s cache as soon as that layer completes. The decode device begins receiving layer 0’s cache while layer 1 is still being computed, so the transfer overlaps with the remaining prefill. For a prompt of $L$ layers, the transfer of the last layer is the only part that cannot be hidden, and it is $1/L$ of the total. This is the mechanism that makes the handoff affordable, and it requires the decode device to have the sequence’s cache blocks allocated before the prompt starts, so the destination is known in advance.

Overlap and the critical path

Even with layer-wise transfer, the handoff inserts a delay between the prompt’s last token and the sequence’s first generated token. That delay lands directly on time to first token, which is the metric the prefill pool exists to improve. The transfer time for a prompt of $T$ tokens at $c$ bytes per token over a link of bandwidth $W$ is $Tc/W$; a 10,000-token prompt at 128 KiB per token is 1.25 GB, which at a fast network’s per-device bandwidth is tens of milliseconds.

The comparison that decides the design is between that delay and the interference it removes. On a single device, a long prompt processed alongside decodes lengthens every decode step for the duration of the prompt’s chunks. If the prompt is 10,000 tokens and the chunk budget is 512 tokens, the decodes are disturbed for twenty steps, and each disturbance is the cost of 512 prefill positions. The aggregate disturbance can easily exceed the transfer delay, which is the regime where disaggregation wins.

Two refinements reduce the delay further. The decode pool can start generating from a partial cache, so the sequence begins emitting after the last layer’s cache arrives rather than after the whole cache arrives, if the transfer is ordered by layer. And the decode pool can accept the sequence into a batch that is already running, so the handoff costs one sequence’s worth of latency rather than a step boundary. Both depend on the two pools sharing a consistent notion of the sequence’s identity and position, which is a control-plane requirement rather than a data-path one.

Setting the capacity ratio

The central planning question is how many devices to give each pool. The answer follows from the workload’s ratio of prompt tokens to generated tokens, weighted by each phase’s throughput on its own hardware.

Let the prefill pool process $P$ prompt tokens per second in aggregate and the decode pool generate $G$ tokens per second. For the system to be balanced, the workload’s ratio of prompt tokens to output tokens must equal $P/G$. If the workload averages 2,000 prompt tokens and 200 output tokens, a ratio of ten to one, then the pools must be sized so the prefill aggregate is ten times the decode aggregate in token throughput. Since decode is the slower phase per token on a per-device basis, and prefill is faster, the device counts can still come out similar; the point is that the ratio is a computed quantity, not a default.

The ratio is also the system’s main fragility. A workload shift toward longer prompts saturates prefill while decode idles, and the queues in front of prefill grow without bound because there is no way to borrow decode capacity for prompt work, at least not without reconfiguring the model placement. This is the price of specialisation, and it is why disaggregation is a good fit for stable workloads with predictable phase ratios and a poor fit for mixed traffic.

There is a middle design that mitigates this: pools that are homogeneous in hardware but partitioned by role at any given moment, with the partition adjustable. A device can be reconfigured from decode to prefill by changing what it is scheduled to run, provided the model weights fit either way. The reconfiguration cost is the drain and refill of the batch, which is seconds rather than milliseconds, so the adjustment happens at the scale of traffic shifts and not per request.

Cache lifetime across pools

The cache’s ownership changes at the handoff, and that creates failure modes that neither pool has alone.

Orphaned cache. If the decode pool cannot admit the sequence because its own memory is full, the cache sits on the prefill device holding memory that pool needs. The prefill device must either hold it until the decode pool has room, which stalls prefill, or discard it, which means the prompt work is lost and the request restarts. Neither is acceptable at scale, so the decode pool’s admission decision has to be made before the prompt is processed, not after.

Duplicate residency. During the transfer both pools hold the cache. The peak memory requirement is therefore higher than the steady state, and a decode pool sized exactly to its steady-state need will fail at the moment of handoff. The transfer buffer must be accounted for in both pools’ capacity planning.

Partial failure. If the link drops mid-transfer, the decode pool holds a partial cache and the sequence cannot proceed. Resuming requires either retransmitting from the prefill pool, if it still holds the blocks, or recomputing the prompt. The former is why prefill pools often retain a sequence’s cache until the decode side acknowledges receipt, which is a reference-counting problem across two independent allocators.

Eviction races. A decode sequence may be evicted for memory pressure shortly after a handoff that cost a large transfer. The transfer is then pure waste. Admission policy on the decode side has to be more conservative for freshly handed-off sequences than for ones that have been running, or the system will spend its interconnect bandwidth moving caches that are immediately discarded. The mechanisms for that pressure are described in preemption by swap versus recomputation.

What disaggregation does not fix

It does not reduce total cache memory. The same bytes must be resident during generation; splitting the pools only changes which device holds them. It does not reduce the weight footprint, since every decode device still needs the whole model. It does not help a workload where prompts are short, because the transfer volume is then small but the handoff’s fixed latency is not, and a short prompt’s prefill is cheaper to do in place than to move.

What it does is decouple two scheduling problems that a single device must solve simultaneously. The prefill pool schedules prompt chunks against a first-token deadline and a compute budget. The decode pool schedules sequences against an inter-token deadline and a memory budget. Each has its own control loop, and each can be tuned without disturbing the other. The cost is a new resource on the critical path, the link between them, and a new failure mode, the sequence that exists in neither pool. Systems that adopt the topology are trading a scheduling problem they understand for a data-movement problem they must measure, and the trade only pays when the movement is cheap relative to the interference.