Chunked Prefill and the Interference It Removes

A transformer request is two workloads wearing one interface. Prefill processes the whole prompt in a single pass, with L positions of work in a matrix multiplication that is dense and large. Decode processes one position per sequence, with a matrix multiplication whose operand is a single row. The first is limited by arithmetic throughput; the second is limited by memory bandwidth, because every weight must be read to produce one token.

Mixing them in the same iteration looks harmless and is not. When a long prompt is admitted alongside forty decoding sequences, the iteration’s duration is set by the prompt’s prefill work, and all forty sequences wait that long before their next token. Inter-token latency for those sequences spikes to the duration of a prefill. The spike is correlated with arrivals, so it appears as irregular stalls under traffic that is otherwise steady.

The arithmetic of a stall

Consider a 7-billion-parameter model in 16-bit precision: 14 GB of weights. Decode reads all of them to produce one token per sequence, so at a memory bandwidth of roughly 3 TB/s the floor is about 5 ms per step, nearly independent of how many sequences share the pass until arithmetic becomes the limit. Forty sequences therefore advance at about 200 tokens per second each.

Now admit a 4,000-token prompt into the same iteration. Prefill over 4,000 positions does 4,000 times the position-work of a single decode row, and its matrix multiplications are efficient enough that the pass is compute-bound. On an accelerator with a few hundred teraflops of 16-bit throughput, the pass takes tens of milliseconds. Every one of the forty decoding sequences experiences a gap of that size. If prompts arrive every few hundred milliseconds, the tail inter-token latency is dominated not by the decode loop but by prefill bursts, and the average latency figure understates what a client observes.

The reverse interference is smaller but real. Decode rows inside a prefill-dominated kernel occupy registers and shared memory that the prefill tile would otherwise use, so the prefill runs slightly below its peak. Neither side gains from the pairing.

Chunking as a scheduling unit

Chunked prefill splits a prompt into segments of bounded length and processes one segment per iteration, mixed with the decode batch. The bound is chosen so that a segment’s work is comparable to the decode work in the same iteration, which keeps iteration duration roughly constant.

The mechanism is a scheduler change, not a model change. A request in prefill state carries a cursor: how many prompt tokens have been processed. Each iteration, the scheduler admits a segment of at most C tokens from the front of the prompt, appends those positions to the batch, and advances the cursor. Attention within the segment reads the keys and values of all earlier positions, which are already in cache, so the segment’s attention is over a growing prefix rather than over itself alone. When the cursor reaches the prompt length, the request transitions to decode and contributes one token per iteration.

Two details make this correct rather than merely plausible.

The first is that attention must be computed against the full prefix. A chunk of positions p..p+C-1 computes queries for those positions and keys for those positions, but attends over blocks 0..p-1 as well, which are resident. The kernel therefore needs a causal mask that is offset by p: position p+i may attend to everything up to p+i and nothing after. Getting the offset wrong produces a subtly wrong model output rather than a crash, which is why chunked prefill is implemented inside the attention kernel rather than by concatenating slices.

The second is that the residual stream must be consistent across chunks. Layer outputs for a chunk are computed from that chunk’s inputs only; there is no cross-chunk dependency in the feed-forward or normalization path, because those are position-wise. Only attention crosses the boundary, and it does so through the cache. This is what makes arbitrary chunk boundaries legal.

def build_iteration(self, budget):
    batch = []
    # decode first: running sequences need exactly one position each
    for seq in self.running:
        batch.append(("decode", seq, 1))
        budget -= 1
    # then spend whatever remains on prefill segments
    for req in self.prefill_queue:
        if budget <= 0:
            break
        take = min(self.chunk_size, req.prompt_len - req.cursor, budget)
        batch.append(("prefill", req, take))
        req.cursor += take
        budget -= take
    return batch

The order matters. Placing decode first guarantees that running sequences get their token before prefill consumes the budget, which bounds inter-token latency by the decode work plus one chunk rather than by an entire prompt.

Choosing the chunk size

The chunk size is a direct trade between two latencies and one throughput figure.

A larger chunk finishes each prompt sooner, which reduces time-to-first-token, and gives the prefill kernel longer tiles to work with, which improves arithmetic efficiency. A smaller chunk shortens the worst-case iteration and therefore tightens inter-token latency for everyone else. The two objectives pull in opposite directions, and the resolution is to set the chunk so that a prefill segment costs approximately the same as the decode batch it accompanies.

There is a second-order effect worth naming. Attention cost per position grows with prefix length, so a chunk near the end of a very long prompt costs more than the same chunk near the beginning. Fixed chunk sizes therefore produce iteration durations that drift upward as prompts advance. Some schedulers compensate by shrinking chunks late in a prompt, at the cost of a longer tail for that request.

Throughput suffers slightly under chunking because each segment re-reads weights that a single large prefill would have amortized over more positions. The loss is small when chunks are in the hundreds of tokens, since the weight read is the same order as the arithmetic either way. It becomes visible when chunks fall to a few tens of tokens, where the pass is bandwidth-bound and the kernel never reaches its arithmetic roof.

Where the scheme stops helping

Chunking bounds the interference between prefill and decode, but it does not remove the underlying asymmetry. If decode sequences are numerous enough to consume the whole iteration budget, no prefill segment is admitted and prompts wait indefinitely. Admission policy has to reserve a share for prefill, and the choice of that share is a fairness decision discussed in Deadline-Aware Batching and Tenant Fairness.

A third limit is the arithmetic of the segment itself. A segment of 512 positions against a prefix of 8,000 does attention over 8,512 keys per position, so the attention term scales with prefix length times chunk length while the feed-forward term scales only with chunk length. At long prefixes, attention dominates and the chunk’s cost is no longer proportional to its size, which defeats the purpose of a fixed bound. Serving stacks that support very long contexts therefore reduce the chunk size as the cursor advances, or accept that iteration duration grows toward the end of a long prompt.

The mechanism also assumes the cache for earlier prompt positions is resident. For prompts longer than the available cache, chunking cannot help: the early blocks must be present when the late chunk attends to them. Long-context serving therefore depends on the allocation policy in Paged KV Cache, and on the option of moving prefill to separate hardware described in Prefill and Decode Disaggregation.

Finally, chunking changes the meaning of a scheduling iteration. The loop no longer advances every sequence by one position; it advances each by a different amount. Any component that assumed a fixed stride, including graph capture with fixed shapes and some forms of speculative decoding, must be adapted to variable per-sequence increments.