Prefill and Decode: The Two Cost Curves Behind One Token Price

A price list that quotes one number per token describes an average of two operations with opposite bottlenecks. The first operation, prefill, consumes the prompt and produces the first output token; it is a small number of large matrix multiplications and behaves like a compute workload. The second, decode, produces one token per sequence per step; each step must read every weight in the model to perform a handful of arithmetic operations on each of them, and behaves like a memory-streaming workload. The cost of a request is the sum of the two, weighted by how many tokens each phase handles.

Getting the split wrong produces capacity plans that look sound on paper and fail on contact: a pool sized from a token-per-second target will be short of arithmetic for prompt-heavy traffic and short of bandwidth for output-heavy traffic, because those are different resources.

Two rooflines instead of one

The roofline model reduces any kernel to two limits. Peak floating-point throughput, $F$ flops per second, bounds the arithmetic. Peak memory bandwidth, $B$ bytes per second, bounds the data movement. A kernel that performs $W$ flops while touching $M$ bytes cannot run faster than $\max(W/F,\ M/B)$. The ratio $W/M$ is arithmetic intensity, and the crossover point $F/B$ is a property of the hardware: on an accelerator with tens of teraflops and hundreds of gigabytes per second, the crossover sits somewhere around a few hundred flops per byte. Kernels far above it are compute-bound; kernels far below it are bandwidth-bound and waste most of the arithmetic units.

Prefill sits far above the crossover. Processing $N$ prompt tokens through a dense model of $P$ parameters costs roughly $2PN$ flops, because each weight participates in one multiply and one add per token. The bytes touched are the weights themselves, $2P$ for a half-precision model, plus activations that scale with $N$ but stay small relative to the weights. Intensity is therefore about $N$ flops per byte, growing linearly with prompt length. A prompt of a few thousand tokens puts prefill firmly in the compute-bound regime, where the achievable rate is close to the peak and batching changes little.

Decode sits far below it. One step reads the entire weight tensor to produce a single token per sequence, so the intensity is a small constant independent of sequence length: on the order of one or two flops per byte. The arithmetic units are idle for most of the step; the accelerator waits on the memory system. This is why a decode step is often described as bandwidth-limited rather than latency-limited, and why its duration is roughly predictable from the model size alone:

$$t_{\text{step}} \gtrsim \frac{2P}{B}$$

For a model whose weights occupy tens of gigabytes and a memory system delivering a few terabytes per second, the quotient lands in the tens of milliseconds per step. The figure moves with precision: halving weight bytes with 8-bit quantization roughly halves the floor on step time, which is the main economic argument for narrowing the weights. The same reasoning applies to the cache, whose element size enters the byte count directly, as the memory budget shows.

Why batching changes the decode curve and not the prefill one

During prefill, adding a sequence to a batch adds work proportional to its length. Doubling the prompt batch roughly doubles the time, so the cost per token is flat and the benefit of batching is confined to whatever efficiency the larger matrix shapes recover.

During decode, adding a sequence to a batch adds almost no weight traffic. The same weight read serves every sequence in the batch, so the bandwidth cost of a step is amortized over the batch: the marginal cost of an extra sequence is its KV cache read plus its share of the arithmetic, and at small batch sizes that share is negligible. Cost per output token therefore falls nearly hyperbolically with concurrency, until the batch is large enough that arithmetic becomes the limit or until KV cache traffic overtakes weight traffic.

That second crossover is the one that surprises capacity models. Each sequence in the batch has a KV cache of $4 \cdot n_{\text{layers}} \cdot d_{\text{model}} \cdot L$ bytes at context length $L$ in half precision, counting both keys and values at two bytes per element. Reading that cache every step costs bytes proportional to batch times context length, while reading the weights costs bytes proportional to parameters. The batch at which KV traffic equals weight traffic satisfies

$$b \cdot 4 n_{\text{layers}} d_{\text{model}} L \approx 2P$$

Beyond it, decode is no longer amortized by batching, and the cost per token starts rising with context length instead of falling with concurrency. The memory budget article works this inequality through as a planning constraint rather than a performance note.

A cost model with the right two terms

The following estimate is deliberately crude; its purpose is to make the two terms visible and to show which input dominates a given workload. It assumes no communication overhead, no kernel inefficiency, and perfect overlap of compute and memory.

def step_cost(params, layers, d_model, batch, ctx, w_bytes,
              kv_bytes, peak_flops, bandwidth):
    # bytes moved: weights once per step, KV cache per sequence
    weight_bytes = params * w_bytes
    kv_bytes_total = batch * 4 * layers * d_model * ctx * kv_bytes
    decode_s = max(weight_bytes, kv_bytes_total) / bandwidth

    # arithmetic: 2 flops per parameter per token, plus attention terms
    per_step_flops = 2 * params * batch + 4 * layers * d_model * ctx * batch
    decode_s = max(decode_s, per_step_flops / peak_flops)

    prefill_flops = 2 * params * ctx + 2 * layers * d_model * ctx * ctx
    prefill_s = prefill_flops / peak_flops

    return prefill_s, decode_s

# long prompt, short answer: prefill dominates the bill
# short prompt, long answer: every output token pays a decode step

Two things follow from the shape of the expression. First, a request with a long prompt and a short answer pays mostly for one compute-bound pass, and its cost per input token is close to the arithmetic cost. Second, a request with a short prompt and a long answer pays a full weight sweep for every output token, and its cost is set by bandwidth. Published price sheets generally reflect this by charging several times more for output tokens than for input tokens, though the exact ratio depends on model size, context length, and how much of the input can be served from cache.

Where the model stops being useful

Three conditions break the clean two-phase picture. Mixture-of-experts models read only the active experts per token, so the effective $P$ in the decode term is the active parameter count while the memory footprint is the total; decode cost falls but capacity is still consumed by weights that sit idle. Chunked prefill interleaves prompt work with decode steps on the same batch, which improves arithmetic utilization at the cost of longer per-step latency for the sequences being decoded, and it blurs the accounting because a single forward pass now contains both kinds of work. Prefix caching removes prefill work entirely for repeated prompt prefixes, which is why cache hit rate becomes a first-class term in the cost model rather than an optimization detail.

Multi-device execution adds a fourth wrinkle. When a model is split across accelerators, either by tensor parallelism within a layer or by pipeline parallelism across layers, every phase pays a communication cost that has no analogue on a single device. For tensor parallelism the cost is an all-reduce of activations per layer, and its size is proportional to batch times width, so it scales with the work rather than staying fixed. Pipeline parallelism instead introduces bubbles: stages idle while waiting for the previous stage’s output, and the fraction of idle time falls as the number of in-flight micro-batches rises. Both arrangements change the constant in the cost model without changing its form, which is why plans that were validated on one device topology do not transfer directly to a multi-device deployment.

The one thing the model gets right in every regime is that cost per token is not a property of the model alone. It is a property of the model, the hardware roofline, the batch, and the context length at which the token was produced, and a capacity plan that fixes only the first two is incomplete. The amortized cost accounting takes these per-request quantities and folds them into an hourly cost of ownership, where the same two phases reappear as the throughput denominator.