Batch Size, Occupancy, and the Marginal Cost of a Token

Cost per token is a quotient, and the denominator is throughput. Throughput is set by how much useful work each accelerator step carries, which is set by the batch. The relationship between the two is not linear in either direction: small batches leave the memory system and the arithmetic units underused, and large batches lengthen each step until latency becomes the binding constraint rather than cost. Between those ends there is a region where the marginal cost of an additional concurrent sequence is close to zero, and finding its edge is most of what capacity planning for a token service consists of.

The step as a shared cost

A decode step is a fixed charge plus a variable one. The fixed charge is the weight read: every parameter must be touched once per step regardless of how many sequences are in the batch. The variable charge is everything that scales with batch: KV cache traffic, attention arithmetic, and the activation buffers. Writing $W$ for the weight bytes, $k$ for the cache bytes per sequence, $b$ for the batch, and $B$ for effective bandwidth, the memory time of a step is approximately

$$t_{\text{mem}}(b) \approx \frac{W + b,k}{B}$$

and the cost per token delivered by that step, with $b$ tokens produced in total, is

$$\frac{t_{\text{mem}}(b)}{b} = \frac{W}{bB} + \frac{k}{B}$$

The first term is the fixed charge divided by the batch, and it falls as $1/b$. The second is the marginal cost of one more sequence, and it is constant. This decomposition is the entire economics of batching in one line: the average cost per token approaches the marginal cost from above, and the gap between them is the amortization of the weights.

At batch 1 the average cost is $W/B + k/B$, dominated by the weight read. At batch $b$ the weight component has shrunk by a factor of $b$. The improvement is steep at first and then flat: going from 1 to 2 halves the dominant term, from 8 to 16 removes another 6 percent of the original, and from 64 to 128 removes under 1 percent. The knee is where $b,k$ becomes comparable to $W$, which is the crossover already derived in the KV cache budget: the point where the cache traffic of the batch equals the weight traffic.

What the model leaves out

The clean $1/b$ curve assumes the step time is constant in batch, which holds only in the bandwidth-bound regime. Once arithmetic becomes the limit, step time grows with batch and the cost per token stops improving and starts to worsen. For a batch of $b$ sequences at context length $L$, the attention arithmetic per step is roughly $4,n_{\text{layers}},d_{\text{model}},L,b$ flops, linear in batch; the weight arithmetic is $2Pb$, also linear. Total flops scale linearly with $b$, so compute time scales linearly too, and the ratio of compute time to memory time is constant in $b$. What changes is which of the two is larger. The optimum sits where they are equal, and pushing the batch past it buys throughput at the price of proportionally longer steps with no reduction in cost per token.

Latency adds a second limit. A step that serves 128 sequences takes longer than one that serves 8, even in the bandwidth-bound regime, because the arithmetic portion has grown. Time to first token is largely unaffected by decode batch size, since prefill is scheduled separately, but inter-token latency is roughly the step time, and a user waiting on a streamed response perceives it directly. The batch that minimizes cost per token is therefore usually larger than the batch that meets a latency target, and the gap between them is the price of responsiveness.

Three further effects push the practical optimum below the theoretical one. Attention over long contexts produces large intermediate tensors, so activation memory grows with batch times context, and the allocator may have to shrink the batch to fit. Kernel launch and scheduling overhead per step is roughly constant, which matters at small batch but not at large. And the tail of the batch matters more than the mean: a batch of 128 sequences in which one has 30,000 tokens of context costs the memory of the long one for every step it remains resident, so a batch assembled by count rather than by bytes can overshoot the budget.

The allocator’s behavior also feeds back into the cost curve in a way that is easy to miss. Admitting a sequence requires reserving cache blocks before the sequence’s eventual length is known, so admission is a gamble on how much memory the request will need. A conservative admission policy keeps the batch below the memory limit at all times and leaves blocks unused; an aggressive one fills memory and then has to preempt when a resident sequence grows. Preemption is expensive in exactly the currency the batch was meant to save: a preempted sequence must either wait for memory to free, which lengthens its response time, or be recomputed from its prompt, which spends prefill flops a second time. Both outcomes lower the effective tokens per hour, so the cost curve has a cliff at the memory limit rather than a smooth maximum.

Continuous batching and the difference between the two curves

Static batching, where a batch is assembled, run to completion, and then replaced, has a ragged profile. Sequences in the batch finish at different times; finished slots sit idle while the slowest sequence runs to its last token. The average occupancy over the batch’s lifetime is well below the nominal batch size, and the cost per token reflects the average, not the nominal.

Continuous batching replaces the batch every step. Finished sequences leave, new ones join, and the batch size is recomputed at each iteration. Two consequences follow for the cost model. The achieved occupancy is close to the configured maximum at every step, so the $1/b$ term uses the real batch rather than a nominal one. And the mix of context lengths in the batch changes continuously, so the marginal cost $k$ is a running average over the resident sequences rather than a property of any one request. A scheduler that admits by remaining-token budget rather than by sequence count keeps that average stable.

The same mechanism explains why output length is the hidden cost driver. Two requests with identical prompts and different output lengths occupy the batch for very different numbers of steps, and the long one pays its share of the weight read for every step it stays. Cost per output token is roughly flat in output length for a given request, but cost per request is linear in it, and the aggregate cost of a workload is the sum over requests of steps times the per-step charge divided by the batch at each step.

Choosing a batch from a cost target

The following computes the knee of the curve for a given model and memory system. It is arithmetic on the model’s own constants, not a measurement.

def cost_curve(params, layers, d_kv, ctx, w_bytes, kv_bytes,
               bandwidth, flops, batch_max):
    W = params * w_bytes                    # weight bytes read per step
    k = 2 * layers * d_kv * kv_bytes * ctx  # cache bytes per sequence
    rows = []
    for b in (1, 2, 4, 8, 16, 32, 64, 128, 256):
        if b > batch_max:
            break
        t_mem = (W + b * k) / bandwidth
        t_flop = (2 * params * b + 4 * layers * d_kv * ctx * b) / flops
        step = max(t_mem, t_flop)
        rows.append((b, step, step / b))    # batch, seconds, seconds/token
    return rows

# the knee is the smallest batch where t_flop >= t_mem,
# or where step/b stops improving by more than a few percent

Read the table by looking for the batch where the third column flattens. Doubling the batch beyond that point does not halve the per-token time; it holds it roughly constant while doubling the step latency. Everything past the knee is a latency-for-nothing trade, and the batch configuration should sit at or just below it.

There is a caveat that changes the answer for real traffic: the knee depends on context length, so a workload with mixed lengths has a knee that moves. Short sequences want a larger batch by count, long sequences want a smaller one. A single configured maximum cannot be optimal for both, which is why serving systems increasingly schedule by token budget per step rather than by sequence count, a decision that also determines the concurrency the capacity plan can assume.

The floor the curve implies

Because the marginal cost term $k/B$ is constant in batch, no amount of batching can push cost per token below it. It is the price of reading one sequence’s cache from memory, and it falls only if the cache gets smaller, the bandwidth gets larger, or the sequence gets shorter. That floor is what makes shrinking the cache an economic lever rather than a micro-optimization: it attacks the constant term, not the amortizable one, and the constant term is what remains after all the batching gains have been taken. Narrowing the cached elements, as discussed in the memory budget, halves $k$ directly.