Why Token Billing Diverges from GPU-Seconds

Two accounting systems are in use for the same service. One measures resource consumption: seconds of accelerator time, bytes of memory held, kilowatt-hours. The other measures output: tokens of input and tokens of output, priced from a rate card. They agree on average, because the rate card is set from the resource cost of a representative workload. They disagree on individual requests, and the disagreement is systematic rather than random. Understanding the direction of the disagreement explains which workloads are underpriced and which are subsidized.

The unit mismatch

A GPU-second is a unit of occupancy. It accrues whenever a replica is resident, whether it is serving one request or a hundred, and whether the request it serves is a 10-token completion or a 100,000-token document. A token is a unit of output, and its price is set as though every token cost the same to produce. Neither unit is wrong; they measure different things, and the mapping between them depends on the workload.

Consider two requests served by the same replica. The first has a 200-token prompt and a 5-token answer. The second has a 200-token prompt and a 2,000-token answer. Under GPU-second accounting, the second dominates: it holds a sequence slot for 2,000 decode steps, and the replica’s occupancy is charged to it in proportion to those steps. Under token accounting at a single blended rate, the difference in charge is 400 times the token count, and if output tokens are priced higher than input tokens, the ratio between the two requests’ charges is far smaller than the ratio of their resource consumption.

The divergence has a clean expression. Resource cost of a request is approximately

$$\text{cost} \approx \frac{t_{\text{prefill}}(N)}{b_{\text{prefill}}} + \frac{t_{\text{step}} \cdot G}{b_{\text{decode}}}$$

where $N$ is prompt length, $G$ is output length, and the denominators are the batch sizes at which each phase ran. Token charge is $p_i N + p_o G$ for input and output prices. The first expression is nonlinear in both lengths and depends on the batch at the moment of service; the second is linear and depends only on lengths. The two can only agree on average, and only for a workload whose length distribution matches the one used to set prices.

Prefill is nearly linear, decode is not

For a single request, prefill cost per token is close to constant: $2P$ flops per token, so the arithmetic is linear in prompt length. What breaks linearity is batching. If prefill runs at a batch of one, the weights are read once for that request and the full weight-read cost lands on its prompt. If prefill runs at a batch of sixteen, the same weight read is shared and the per-token cost falls by a factor of sixteen. Two requests with identical prompts can therefore have resource costs differing by an order of magnitude, purely because of when they arrived relative to other traffic.

Decode is more strongly nonlinear. Each output token requires a step that reads the full weight tensor, and the cost of that step is shared across the batch. At batch 1, every output token carries a full weight read. At batch 32, each carries one thirty-second of it. The per-token price cannot reflect this, because the batch at service time is not known when the price is set. The rate card is therefore an average over the batch distribution, and any request served at a batch far from the mean is mispriced in one direction or the other.

Context length adds a second nonlinearity. The attention term in the decode step grows with context, and the KV cache traffic grows with context as well, so a token generated at position 30,000 costs more than a token generated at position 100. A flat per-token price charges them identically. Systems that price by context tier, with a higher rate above some length, are correcting exactly this, and the tier boundaries are chosen to approximate where the cost curve bends.

The workloads that land on either side of the average

A request is underpriced relative to its resource cost when it consumes more of a shared resource than the average request the price was set from. Long outputs at low batch are the clearest case: a single request streaming thousands of tokens on an otherwise idle replica pays a per-token price that assumes the replica was shared. Long prompts are a second case, for the same reason applied to prefill. Long contexts are a third, because the attention and cache costs scale with position while the price does not.

A request is overpriced when it consumes less: short prompts and short outputs served at high batch, especially when a prefix cache removes most of the prefill. The overpricing is real but less visible, because it appears as margin rather than as a loss. The cross-subsidy runs from high-volume, short-request traffic toward low-volume, long-request traffic, and the magnitude of the subsidy is the difference between the two requests’ resource costs at their respective batches.

The direction of the cross-subsidy matters for capacity planning. A service whose traffic mix shifts toward long outputs will see its resource cost per token rise while its revenue per token stays fixed, and the shift is invisible in token counts. Tracking accelerator-seconds per million tokens, alongside the token volume, is the measurement that exposes it: the ratio rises when the mix moves toward expensive requests, even if the total token count is flat. The amortized cost analysis treats this ratio as the fundamental unit of cost, with the rate card as a projection of it.

Attribution inside a batch

A single decode step serves every sequence in the batch, and the step’s cost must be divided among them. Three divisions are defensible and they produce different answers.

Equal division charges each sequence $1/b$ of the step. This is simple and it undercharges long-context sequences, which contribute more KV traffic and more attention arithmetic than short ones. The step time is a max over resources, though, so the marginal cost of one more short sequence is genuinely near zero when the batch is bandwidth-bound on weights; equal division overcharges the short sequences in that regime and undercharges them when the batch is compute-bound.

Proportional division by cache bytes charges each sequence in proportion to its KV footprint. This reflects memory occupancy, which is the resource that limits concurrency, but it ignores the weight traffic that the batch amortizes. A sequence with a tiny cache still benefits from the shared weight read, and charging it only for cache bytes gives it the amortization for free.

Proportional division by flops charges each sequence for its arithmetic contribution, which is the right measure in a compute-bound regime and the wrong one in a bandwidth-bound regime. Since real serving alternates between the two as the batch and context vary, no fixed attribution rule is correct throughout. A common compromise is to charge the shared weight read equally and the variable cache and attention terms proportionally, which at least gets the direction right in both regimes.

def attribute_step(params, layers, d_kv, ctx, batch, w_bytes,
                   kv_bytes, bandwidth, flops):
    W = params * w_bytes
    shared = W / bandwidth / batch          # weight read, split equally
    out = []
    for c in ctx:
        kv = 2 * layers * d_kv * kv_bytes * c
        attn = 4 * layers * d_kv * c / flops
        out.append(shared + kv / bandwidth + attn)
    total = sum(out)
    return [x / total for x in out]         # each sequence's share of the step

# shares sum to 1; note they change as batch and context change

The shares produced by this function are not stable across a request’s lifetime. As the batch fills and empties, and as context grows, the same request’s share moves. That instability is inherent: the cost of serving a request genuinely depends on what else was being served at the same time, and any attribution scheme is a convention rather than a measurement.

What a rate card can and cannot express

A rate card with two prices, one for input and one for output, can express the average ratio between prefill and decode costs for a representative workload. It cannot express batch dependence, context dependence, or cache hit rates. Rate cards that add tiers for context length and a separate cached-input price recover part of that expressiveness, at the cost of a more complex price sheet and boundary effects at the tier edges; the reasoning behind the cached-input discount is worked through in the cache economics.

The alternative is billing for resource consumption directly, in accelerator-seconds, which is accurate and unpopular because the customer cannot predict the charge from the request. The compromise in wide use is to price tokens but to shape traffic so that the batch and context distribution stays near the one the prices were set from: admission control that limits concurrent long-context requests, routing that sends long prompts to replicas configured for prefill, and separate pools for interactive and batch traffic. Those controls are capacity-planning decisions, and their justification is that they keep the accounting system and the resource system from drifting apart.