Speculative Decoding Cost Accounting: The Acceptance Rate Threshold

A decode step costs one full pass over the weights and produces one token per sequence. That ratio is the inefficiency speculative decoding attacks: if the weights must be read anyway, the arithmetic that follows them has room for more than one token of work. Proposing several tokens cheaply and then verifying them in a single pass over the target model converts the extra arithmetic into additional output, provided enough proposals survive verification. The cost question is how many survive, and the answer determines whether the technique saves anything at all.

The verification pass and what it guarantees

A draft model, smaller than the target and running the same tokenizer, generates $k$ candidate tokens autoregressively. The target model then evaluates all $k$ candidates in one forward pass, computing its own distribution at each of the $k$ positions. A candidate is accepted at position $i$ if a uniform sample falls within the ratio of target to draft probability for that token, and the first rejection truncates the accepted prefix. The scheme is exact: the tokens produced follow the target model’s distribution, not an approximation of it. That exactness is why the technique can be applied without changing output quality, and it distinguishes it from approaches that simply generate with a smaller model.

The pass itself costs what a prefill of $k$ tokens costs, not what a decode step costs. That is the crucial accounting detail: the verification is compute-bound work over $k$ positions, and its weight traffic is the same as a single decode step’s. So the cost of producing $n$ accepted tokens is one weight read plus $k$-token prefill arithmetic, against $n$ weight reads for the baseline. The saving comes from amortizing weight traffic across the accepted tokens; the cost is the added arithmetic and the draft model’s own compute.

The acceptance rate, and the tokens per step it yields

Let $\alpha$ be the probability that a single candidate is accepted, assumed roughly constant across positions for a first approximation. With $k$ candidates, the expected number of accepted tokens is

$$E[\text{accepted}] = \frac{1 - \alpha^{k+1}}{1 - \alpha}$$

which for $\alpha$ near 1 approaches $k+1$ and for small $\alpha$ approaches 1. The formula shows the diminishing return directly: at $\alpha = 0.8$, one candidate yields 1.8 tokens per step, two yield 2.44, four yield 3.36, and eight yield 4.33. Each doubling of $k$ buys less than the previous one, while the verification arithmetic grows linearly in $k$.

The acceptance rate is not a property of the draft model alone. It depends on how well the draft’s distribution matches the target’s on the specific text being generated. Code with rigid syntax and repeated identifiers tends to accept at high rates; open-ended prose with many equally plausible continuations accepts at lower ones. Temperature matters too: sampling from a flat distribution makes the draft’s choices harder to predict, so acceptance falls as temperature rises. Any figure quoted for a model pair is conditional on the workload, and treating a published number as a property of the technique rather than of a corpus is the most common error in planning with it.

The break-even condition

Let $c_d$ be the cost of one draft step, expressed as a fraction of a target decode step, and let $c_v$ be the cost of the verification pass, also as a fraction of a target decode step. For a target decode step normalized to 1, the speculative scheme costs $k c_d + c_v$ and produces $E[\text{accepted}]$ tokens. The baseline costs $E[\text{accepted}]$ steps to produce the same tokens. The scheme is worthwhile when

$$k c_d + c_v < E[\text{accepted}]$$

Substituting the acceptance formula gives a condition on $\alpha$ for each $k$. The interesting feature is that both $c_d$ and $c_v$ depend on the regime. In a bandwidth-bound decode, the draft model’s weights are much smaller than the target’s, so $c_d$ can be a small fraction; the verification pass costs roughly one target weight read plus arithmetic proportional to $k$, so $c_v$ is close to 1 for small $k$ and grows as $k$ grows. The inequality is therefore easiest to satisfy at moderate $k$ and moderate-to-high $\alpha$, and it can fail at both ends: small $k$ leaves too little amortization, large $k$ spends too much on verification.

At large batch, the picture inverts. When the target model is already compute-bound because the batch is large, the verification arithmetic is no longer free capacity; it is the scarce resource, and the scheme adds work to a resource that is already the limit. The batch economics analysis predicts this: past the knee, throughput is set by flops, and speculative decoding increases flops per accepted token. Deployments therefore often see the technique help at low concurrency, where latency matters and bandwidth is idle, and hurt at high concurrency, where throughput matters and arithmetic is saturated. The same configuration cannot be optimal for both.

Draft cost in the three common arrangements

A separate draft model is the original arrangement: a small model of the same family, trained on the same data, running its own forward passes. Its cost is a weight read of its own size per draft step, and its quality is bounded by how well a smaller model approximates a larger one. The ratio of sizes sets both $c_d$ and $\alpha$, and they move in opposite directions, so there is an optimum: a draft too small is cheap but accepts rarely, and a draft too large accepts often but costs nearly as much as the target.

A multi-head arrangement attaches extra prediction heads to the target model itself, predicting several future positions from the target’s own hidden state in a single pass. There is no separate model and no separate weight read, so $c_d$ is close to zero; the cost is the added parameters and the training required to make the heads accurate. The acceptance rate is lower than a dedicated draft model’s in many settings, but the near-zero draft cost shifts the break-even threshold down substantially.

Self-speculation reuses part of the target model, for instance by skipping layers or by using a subset of the parameters, which keeps the draft within the same weight set. The accounting is similar to the multi-head case: the marginal cost of drafting is small, and the technique’s value hinges on acceptance.

def expected_tokens(alpha, k):
    return (1 - alpha ** (k + 1)) / (1 - alpha) if alpha < 1 else k + 1

def break_even_alpha(k, c_draft, c_verify):
    """Smallest alpha at which k drafts plus verification beat plain decode."""
    lo, hi = 0.0, 0.9999
    for _ in range(40):                      # bisection on the monotone condition
        mid = (lo + hi) / 2
        if k * c_draft + c_verify < expected_tokens(mid, k):
            hi = mid
        else:
            lo = mid
    return hi

# c_draft and c_verify are measured as fractions of one target decode step

The threshold this returns is the number to compare against an observed acceptance rate. If the observed rate sits above it, the scheme is a net saving in the regime where the fractions were measured; if below, it is a net loss that the latency profile may still justify at low concurrency.

Latency and the throughput that is not saved

Speculative decoding is usually justified by latency rather than by throughput. At batch 1, a decode step is bandwidth-bound and the accelerator is mostly idle, so verification arithmetic costs almost nothing in wall-clock terms. Producing three tokens per step instead of one cuts inter-token latency by roughly a factor of three for the accepted fraction. That is the regime where the technique shines and where it was originally evaluated.

The throughput claim needs more care. Tokens per second per replica rises only if the added arithmetic does not lengthen steps proportionally and if the draft model’s compute is not itself a bottleneck. At high concurrency neither holds, and the aggregate token throughput of a pool can fall even while per-request latency improves. A capacity plan that adopts speculative decoding on the strength of a latency result and then applies the resulting speedup factor to a throughput calculation will over-provision the saving.

There is also a memory cost that does not appear in the flop accounting. The draft model’s weights occupy accelerator memory that could otherwise hold KV cache, which reduces the number of concurrent sequences, which reduces the batch. In a bandwidth-bound regime that is a small loss; in a memory-constrained deployment it can erase the gain. The draft model’s weights are the price of admission, and they should be subtracted from the memory budget before the concurrency is computed, not after.

When not to use it

The technique does not help when the acceptance rate is structurally low: high-temperature sampling over open-ended text, or a draft model whose training distribution diverges from the target’s. It does not help when the target is already compute-bound, which is the normal state of a well-utilized large batch. And it does not help when the workload is dominated by prefill, since verification only accelerates decode; a prompt-heavy service gains nothing from it, as the prefill and decode split makes clear. The decision belongs to the workload profile, not to the technique.