Why Perplexity Is Not Comparable Across Tokenizers

A language model assigns a probability to each token it sees, given the tokens before it. Multiply those probabilities across a held-out text and invert, and the result is the reciprocal of the geometric mean probability per token. Take the $N$-th root of that reciprocal for a text of $N$ tokens and the number is perplexity: the size of a uniform distribution over tokens that would produce the same average surprise. A model with perplexity 20 on a corpus is, in this precise and limited sense, as uncertain as a uniform choice among 20 symbols.

The definition is clean. The problem starts when the number is used to compare models, because “a token” is not a fixed unit of text. Two models with different vocabularies tokenize the same sentence into different numbers of pieces, and perplexity per token is an average over those pieces. Shortening the average by merging text into fewer, larger tokens lowers perplexity without any change in how well the model predicts the underlying characters.

The arithmetic of the unit change

Write the average negative log-likelihood per token as $H = -\frac{1}{N}\sum_{i=1}^{N} \log p(t_i \mid t_{<i})$, using natural logarithms. Perplexity is $\exp(H)$. The same quantity in base 2 is the cross-entropy in bits per token, and $\text{perplexity} = 2^{\text{bits per token}}$.

Now suppose a second tokenizer encodes the same text as $M$ tokens instead of $N$, with $M < N$ because it merges more aggressively. If the model’s total probability of the text is unchanged (the same characters, the same model quality) then the average surprise per token rises, because the same total surprise is spread over fewer predictions. Perplexity goes up even though the model predicts the same text equally well. The direction is counterintuitive: a coarser tokenizer produces a higher per-token perplexity for an equivalent model, because each prediction covers more characters and is therefore individually harder.

The inverse is what people actually observe in practice when they compare a small-vocabulary character-level model to a word-level model. The character model reports a much lower perplexity, since each prediction is among a few dozen symbols, while the word model reports a higher one over tens of thousands. Neither number says which model assigns more probability to English.

Dividing by token count also means perplexity is not additive across documents of different token counts. A corpus-level perplexity is computed by summing log-likelihoods and dividing by the total token count, not by averaging per-document perplexities. Averaging per-document values weights a ten-token document the same as a thousand-token document, and because short documents have noisy averages, the result is dominated by the short ones. The standard formula exponentiates the corpus-level average, which is the geometric mean of the per-token probabilities across the whole corpus.

Bits per byte as a unit-free substitute

The fix is to change the denominator from tokens to bytes of the original text. Sum the negative log-likelihood in bits over the corpus, divide by the number of UTF-8 bytes in the un-tokenized text, and the result is bits per byte. Because the byte count is a property of the text and not of the tokenizer, two models with different vocabularies can be compared on the same scale. A character-level model and a word-level model that assign the same probability to a document produce the same bits per byte, by construction.

import math

def bits_per_byte(logprobs, token_bytes, raw_bytes):
    """logprobs: natural-log probability per token, in order.
       token_bytes: bytes each token spans in the original UTF-8 text."""
    assert len(logprobs) == len(token_bytes)
    total_nats = -sum(logprobs)
    return total_nats / math.log(2) / raw_bytes

def perplexity(logprobs):
    return math.exp(-sum(logprobs) / len(logprobs))

The conversion requires knowing how many bytes each token covers, which means the evaluation harness has to keep the mapping from token to original byte span. Byte-level tokenizers make this exact, since every token is a sequence of bytes by construction. Vocabulary-based tokenizers that normalize text, such as lowercasing, accent stripping, and whitespace collapsing, break the mapping, because the bytes the model saw are not the bytes on disk. The mismatch is small per token and systematic across a corpus, and it is one of several reasons the tokenizer must be part of the reported configuration.

Bits per byte also gives a natural interpretation for multilingual comparison. A text in a language with a writing system the tokenizer handles poorly is split into more tokens per character, which inflates per-token perplexity relative to a language the tokenizer was tuned for. Measured per byte, the discrepancy shrinks to whatever real difference in predictability exists, rather than encoding a tokenizer’s vocabulary bias.

The measure is not without its own dependency. It rewards models that model byte-level structure, so a model trained on normalized text and evaluated against raw bytes pays for every character it never learned to predict. That is arguably correct, since it failed to predict those bytes, but it makes the score sensitive to preprocessing choices in the training pipeline, which belong in the evaluation record alongside the number.

What perplexity does and does not predict

Perplexity is a proper scoring rule for the distribution the model assigns, which is the sense in which it is a legitimate measure: a model that assigns higher probability to held-out text is, by the definition of log loss, a better probabilistic predictor of that text. That is a statement about the distribution over the evaluation corpus, not about generation quality. A model can have excellent perplexity and produce repetitive samples, because perplexity averages over all positions while generation is driven by the tail of the distribution at each step.

The gap between the two shows up in decoding. Perplexity is computed on the model’s own conditional distributions over reference text. At generation time the model is fed its own previous outputs, and errors compound: a token sampled from a low-probability region moves the context somewhere the model assigns low probability to everything that follows. Teacher-forced perplexity never sees this, since every context is a real prefix from the corpus. The discrepancy is largest for models whose per-token distributions are broad, and it is one reason perplexity rankings do not transfer cleanly to rankings by human judgment.

There is also the question of what corpus. Perplexity is defined against a specific held-out set, and a model trained on data that overlaps the evaluation corpus reports a number that measures memorization rather than generalization. Detecting that overlap is the same problem as benchmark contamination, and it applies with particular force to perplexity because the metric is computed directly on text that may have been in the training set.

Using it anyway

Perplexity remains the cheapest signal available during training, where the comparison is between checkpoints of the same model with the same tokenizer on the same corpus. In that setting all the confounds above are constant, and the number is a faithful readout of the quantity it measures. It is also the right metric when the model’s job is genuinely to assign probability, as a compression model, a speech recognizer’s language model, and a candidate re-ranker, rather than to generate text for a reader.

The rule that follows is narrow and worth stating plainly: report perplexity with the tokenizer, the corpus, and the normalization applied to both, or convert to bits per byte and report that. A perplexity figure without those details is a number in an unknown unit, and comparing it to a figure from another paper is comparing lengths in different currencies. Downstream evaluation of generated text needs a metric that reads the output itself, whether that is overlap against references or a preference judgment, and those are different instruments with different failure modes.

There is one further confound worth naming, because it is easy to mistake for model quality. Perplexity is computed over a corpus whose token count depends on the tokenizer, and the same corpus tokenized two ways produces two different totals. If the evaluation harness tokenizes the corpus once and reuses that tokenization for a model trained on a different vocabulary, every token the model cannot represent is scored as a rare symbol, and the resulting loss reflects the vocabulary mismatch rather than the model. A character-level fallback in a byte-level tokenizer hides this by making every byte representable, at the cost of longer sequences and therefore more forward passes per document, which is the compute side of the same trade.

Evaluations that compare models trained under different tokenizers have settled on the byte-normalized figure for exactly this reason, and the convention is now common enough that a reported perplexity without a stated unit is usually assumed to be per token, which is the assumption most likely to mislead. The safer habit is to report the raw cross-entropy in nats per token together with the average token count per byte for the corpus, since those two numbers let a reader reconstruct any normalized form. That pair is also what makes the figure checkable against the text normalization discipline used in exact-match scoring, where the same rule applies in the other direction: the transformation applied to the reference has to be applied identically to the output, and stated, or the score is a number in an unknown unit.