Clipped N-Gram Precision and the Brevity Penalty

A generated sentence is compared to one or more human references, and the comparison has to reduce to a number that ranks systems the same way a human panel would. The oldest family of such numbers is built from n-gram overlap, and the design problem is not counting matches. It is deciding which matches should count, because a generator that emits one safe phrase repeatedly can match a reference many times without producing anything a reader would accept.

BLEU resolves that problem with a single mechanical rule: a candidate n-gram is credited only as many times as it appears in the reference that contains it most often. That rule, plus a length correction, defines the metric. Everything else in the metric is bookkeeping around those two decisions.

Counting only matches the reference could have supplied

Consider a reference of twelve words and a candidate that repeats one of its trigrams five times. Raw precision would count five matches out of however many candidate trigrams exist. Clipping caps the credit for that trigram at one, because no reference sentence contains it more than once. The candidate’s precision then falls to something proportional to its genuinely varied overlap.

The clip is computed per n-gram type, not per position. For each n-gram $g$ in the candidate, let $c_g$ be its count in the candidate and $m_g$ its maximum count across the references. The credited count is $\min(c_g, m_g)$. Summing the credited counts and dividing by the total number of candidate n-grams gives the modified precision $p_n$ for that order. With multiple references, taking the maximum count per n-gram is what makes additional references monotonically helpful: adding a reference can only raise $m_g$, never lower it.

from collections import Counter

def modified_precision(candidate, references, n):
    cand_ngrams = Counter(tuple(candidate[i:i + n])
                          for i in range(len(candidate) - n + 1))
    if not cand_ngrams:
        return 0.0
    max_ref = Counter()
    for ref in references:
        ref_ngrams = Counter(tuple(ref[i:i + n])
                             for i in range(len(ref) - n + 1))
        for gram, count in ref_ngrams.items():
            # keep the largest count seen in any single reference
            max_ref[gram] = max(max_ref[gram], count)
    clipped = sum(min(count, max_ref[gram])
                  for gram, count in cand_ngrams.items())
    return clipped / sum(cand_ngrams.values())

Unigram precision alone is nearly useless as a quality signal. Function words dominate it, and a candidate that shares only articles and prepositions with the reference can score well. Higher orders carry more information about phrase structure, which is why the metric combines orders one through four rather than reporting them separately. The combination is a weighted geometric mean, with the four weights summing to one and set uniformly to $1/4$ in the original formulation.

The geometric mean has a consequence that surprises people who treat BLEU as an average: if any order has zero matches, the whole score is zero, regardless of how good the other three orders are. A four-token candidate cannot contain a 4-gram match unless it is exactly a reference 4-gram, so short candidates collapse to zero on the highest order. That hard zero is a deliberate structural statement, that no phrase of length four was reproduced, but it makes the metric discontinuous near the short end, and it is the reason smoothing variants exist.

Why length needs a separate correction

Clipping removes the incentive to repeat, but it leaves a second exploit open: emit very little. Precision is a ratio over candidate n-grams, so a candidate containing one word that appears in the reference has unigram precision of 1.0. The geometric mean over four orders is still zero for such a candidate, but the general principle holds for longer outputs that stay short and safe. A system that translates half a sentence correctly and stops can beat a system that translates the whole sentence with one error, if precision is the only axis.

The brevity penalty closes that hole by multiplying the score with a factor that depends on candidate length $c$ and the effective reference length $r$. If $c \ge r$ the factor is 1. If $c < r$ it is $\exp(1 - r/c)$. The choice of $r$ matters with multiple references: the original metric uses the reference length closest to the candidate length, not the shortest reference, so a candidate is not penalized for matching a shorter valid translation. The penalty is smooth rather than a cliff, but its slope is steep for very short candidates: at half the reference length the factor is about 0.37, and at a quarter it falls to about 0.05.

The penalty corrects length only in aggregate. It has no notion of which words are missing, so a candidate that drops an entire clause and pads with a repeated phrase from elsewhere in the reference is treated as a length-correct output with reduced precision. Nothing in the metric distinguishes omission from substitution.

Aggregating over a corpus changes the quantity

Per-sentence scores are almost never what gets reported. The standard procedure accumulates clipped match counts and totals across the whole test set, computes the four precisions from the sums, and applies one brevity penalty computed from total candidate length against total effective reference length. This corpus-level aggregation is not the same as averaging sentence scores, and the difference is large when sentences vary in length. Averaging sentence-level BLEU gives a one-word sentence the same weight as a forty-word sentence, and because short sentences hit the zero-on-high-order problem constantly, the average is dominated by them.

Reporting practice around this distinction is loose enough that scores are not comparable across papers unless the tokenization, casing, and smoothing are known. The sacreBLEU conventions exist to fix those variables: a standard tokenizer, a fixed reference set, and a signature string that records what was done. Two numbers produced under different tokenizers can differ by several points on the same outputs, which is larger than most reported system improvements.

Tokenization is the deepest dependency. The metric counts word n-grams, and word boundaries are a decision, not a fact. Languages without whitespace segmentation need a segmenter whose choices change the n-gram inventory. Normalization of case, punctuation, and Unicode form changes match counts too. Any evaluation that compares a candidate against references must fix all of these before the score means anything, which is the same constraint that governs answer normalization in extractive question answering.

Where the score stops tracking quality

The metric is a precision measure with a length correction. It has no recall term beyond the implicit one inside clipping, so it cannot distinguish a faithful but incomplete output from a complete but awkward one when both have similar n-gram overlap. It also treats all n-grams as equally informative: a swapped modifier inside a fluent phrase and a wrong named entity can produce the same match count, though only one changes the meaning of the sentence.

Reference variation is the other failure axis. With a single reference, valid paraphrases score poorly, and the metric rewards outputs that resemble the reference’s surface form. Adding references helps, but the maximum-count rule means that a large reference set steadily raises the ceiling toward “any word sequence a human might have written”, which inflates scores for systems that produce generic text.

Meta-evaluations of the metric against human judgments found moderate correlation at the system level and weak correlation at the sentence level, which is the pattern the metric’s design predicts: corpus aggregation averages away the per-sentence noise, and the remaining signal is largely adequacy plus fluency in aggregate. For tasks where the correct output is a short span rather than a sentence, overlap metrics are the wrong instrument entirely, and exact-match or span-level scoring is used instead.

The metric survives because it is cheap, deterministic, and reference-only, which makes it usable as a regression check during training. Its known blind spots are handled by pairing it with a likelihood-based measure such as perplexity or with a recall-oriented score such as ROUGE, neither of which fixes the underlying issue but each of which fails in a different direction.