Recall-Oriented Summary Scoring with ROUGE

A summary is judged by what it retained, not by what it emitted. That asymmetry with translation evaluation is the whole design premise of the ROUGE family, and it produces a metric that divides overlap by the reference’s length instead of the candidate’s. The consequence is immediate and intentional: a system that copies the entire source document into the summary field achieves perfect recall on every n-gram order. ROUGE alone cannot tell a summary from a transcript, and every practical use of the metric depends on a length constraint outside the score itself.

The denominator is the reference

For a reference summary containing $|R|$ n-grams of order $n$ and a candidate containing $|C|$, the recall numerator is the count of reference n-grams that also appear in the candidate, capped at the candidate’s own count for that type. The score is that numerator over $|R|$. With multiple reference summaries, the denominator is the sum of the per-reference counts, and the numerator credits each reference independently. A candidate that matches only the first of four references can therefore score at most a quarter.

That multi-reference rule is the sharpest structural difference from BLEU. In BLEU, adding a reference can only raise a candidate’s score, because clipping takes the maximum count per n-gram across references. In ROUGE, adding a reference raises the denominator by the reference’s full n-gram count while adding to the numerator only the part the candidate happens to match, so extra references usually lower the score. The two metrics are answering different questions: BLEU asks whether the output looks like something a human would have written, ROUGE asks whether the output covers what the humans put in their summaries.

Clipping still applies inside ROUGE-N. A reference n-gram type that appears twice in the reference and once in the candidate contributes one match, not two, so the maximum achievable recall for a candidate that omits one repetition is bounded below 1.0. This mirrors the reference count exactly, which means an output with redundant phrasing is not punished for the redundancy itself, only for failing to cover the second occurrence.

from collections import Counter

def rouge_n(candidate, references, n):
    cand = Counter(tuple(candidate[i:i + n]) for i in range(len(candidate) - n + 1))
    matched = 0
    total = 0
    for ref in references:
        ref_counts = Counter(tuple(ref[i:i + n]) for i in range(len(ref) - n + 1))
        total += sum(ref_counts.values())
        for gram, count in ref_counts.items():
            matched += min(count, cand.get(gram, 0))
    return matched / total if total else 0.0

Unigram recall is the most forgiving order and the one most often reported as ROUGE-1. It measures content-word overlap with no regard to order, so a summary that lists the right entities in the wrong relationships scores well. Bigram recall, ROUGE-2, is the standard proxy for fluency in the summarization literature because it requires adjacent pairs to survive intact.

Longest common subsequence as a single-number order-free measure

ROUGE-L replaces the fixed n-gram order with the longest common subsequence of candidate and reference. An LCS of length $L$ requires the matched tokens to appear in both sequences in the same relative order, but not contiguously. Recall is $L/|R|$, precision is $L/|C|$, and the reported score is their weighted harmonic mean with a parameter $\beta$ that usually favors recall. The same $\beta$-weighted F-measure is used in the span-level scoring of exact-match evaluation, where the recall direction has the same motivation.

The LCS formulation avoids one weakness of fixed-order n-grams: it does not need a match at every position to credit a long shared span. A candidate that reproduces a twenty-token sentence from the reference but inserts one word in the middle scores zero on ROUGE-4 for that region while scoring close to full on ROUGE-L. Conversely, LCS has no notion of how much it had to skip. Two matches separated by a large gap in the candidate count the same as two adjacent matches, which is why the weighted variant exists: it assigns a reward that grows with the length of each consecutive run, using a dynamic program whose state tracks the current run length rather than only the LCS length.

The LCS dynamic program is the standard quadratic recurrence over prefixes, with $O(|C| \cdot |R|)$ time and $O(\min(|C|,|R|))$ space if only two rows are kept. For a summary capped at a few hundred tokens against a reference of similar size, the cost is trivial. For document-level references of tens of thousands of tokens the quadratic term becomes the binding constraint, and the score is usually computed per sentence and averaged instead.

A third variant, skip-bigram co-occurrence, counts ordered pairs of tokens that appear in both texts within a maximum gap, and the extension that also counts unigrams exists to give partial credit when no pair survives. The family is large because each member encodes a different assumption about which kinds of overlap indicate a good summary.

What the recall direction permits

Because the denominator ignores candidate length, the metric has no internal defense against verbosity. The standard countermeasure is external: evaluation sets specify a summary length, and systems are truncated or penalized outside the metric. Historically, shared-task evaluations fixed a hard cap in words or bytes, and any submission exceeding it was cut at the boundary before scoring. That makes the score comparable across systems while leaving the metric itself unbounded.

The second exploit is redundancy. A summary that repeats the same high-frequency sentence three times covers no additional reference content, so its recall does not rise, but a summary that repeats three different sentences drawn from different parts of the reference does raise recall while reading as a list rather than a summary. Recall-only metrics reward coverage, and coverage is exactly what redundancy plus selection can manufacture without producing a coherent text.

The third is positional. Reference summaries in news corpora are written by people who tend to place the most important material first, and source documents tend to lead with it too. A system that emits the first few sentences of the source therefore scores respectably on ROUGE without performing any abstraction. This lead bias is a property of the data, not of the metric, but it means that a ROUGE gain of a point or two can reflect a better sentence-ranking heuristic rather than better language generation. Evaluations that want to distinguish abstractive from extractive behaviour need a separate check on how much of the output is copied verbatim, which is a string-matching question rather than an overlap-counting one.

There is a fourth, which is the interaction between recall and factual grounding. A summary can achieve high unigram recall by stringing together the reference’s content words in an order that asserts something the source does not. Because ROUGE-N ignores order and ROUGE-L only requires relative order, a rearrangement of the same tokens preserves most of the score while inverting the meaning of a sentence. Negation is the clearest case: a reference saying a treatment reduced a measure and a summary saying it did not share almost all their n-grams. The metric has no mechanism for noticing, and this is the specific reason summarization evaluation moved toward claim-level scoring, where each assertion is checked against the source separately. The trade is the one described under atomic claim decomposition: a finer unit of judgment, a more expensive protocol, and a denominator that has to be fixed in advance.

Reference quality bounds the score

ROUGE inherits the reference dependence of all overlap metrics and adds a specific sensitivity: the recall denominator grows with reference length, so a long reference summary containing many details makes high recall harder to reach than a short one containing only the headline facts. Two annotators summarizing the same article at the same length budget routinely select different sentences, and a candidate can match one annotation closely while scoring poorly against the other. The ceiling on the score is therefore set by inter-annotator agreement, which is why agreement measurement is a prerequisite for interpreting any absolute ROUGE number.

Modern summarization evaluation increasingly reports learned metrics that score semantic equivalence rather than surface overlap, and those metrics change the failure profile rather than removing the problem: they trade the determinism and cheapness of n-gram counting for model-dependent judgments that are harder to audit. ROUGE remains the fixed reference point, and its most defensible use is comparing systems on a frozen test set with fixed tokenization, fixed references, and a fixed length cap, where the score’s biases are constant across the systems being compared and therefore cancel in the difference.