Answer Normalization and the Fragility of Exact Match

Extractive question answering reduces a generation problem to a comparison: the model returns a span of the passage, and the span either matches the annotated answer or it does not. The metric is a string equality check, and its appeal is that it involves no judgment, no model, and no reference set beyond the annotations themselves. That appeal dissolves on contact with real answers, because two strings that denote the same thing are almost never byte-identical. “Barack Obama”, “Obama”, “barack obama”, and “President Obama” are one answer written four ways, and a bare equality test scores three of them as wrong.

The normalization pipeline exists to convert those variants into a canonical form before comparison. Every step in it is a decision about which differences are spelling and which are meaning, and the score moves with those decisions. Reporting an exact-match figure without the pipeline is reporting a number whose unit is undefined.

The canonical steps and what each one buys

Case folding removes the distinction between “paris” and “Paris”. It is safe for most proper nouns and unsafe for a handful of cases where capitalization carries information, and it is nearly universal in QA scoring because annotation case is not consistent even within one dataset.

Article and determiner removal strips “a”, “an”, and “the”. This changes “the Nile” and “Nile” into the same string. It is the step that most clearly crosses from spelling into grammar, and it occasionally collapses distinctions: “a few” and “few” differ in meaning, and a numeric answer of “1” versus “the 1” is not a real ambiguity but “one” versus “a one” might be.

Punctuation removal drops commas, periods, and quotation marks. A number written “1,000” becomes “1000”, which is desirable; a title written “MAS*H” loses its asterisks, which is not.

Whitespace collapsing turns any run of spaces into a single space and strips the ends. This handles the most common source of spurious mismatch in generated output, where a decoder emits a leading space after a prompt boundary.

Unicode normalization is the step most often omitted and most consequential for non-ASCII text. The standard defines four normalization forms. NFC composes sequences into precomposed characters where a canonical composition exists; NFD decomposes them. “é” as a single code point and “e” followed by a combining acute accent are canonically equivalent and compare unequal as byte strings. Applying NFC to both sides makes them equal. The standard’s conformance clauses require that canonical equivalence be preserved by processes that claim to handle text correctly, which is the formal statement of why the step belongs in the pipeline.

import re
import string
import unicodedata

ARTICLES = {"a", "an", "the"}

def normalize(text):
    text = unicodedata.normalize("NFD", text)
    # strip combining marks: cafe\u0301 -> cafe
    text = "".join(ch for ch in text
                   if not unicodedata.combining(ch))
    text = text.casefold()
    text = "".join(ch for ch in text if ch not in string.punctuation)
    tokens = [t for t in text.split() if t not in ARTICLES]
    return " ".join(tokens)

def exact_match(prediction, gold):
    return int(normalize(prediction) == normalize(gold))

Note that the function decomposes to NFD and then drops combining marks, which folds accents away entirely rather than recomposing them. That is a stronger decision than NFC: it makes “café” and “cafe” equal. Whether that is correct depends on the language and the task. In a dataset where names are transliterated inconsistently it helps; in one where the accent distinguishes two different words it destroys information. The choice has to be made per dataset and recorded.

The asymmetry between annotation and generation

Human annotations in extractive datasets are spans selected inside a passage. The annotator clicks a region, and the stored answer is the text of that region, including whatever punctuation and capitalization the passage used. The gold string is therefore coupled to the passage’s typography, not to a canonical form of the answer.

A generative model producing the same answer has no such coupling. It emits tokens from its own vocabulary and its own formatting habits, which may include a leading article that the passage did not have, a different numeral style, or a spelled-out form where the passage used digits. Normalization cannot close this gap entirely because the model may produce a correct answer the pipeline was not designed to fold. A model that answers “eighteen” where the annotation says “18” is wrong under any string comparison and right in substance.

This is why exact match is a lower bound on accuracy rather than an estimate of it. The size of the gap between exact match and true correctness depends on how much freedom the output format has. For span extraction constrained to the passage, the gap is small, since the model can only return text that appears in the input. For free-form generation it is large and grows with the model’s fluency, because a more fluent model is more likely to paraphrase rather than copy.

The reverse error also exists. A normalization pipeline aggressive enough to fold many variants can fold a wrong answer into the gold string. Removing all punctuation and articles makes short answers collide: a pipeline that strips everything can map “no” and “no.” together, which is fine, and can also map distinct short numeric answers together if the digit strings are similar. Each removal step raises the false-positive rate while lowering the false-negative rate, and there is no setting that minimizes both.

Numbers, dates, and units as a separate problem

Numeric answers break string normalization in a specific way, because the same quantity has many textual encodings and the mapping between them is not a string transformation. “1,500”, “1500”, “1.5k”, and “fifteen hundred” denote one number. “3.5” and “3,5” denote the same number in different locale conventions and different numbers if read under one convention. A date written “March 3, 2019” and “2019-03-03” is the same date; “3/4/2019” is ambiguous between two conventions and cannot be normalized without knowing the source locale.

A pipeline can parse numerals and compare numerically, which handles the first family and not the last. Datasets with substantial numeric content typically score with a tolerance or with a separate numeric-accuracy metric alongside exact match, precisely because the string comparison cannot be trusted for those items. The benchmark that introduced discrete reasoning over paragraphs made this explicit by scoring numbers separately from spans, since a system that computed the right value in the wrong format was being counted as wrong.

Units are the same problem one level up. “5 km” and “5000 m” are one measurement. Normalizing units requires a dimension table and a canonical unit per dimension, which is a domain-specific addition rather than a general text transformation. Most pipelines do not attempt it, and the resulting misses are indistinguishable in the aggregate from genuine errors.

Reporting what the pipeline did

Because the score depends on the pipeline, the pipeline is part of the metric definition. Two evaluations of the same model against the same data can differ by several points purely from whether articles were stripped and whether Unicode normalization was applied. Papers that report exact match usually cite the dataset’s official evaluation script, which fixes the choices; numbers produced by a different script are not comparable, and the difference is often larger than the improvement being claimed.

The metric also has no partial credit, which makes it insensitive to near-misses and hostile to systems that are almost right. That is a deliberate trade for determinism and cheapness, and for tasks where the answer is a single entity or a single number the trade is reasonable. For tasks where the answer is a phrase or a sentence, the same comparison problem is better handled by a scorer that measures overlap, such as the clipped n-gram precision used for translation, or by a judgment of equivalence that does not depend on string form at all. The span-level F-measure used in reading comprehension sits between the two: it gives partial credit for partial span overlap while remaining a deterministic function of the strings.

The deeper issue is that exact match measures the model’s formatting behaviour as much as its knowledge, and the two are entangled in a single number that cannot be decomposed after the fact. Keeping the raw predictions and the normalization code alongside the reported score is the only way to tell them apart later.

A numeral written as words is a case the pipeline could in principle handle by parsing both sides as numbers and comparing values. Extending the pipeline that way converts the metric from string equality to value equality, which is a different metric with a different error profile, and the reporting obligation grows accordingly: a score computed under value comparison is not comparable to one computed under string comparison, and the two are routinely conflated when a benchmark’s evaluation script is changed between releases. The inter-annotator agreement figure for the annotation set is the other half of the picture, since a gold set that two annotators would label differently places a ceiling on what any comparison against it can measure.