Span Attribution: Making Generated Claims Point at Real Text

A citation in generated text is a claim about provenance: these words came from that passage. The claim is checkable only if the passage can be identified, and identifying it requires that the coordinates established at index time still resolve to the same bytes at display time. Every stage between the source file and the rendered answer transforms text, and each transformation is an opportunity to lose the mapping. The engineering problem is not the generation of citation markers; it is the preservation of offsets through a chain of transforms that were not designed to preserve them.

The problem shows up concretely when a passage is edited. The chunk was cut at byte offset 41,200 of a document, the passage was highlighted in the interface, and a reader following the link lands on a paragraph that has since been rewritten. The citation is now false in a way that no model output check can detect, because the text it points at has changed rather than the text that cited it. Versioning is therefore part of attribution, not an afterthought: a citation must name a version of the source, and the version must be resolvable after the source changes.

Offsets are cheaper to keep than to recover

The cheap design records a span as a document identifier plus a start and end offset in a stable coordinate system. The expensive design tries to recover the span later by searching for the quoted text, which fails whenever the text is paraphrased, reformatted, or translated, and fails silently when the search finds a different occurrence of the same sentence.

Choosing the coordinate system is the first decision. Byte offsets into the original file are stable as long as the file does not change, but they are meaningless to a reader without the file. Character offsets into a normalised string are stable under a fixed normalisation and readable, but only if the normalisation is recorded and applied consistently. Token offsets into the encoder’s tokenisation are convenient for the model and useless for display, because the tokeniser’s boundaries do not correspond to character boundaries in any recoverable way for a reader.

The common resolution keeps two coordinates and the mapping between them: a byte range in the source, and a character range in the normalised text used for display. The mapping is recorded when the normalisation runs, as a list of edits rather than as a formula, because the normalisation may be non-uniform. Unicode normalisation is the specific case that makes this necessary: converting to a composed form can change a string’s length, so a character offset in the original and in the normalised form diverge by a variable amount, and the divergence is not monotone in any way that can be recomputed from the normalised string alone.

import unicodedata

def normalize_with_map(text, form="NFC"):
    # returns normalized text and, per output character, the source span it came from
    out, src_map = [], []
    i = 0
    while i < len(text):
        j = i + 1
        # extend the cluster over following combining marks, whose class is nonzero
        while j < len(text) and unicodedata.combining(text[j]):
            j += 1
        composed = unicodedata.normalize(form, text[i:j])
        for _ in range(len(composed)):
            src_map.append((i, j))
        out.append(composed)
        i = j
    return "".join(out), src_map

The map is length-equal to the normalised text, and a span in normalised coordinates becomes a span in source coordinates by taking the minimum start and maximum end over the covered indices. That widening is deliberate: a span that covers a partially composed cluster should highlight the whole cluster in the source rather than a fragment.

What the generator actually emits

There are three ways to get a citation out of a generative model, and they differ in what can be verified.

The first is a marker in the output that refers to an enumerated context block, such as a bracketed index. The model emits the index; the pipeline maps the index to the chunk identifier recorded when the prompt was assembled. Verification is exact, because the mapping is a table lookup, and the granularity is the whole chunk. The model can cite a chunk that does not support the sentence, and the system cannot tell from the marker alone; it can only confirm that the chunk was in the context.

The second is a verbatim quote. The model is asked to reproduce the supporting sentence exactly, and the pipeline searches the cited chunk for that string. This is verifiable by string comparison, and the comparison is where the subtlety lives. Exact byte equality fails on whitespace differences, on quotation-mark style, on hyphenation, and on the model’s tendency to normalise punctuation while copying. Normalising both sides before comparison makes the check pass more often and makes it weaker, since a normalisation aggressive enough to absorb formatting differences also absorbs differences that matter.

The third is a structured span: the model emits a chunk identifier and a character range within the chunk. This is the most precise and the least reliable, because models are poor at counting characters and the emitted ranges are frequently off by a few positions. Treating the emitted range as a hint and re-anchoring it by searching the chunk for the quoted text is the usual compromise: the hint disambiguates repeated sentences, and the search corrects the off-by-a-few error.

def resolve_citation(quote, chunk_text, hint=None, window=200):
    norm_q = normalise_for_match(quote)
    norm_c = normalise_for_match(chunk_text)
    idx = norm_c.find(norm_q)
    if idx < 0 and hint is not None:
        # fall back to a bounded search near the model's guess
        lo = max(0, hint - window)
        idx = norm_c.find(norm_q, lo, hint + window)
    if idx < 0:
        return None                     # unverifiable: report rather than guess
    return (idx, idx + len(norm_q))     # in normalised chunk coordinates

Returning None rather than the nearest fuzzy match is the important line. A fuzzy match that resolves to the wrong sentence produces a citation that looks verified and points at unrelated text, which is worse than an explicit failure.

What verification proves

A quote that resolves to a span proves that the span exists in the cited chunk and that the quoted string occurs there. It does not prove that the span supports the claim. Support is a semantic relation between the sentence and the source, and string containment is a necessary condition rather than a sufficient one.

Three failure patterns pass containment and fail support. The first is the negation flip: the source says a mechanism is not recommended for datasets above a size, and the generated sentence drops the negation while quoting the rest. The second is scope narrowing: the source states a limit for one configuration, and the sentence states it generally. The third is quote mining: the span is accurate and the surrounding passage contradicts it, which is why a citation should resolve to a span with context rather than to a bare sentence.

Detecting those requires a second model pass over the sentence and the span, which is a judgement rather than a string operation and inherits the judgement’s error rate. The architecture that keeps the two separate is worth stating: the string check runs first because it is exact and cheap and catches the majority of fabricated quotes, and the semantic check runs only on spans that passed, because it is expensive and noisy. A pipeline that runs only the semantic check loses the ability to distinguish “the model paraphrased loosely” from “the model invented a quote”, and those need different responses.

Coverage is the other dimension. A generated answer of ten sentences with two citations is mostly uncited, and the two citations do not make the other eight supported. Measuring the fraction of claims that carry a verified span, rather than the fraction of citations that verify, is what distinguishes a grounded answer from a decorated one. The distinction matters because citation density is trivially inflatable: a model that appends a marker to every sentence produces high density and no additional grounding.

Identifiers, versions, and the link that has to resolve

The citation’s value to a reader is that it can be followed. Following it requires a stable identifier for the source at a specific version, and that requirement collides with how documents are usually stored.

A path is not an identifier: it can be renamed, and the same path can hold different content at different times. A content hash is stable but changes on every edit, so a citation to a hash resolves to an immutable snapshot, which is correct and requires keeping snapshots. A document identifier plus a version number is the usual compromise: the identifier survives renaming, the version pins the content, and the resolution step fetches the versioned copy.

When a cited version is no longer retained, the citation must degrade explicitly: either it resolves to the current version with a notice that the text has changed, or it reports that the cited version is unavailable. Silently resolving to current content is the failure mode described at the start, and it is common because it is the cheapest behaviour to implement. The retention policy for cited versions is therefore a product decision with a technical consequence, not a storage detail.

The last edge is a span that crosses a chunk boundary. Chunking cuts text at arbitrary points, so a sentence that supports a claim may be split across two chunks, and neither contains the whole quote. The pipeline can handle this by expanding the search to the chunk’s neighbours within the same document, using the parent pointer recorded at index time, and by reporting the resulting span against the document rather than against a chunk. Without that expansion, boundary-straddling quotes are reported as unverifiable, which is a false negative that grows with the fraction of text near boundaries and is a direct cost of the splitting decisions described in Chunk Boundaries Decide What Can Be Retrieved.

One final constraint ties attribution back to retrieval depth. A claim can only be attributed to a passage that was retrieved, so the ceiling on citation coverage is the recall of the retrieval stage. Adding more retrieved passages raises that ceiling and simultaneously lowers the chance that the model uses any particular one, since a longer context gives each passage a smaller share of attention. That tension is the subject of Context Dilution and Distractor Passages, and it means attribution quality cannot be improved by retrieval volume alone.