Decomposing Generated Text into Atomic Claims for Factuality Scoring

A response containing one false sentence and nine true ones is not 90 percent factual in any useful sense, because the false sentence may be the one the reader acts on. Sentence-level accuracy treats each sentence as one judgment, which hides the number of distinct assertions in it, the number that were checkable, and the number that were supported by the source. Claim decomposition addresses the denominator: instead of scoring sentences, split each into atomic assertions and score those.

The method is a response to a specific structural problem. Sentences in generated text are compound. “The treaty was signed in 1648 and ended the Thirty Years’ War, though fighting continued in Catalonia until 1652” contains at least three assertions with different evidentiary status. A single label of “supported” or “not supported” applied to that sentence is either too generous or too strict, and either choice distorts the aggregate.

What counts as atomic

An atomic claim is a single proposition that can be true or false on its own, stated without pronouns or deictic references that depend on surrounding text. “It was signed in 1648” is not atomic in that sense, because “it” has no referent outside the sentence. The decomposition resolves the reference: “The Peace of Westphalia was signed in 1648.” This resolution step is what makes the claims checkable independently, and it is also where decomposition can introduce errors, since a misresolved pronoun produces a claim the original text did not make.

Atomicity is relative to the verification method, not an absolute property. A claim like “the algorithm runs in quadratic time” is atomic for a human reader but requires defining which algorithm and which input model. A claim like “the bridge is 2.4 kilometres long” is atomic and checkable against a measurement. The granularity has to be fixed by the protocol, because a finer decomposition produces more claims, a lower support rate on the same text, and a different aggregate.

The practical rule used in factuality evaluation is that each claim should contain exactly one relation between entities, optionally with qualifiers that constrain it. Conjunctions split. Comparatives stay attached to their comparison. Negation stays with the proposition it negates, since splitting “the drug does not reduce mortality” into “the drug” and “reduces mortality” reverses the meaning. Conditional and modal claims are kept whole, because the modality is part of the assertion: “may cause” and “causes” are different claims with different evidence requirements.

def score_claims(claims, verdicts):
    """verdicts: dict claim -> 'supported' | 'refuted' | 'unverifiable'."""
    counts = {"supported": 0, "refuted": 0, "unverifiable": 0}
    for claim in claims:
        counts[verdicts[claim]] += 1
    total = len(claims)
    if total == 0:
        return None
    return {
        "support_rate": counts["supported"] / total,
        "refute_rate": counts["refuted"] / total,
        "unverifiable_rate": counts["unverifiable"] / total,
        "checkable": (counts["supported"] + counts["refuted"]) / total,
    }

The three-way verdict and why the third category is load-bearing

A binary supported/not-supported verdict forces every claim into one of two buckets, and a large fraction of claims in generated text are neither. A claim can be unverifiable because the source does not address it, because it is a prediction, because it is a value judgment, or because verifying it requires evidence outside the provided context. Folding those into “not supported” penalizes a model for saying something the source does not cover, which is not the same failure as saying something false.

The three-way split separates the failure modes. A refuted claim is a factual error against the available evidence. An unverifiable claim is a coverage gap: the model asserted something the evidence does not adjudicate. A supported claim is correct. Reporting the three rates rather than a single precision figure makes the distinction visible, and the distinction matters because the interventions differ: refutations are fixed by better grounding, unverifiable claims by better calibration about what the evidence supports.

The unverifiable category is also where the evaluation is most vulnerable to gaming. A model that produces only hedged, generic, widely-known statements has a high support rate and a low information content. Precision alone rewards that. The counterweight is a coverage or recall measure against the claims a good answer should have made, which requires a reference set of expected claims. That is the same reference dependence that recall-oriented summarization scoring faces, and with the same consequence: the metric needs a length or content constraint outside itself to prevent degenerate strategies.

Evidence retrieval and the attribution boundary

Scoring a claim requires evidence. In a retrieval-grounded setting the evidence is the retrieved context, and the question is whether the claim is entailed by it. In an open-domain setting the evidence has to be found, which introduces a retrieval step whose quality bounds the verification. A claim marked unverifiable may be true and simply not retrievable by the verifier, and the metric then measures the retrieval system as much as the generator.

This is the attribution boundary, and it has to be stated explicitly in any reported number. Claim-level support against provided documents answers “is the output consistent with what was retrieved”. Claim-level support against a search index answers “is the output consistent with what a search engine could find”. Those are different properties, and a system that retrieves well will look better on the second even if its generation is identical.

The verification step itself has three possible designs. Entailment by a trained natural language inference model is fast and cheap and inherits the model’s errors, particularly on claims involving numbers, negation, and entity substitutions, which are exactly the cases that matter most. Verification by a strong generative model reading the evidence is more accurate on those cases and is not deterministic, so repeated runs on the same claim can disagree. Verification by a human is the reference standard and does not scale past a sample.

The choice interacts with the aggregation. A verifier with a false-positive rate on refutations will report a support rate higher than the truth, and the error is not uniform across claim types. Claims with many named entities are harder to verify and more likely to be misjudged. Stratifying the reported rates by claim type exposes the pattern; a single aggregate hides it.

The denominator problem

Decomposition changes the denominator, and the denominator is the part of the metric that is easiest to manipulate. A generator that produces short, simple sentences yields more claims for the same content, and if the claims are individually easy to support, the support rate rises. A generator that packs several assertions into one complex sentence yields fewer claims, each harder to verify, and its support rate falls even when the underlying factual content is the same.

There is no decomposition-independent denominator, because the number of atomic claims in a text is a property of the text plus the decomposition protocol. What is controllable is consistency: the same protocol applied to every system being compared, with the granularity rule fixed in advance and checked on a sample. Reported support rates are comparable only under that condition, which is the same condition that makes BLEU comparisons meaningful only under a fixed tokenizer.

The second denominator issue is that decomposition itself is usually done by a model, and models decompose inconsistently. The same sentence submitted twice can yield two or three claims depending on sampling, and the difference propagates to the score. Running the decomposition at a fixed low temperature and checking agreement across repeated runs on a sample gives a rough measure of how much of the score’s variance comes from this step rather than from the generation being evaluated.

What the score supports

A claim-level factuality score supports a statement of the form: under this decomposition protocol and this verifier, this fraction of the assertions in the output was supported by evidence retrieved under this policy. Every qualifier in that sentence is load-bearing. The number does not say the output is true, since the evidence may be incomplete; it does not say the output is complete, since unsupported assertions are not the same as missing ones; and it does not transfer across verifiers, since the verifier’s error profile is part of the measurement.

What it does better than sentence-level scoring is localize failure. A per-claim verdict list identifies which specific assertions failed and what evidence contradicted them, which is actionable in a way that a sentence-level label is not. The list also exposes the case that sentence-level scoring hides most effectively: a sentence that is mostly correct with one wrong detail. Under sentence-level scoring that is a failure indistinguishable from a sentence that is entirely fabricated, and the two require different responses. Decomposition makes the difference visible, which is the entire argument for paying its cost.