Disjunction-Max Combination for Multi-Field Scoring

A document has a title, a body, a set of headings, an anchor text summary, and perhaps a caption on an image. The same query term can match in any of them, and a match in the title is worth more than a match in the body. The obvious implementation is to score each field independently and add the results, and that implementation fails in a specific, predictable way that is worth understanding before choosing a combination rule.

The failure mode of additive field scoring

Suppose a document’s title is short and contains the query term twice, and its body is long and contains the same term twice. Score the title with BM25 and the body with BM25, then sum. The title score is large because the title is short: two occurrences in eight tokens sit far along the saturation curve. The body score is modest. The sum rewards the document for having the term in two places.

Now consider a document whose body is a concatenation of ten sections, each a reasonably short passage, and suppose the term appears twice in every section. If each section is scored as a separate field and the scores summed, the document collects ten nearly-maximal field scores. Nothing in the arithmetic distinguishes this from ten genuinely distinct pieces of evidence, and the document outranks a page that is actually about the query. Adding fields does not add independent evidence; it adds opportunities for the saturating function to restart from zero.

This is the core structural issue. The saturation that makes BM25 well-behaved within one field assumes the count is evidence about one topical unit. Splitting a document into many units and summing per-unit scores breaks that assumption ten times over.

Two combination rules, two different problems

There are two mechanisms that go by related names and solve different halves of the problem.

The first is field-frequency folding: compress all fields into one weighted term frequency before applying saturation once. Each field contributes its own raw count multiplied by a field weight, and its own length normalization contributes to a weighted length. The result is a single pseudo-count, saturated once, so the ceiling property holds across the whole document.

The second is disjunction-max at the query level: treat each field as a separate subquery producing its own score, take the maximum of those scores, and add a small multiple of each remaining score as a tiebreaker. This keeps field scores comparable in their original units and stops the sum from accumulating.

Lucene’s BM25 implementation takes the first route for a single field but exposes the second through its disjunction-max query, and the two are often confused because both are used to make multi-field search behave.

How the max-plus-tiebreaker rule is built

The dismax rule computes, for a document $D$ and a set of field subqueries ${q_i}$:

$$\text{score}(D) = \max_i s_i(D) + t \sum_{j \neq \arg\max} s_j(D)$$

where $t$ is the tiebreaker multiplier. With $t = 0$ the document’s score is purely the best field score, and the remaining fields are ignored entirely. With $t = 1$ it degenerates back to a plain sum. The interesting range is $0 < t \ll 1$, typically 0.1 or smaller.

The tiebreaker exists to break ties between documents whose best-field scores are identical. Two documents both matching “lucene” strongly in the title, one of which also mentions it in the body, ought not be separated by field ordering artifacts alone, and a small $t$ provides that ordering without letting secondary fields dominate.

Implementation is a two-pass scan over the matched subqueries: track the max and the sum simultaneously, then subtract the max from the sum before applying $t$.

/** Combine one document's subquery scores: max, plus a tiebreak share of the rest. */
static double dismax(double[] fieldScores, double tie) {
    double max = Double.NEGATIVE_INFINITY;
    double sum = 0.0;
    for (double s : fieldScores) {
        max = Math.max(max, s);
        sum += s;
    }
    return max + tie * (sum - max);
}

A streaming implementation cannot collect the array, because subquery iterators advance independently and documents arrive in ascending order on each. Instead it keeps two arrays indexed by document: a running sum and a running maximum. When a subquery advances to document $d$, its score is added to sum[d] and used to raise max[d]; a document is scored when all subqueries have passed it. The arithmetic is the same as the array version, and the memory cost is two doubles per live document.

The subtraction sum - max is where implementations differ in the last bits. Computing the maximum separately and subtracting it is exact; recomputing the maximum from the array at scoring time is not. Both give the same ordering in nearly all cases, and the difference only shows up when two field scores are within one unit in the last place of each other.

What the tiebreaker does not fix

Dismax bounds accumulation but it does not model field importance. A title hit and a body hit produce scores in the same units; the max simply picks whichever is numerically larger, and since a short title field nearly always produces a larger BM25 score for the same count, dismax implicitly favours the shorter field. That is often correct, but it is an accident of length normalization rather than an explicit weight.

To get explicit field importance with max-combination, the subquery scores must be scaled. Applying a boost to a subquery before the max means the maximum is chosen over weighted scores, and a heavily boosted body field can beat an unboosted title. Boosts multiply the score, and because BM25 scores are unbounded above in principle, the ordering effects are not linear in the way a weight on a linear model would be.

The other thing it does not fix is that max-combination discards information. A document matching the query term in title, body, headings and anchor text loses everything except the largest contribution, scaled down to a tiebreaker fraction. If those co-occurrences are genuinely informative, and for navigational queries they usually are, the model throws away a signal that a folding approach would keep.

Choosing between folding and maxing

The decision turns on whether field matches are independent evidence or alternative evidence for the same fact.

Alternative evidence is the common case. If a document is about “inverted index”, the title and the body are two observations of one underlying property, and the second observation should raise confidence sublinearly, not additively. Max-combination expresses exactly this. Folding into a weighted pseudo-frequency expresses a stronger version of it: the counts are merged and then saturated once, so the marginal value of a second field match is governed by the same curve as a second occurrence in one field.

Independent evidence is rarer and usually not about fields at all. Two different query terms matching are independent evidence; one term matching in two fields is not. This is why the saturating denominator in BM25 Term Saturation and Document Length Normalization is acceptable per-field but dangerous across fields.

A practical arrangement used in multi-field engines combines both: fold the counts within a field family (all body-like fields share a length statistic), keep field families scored separately, and combine families by max with a small tiebreaker. The consequence is that a document matching in three families gets a bounded bonus rather than three full scores.

Failure modes at the edges

A subquery that matches no documents contributes nothing, and with max-combination the maximum over an empty set is undefined. Implementations skip unmatched subqueries and fall back to a null score for documents matched by none, which is why a dismax query over a field a document does not have must not accidentally treat “field absent” as “score zero” when other subqueries produce negative scores. Negative scores are possible with additive idf, and with a negative maximum the tiebreaker arithmetic inverts: subtracting the max adds rather than reduces. Clamping subquery scores to non-negative before combination avoids the inversion.

Ties in the maximum are another edge. When two fields produce exactly the same score, which one is treated as “the max” is implementation-defined unless the arithmetic is written symmetrically. The formula above handles it because the same value is added and subtracted, but an implementation that stores an index of the winner and then subtracts scores[winner] can differ in the last bits from one that subtracts the numeric maximum.

Field length statistics are computed per field, not per document, so avgdl for a title field is the mean title length across the collection. Mixing a title-field length into the body-field normalization produces scores that look reasonable but compare documents on inconsistent bases. When fields are added to an index whose statistics were not rebuilt, the new field’s average starts at a degenerate value and the ranking shifts until enough documents are indexed.

Where this sits relative to learned ranking

Both combination rules are hand-designed attempts to answer a question a model can answer directly: how much is a title match worth relative to a body match? Once ranking moves to a learned model, field-level match signals become features and the combination becomes a learned weight, which removes the awkwardness of choosing between max and sum. The feature extraction that replaces these rules is described in Query-Document Feature Extraction; the hand-designed scores described here typically survive in the stack as one feature among many rather than as the final ordering.