Score Calibration for Blended Ranking

A ranker trained with a pairwise or lambda objective is trained to get the order right. Nothing in its loss constrains the magnitude of its output, so two models that produce identical orderings can produce scores on wildly different scales, and the same model can produce scores whose range depends on the query. The moment a score is used for anything other than sorting within one query, that freedom becomes a defect.

The three places an uncalibrated score causes damage

The first is blending. Combining a learned ranker with a static prior, a vector similarity, or a business rule requires adding or multiplying the numbers, and addition of quantities on different scales produces a result dominated by whichever has the larger variance. A tree ensemble’s output typically sits in a narrow band around a mean, while a BM25 score ranges over an order of magnitude depending on query length. Adding them unnormalized means the BM25 term decides the ranking and the ensemble contributes a small perturbation.

The second is thresholding. A system that shows a “no good results” message, or that gates an expensive downstream stage on confidence, needs a score with a fixed meaning. “Score above 0.7” is only a policy if 0.7 corresponds to a stable probability of relevance. On an uncalibrated scale, 0.7 might be the 40th percentile for one query and the 95th for another.

The third is cross-query comparison. Merging results from several queries, as in a federated search across collections or a recommendation feed mixing results for different inferred intents, requires comparing documents that were scored against different queries. Without calibration there is no basis for the comparison, and the merge is decided by whichever query happened to produce higher raw scores.

What calibration means precisely

A score is calibrated if it equals the probability of the event it claims to estimate. For a binary relevance label, calibration means that among all documents assigned score $s$, the fraction that are relevant is $s$. This is a statement about a population, not about any individual document, and it can only be checked in aggregate.

The standard diagnostic is the reliability diagram: bin the scores, plot the observed relevance rate against the mean score in each bin, and look for deviation from the diagonal. Two summary statistics capture the common failures. Expected calibration error averages the absolute deviation across bins, weighted by bin population. A refinement separates the two ways a model can be wrong: a model can be systematically overconfident, with scores too extreme in both directions, while still being well ordered, or it can be well calibrated on average while being badly ordered within a region.

The distinction matters for ranking because ordering and calibration are independent properties. A monotone transform of the scores changes calibration and leaves the ordering untouched, so any calibration procedure that is monotone cannot fix an ordering problem and any ordering problem cannot be fixed by calibration. Isotonic regression, the standard nonparametric calibrator, is monotone by construction.

Fitting a monotone map

Isotonic regression solves for the monotone non-decreasing function that minimizes squared error against the observed labels, subject to the monotonicity constraint. The solution is a step function found by the pool-adjacent-violators algorithm: walk the sorted scores, and whenever a violation of monotonicity appears, meaning the observed rate decreases as the score increases, merge the two adjacent blocks and replace both with their weighted average.

def isotonic(scores, labels):
    """Pool-adjacent-violators; returns (block_mean, block_weight)."""
    order = sorted(range(len(scores)), key=lambda i: scores[i])
    blocks = []  # [mean, weight, max_score]
    for i in order:
        blocks.append([float(labels[i]), 1, scores[i]])
        while len(blocks) > 1 and blocks[-2][0] > blocks[-1][0]:
            m2, w2, s2 = blocks.pop()
            m1, w1, _ = blocks.pop()
            w = w1 + w2
            blocks.append([(m1 * w1 + m2 * w2) / w, w, s2])
    return blocks

The output is a step function: every score up to the first block’s maximum maps to that block’s mean. New scores at prediction time are mapped by finding the block whose maximum exceeds them. This is the nonparametric answer, and it needs a held-out set: fitting on training data produces a map that memorizes the training labels and is monotone but overfit.

Platt scaling is the parametric alternative: fit a logistic function $1/(1 + \exp(a s + b))$ to the scores. It has two parameters, so it cannot overfit in the way isotonic can, and it produces a smooth, strictly monotone map. Its limitation is the shape assumption: if the true relationship between score and relevance rate is not sigmoid, Platt scaling will be systematically wrong at one end. The classical result from the probability-estimation literature is that both methods improve poorly calibrated models substantially and that the choice between them matters less than whether any calibration is applied.

def platt_fit(scores, labels, iters=200, lr=0.01):
    a, b = 1.0, 0.0
    n = len(scores)
    for _ in range(iters):
        ga = gb = 0.0
        for s, y in zip(scores, labels):
            p = 1.0 / (1.0 + math.exp(-(a * s + b)))
            ga += (p - y) * s
            gb += (p - y)
        a -= lr * ga / n
        b -= lr * gb / n
    return a, b

The gradient steps above are the plain logistic-regression gradients on the score as a single feature. Fitting Platt scaling is exactly fitting a one-feature logistic regression, which is a useful way to remember it: the calibration map is a model, and it needs the same care about train/test separation as any other model.

Query-level versus global calibration

A single global map assumes the score distribution is stable across queries. For a tree ensemble over query-dependent features that assumption is false, and the failure is systematic rather than noisy.

Consider a query with many matching documents and a query with three. The ensemble’s scores for the first span a range; for the second they cluster. A global isotonic map fitted on a mixture will map the narrow cluster to whatever the average relevance rate at that score value is, which can be badly wrong for the specific query. The documents in a three-document query are being compared against the wrong reference population.

Two approaches address this. The first conditions the calibration on query features (query length, the score spread within the candidate set, the fraction of the collection matched) by fitting a separate map per bucket. This is more data-hungry and the buckets must be defined before fitting, but it captures the first-order dependence.

The second normalizes within the query before applying a global map: rank the candidates, convert ranks to a within-query distribution, and calibrate that. This discards magnitude entirely and reduces to a rank-based calibration, which is sound for cross-query comparison but throws away the confidence information the scores carried.

There is a third path that is often the right one: do not calibrate the learned score at all, and instead calibrate the final blended score. If the output of the whole stack is what gets thresholded or compared across queries, the intermediate scores’ scales are irrelevant and only the composite needs to be interpretable. This is simpler and avoids the problem of propagating calibrated intermediate values through a nonlinear combination, which would destroy the calibration anyway.

The shift that invalidates a fitted map

A calibration map is fitted on a sample of queries and documents, and it encodes the relationship between score and relevance for that sample. Three things change it.

A change in the ranker invalidates the map entirely, since the map is a function of the score distribution and the new model produces a different one. Retraining a model and reusing the old calibration is a common and silent error: the ordering improves and the calibrated probabilities become wrong in a way no ranking metric detects.

A change in the query mix shifts the population. If the traffic mix moves toward a category the model handles differently, the aggregate relevance rate at a given score changes even if the model is unchanged. The map is then wrong in aggregate while being right for each subpopulation, which means it should be refit per segment.

A change in the judgment policy shifts the labels. If the assessment guidelines change, for instance if what counts as “relevant” is tightened, the target the calibration predicts changes, and the map must be refit against the new labels. A map fitted against a loose policy overestimates the probability of relevance under a strict one.

Calibration as a diagnostic rather than a product

Because calibration is checked against held-out labels, the reliability diagram is one of the few tools that detects a class of failure ranking metrics miss. A model can improve NDCG while becoming badly overconfident: it pushes the top documents to extreme scores and the rest to very low ones, which sharpens the ordering and makes every score meaningless as a probability. The NDCG curve looks better; the reliability diagram shows the scores clustering at 0 and 1 with observed rates near 0.5 in both bins.

The inverse also happens. A model whose scores are compressed, because of strong regularization or because the ensemble is shallow, can be well calibrated and poorly ordered, and the calibration check passes while the ranking is mediocre.

Neither diagnostic subsumes the other, which is the reason to run both. And neither detects the failure that comes from the labels themselves: if the judgments are biased toward documents the current system ranks highly, both the ordering metric and the calibration curve will agree that the model is excellent, because both are measured against the same biased target. The bias problem in judgments is the same one that afflicts click-derived labels, described in Click Features and Position Bias.

Where calibrated scores change the design

Once scores carry a probability interpretation, several things that were heuristics become policies. A cutoff for “insufficient results” can be set from a target false-negative rate rather than by eyeballing a score. A blend weight between two retrievers can be derived from the relative reliability of their probability estimates rather than tuned. And a cascade can decide per document whether the expensive stage is worth running, because the cheap stage’s score now means something.

The cost is that the calibration must be maintained, refit whenever the model or the traffic changes, and evaluated separately from the ranking metric. That maintenance burden is the reason many systems skip calibration and use rank fusion instead, accepting that their scores are ordinal-only. For a system whose final output is a sorted list and nothing else, that is a defensible choice. For a system that blends, thresholds, or merges across queries, it is not, and the next layer up, how to order a list so that it is not merely a list of near-duplicates, is the subject of Diversity-Aware Ranking, which depends on comparable scores to decide when two results are redundant.