Position Bias in Pairwise Preference Judging

Pairwise comparison asks a simpler question than absolute scoring: given two responses to the same prompt, which is better. The simplification is real. Two outputs can be ordered by a reader with reasonable consistency even when neither can be assigned a defensible score out of ten, and the comparison avoids the problem of calibrating a scale across prompts of wildly different difficulty. It introduces a different problem, which is that the comparison is presented in an order, and the order is not neutral.

A judge that reads response A before response B has already formed an impression by the time B arrives. The impression acts as a prior. When the two responses are close in quality, that prior decides the verdict. The effect is strong enough that a judge’s agreement with itself, on the same pair presented in both orders, is far from perfect, and the disagreement rate is a directly measurable property of the judging setup.

The swap experiment and what it measures

Take a set of prompt-response pairs, present each pair once as $(A, B)$ and once as $(B, A)$, and record the verdict both times. Four outcomes are possible for each pair. The judge can prefer the same response in both presentations, which is consistent; it can prefer the response that appeared first in both, which is first-position bias; it can prefer the response that appeared second in both, which is recency bias; or it can tie in one presentation and prefer something in the other.

The rate of consistent preference is the quantity that matters. If a judge is order-invariant, every pair yields the same winner in both presentations, and the consistency rate is 1. The observed rate is lower, and the gap between 1 and the observed rate is the share of verdicts that were decided by position rather than content.

The two failure directions have different causes. First-position bias comes from the judge anchoring on the opening response and evaluating the second against it. Recency bias comes from the second response being fresher in context when the verdict is produced. Which one dominates depends on the judge and on how the prompt is framed, and it can flip when the framing changes.

A judge’s position bias is not a fixed constant. It interacts with length: longer responses tend to be preferred, and a judge that reads a long response first may apply a length prior to the second. It interacts with quality distance: when one response is clearly better, the order matters less, because the content signal overwhelms the prior. It also interacts with the rubric, since a rubric that asks for a specific comparison criterion focuses attention on that criterion and reduces the influence of the opening impression.

Correcting for it without doubling the cost

The obvious fix is to run every pair in both orders and count a winner only when both presentations agree, treating disagreement as a tie. This is correct in the sense that it removes position as a factor from the final tally, but it costs twice as many judgments and it throws away information: a pair where the judge preferred A first and B second is genuinely ambiguous, and calling it a tie is a defensible but lossy resolution.

The alternative is to keep the duplicated verdicts and model the bias explicitly. If the probability of preferring the first-presented response is inflated by a factor $\beta$ independent of content, then the two presentations give two observations of the same underlying comparison with the bias applied in opposite directions. A logistic model over the pair, with a per-presentation offset, recovers the content-driven preference and estimates $\beta$ from the same data. The approach is the one used in paired-comparison rating systems, where each match contributes evidence about both participants’ strengths, and an explicit order term absorbs the presentation effect.

import math

def fit_order_effect(records):
    """records: (score_a, score_b, a_first, winner_is_a).
    Fits P(a wins) = sigmoid(s_a - s_b + beta * (1 if a_first else -1))."""
    beta = 0.0
    for _ in range(200):                      # fixed-step gradient ascent
        grad = 0.0
        for s_a, s_b, a_first, a_wins in records:
            z = s_a - s_b + beta * (1 if a_first else -1)
            p = 1.0 / (1.0 + math.exp(-z))
            sign = 1 if a_first else -1
            grad += (a_wins - p) * sign
        beta += 0.01 * grad
    return beta

Fitting this requires the judge to produce a probability or at least a stable verdict, and it requires enough pairs to estimate one extra parameter, which is a mild requirement relative to the cost of the judgments themselves. The estimate of $\beta$ is itself a useful diagnostic: it says how much of the verdict distribution is explained by presentation order.

Why this is not only a model-judge problem

Human annotators exhibit the same effect, and the annotation literature has known it for decades. Studies of rater behaviour in essay scoring found that raters drift within a session, that a rater’s score depends on the essays seen immediately before, and that the order of presentation within a batch shifts scores systematically. The standard mitigations in that literature are the ones that carry over: randomize presentation order across items, balance the design so that each item appears in each position equally often, and check for order effects in the collected data rather than assuming they are absent.

Balanced design is the cheapest correct answer when the total number of judgments is fixed. If every pair is shown in both orders to different judges, or to the same judge across different sessions, the position effect cancels in the aggregate comparison without any modeling. What it does not do is cancel the effect at the level of an individual item, so a per-item score derived from a single presentation remains biased even when the corpus-level comparison is clean.

The distinction matters for how results are reported. A system-level win rate computed from a balanced design is unbiased even if every individual verdict is biased. A per-prompt analysis, or any claim about which specific prompts a system handled better, inherits the bias and needs the per-item treatment.

Length, verbosity, and the confound underneath

Position is the easiest bias to measure because the experiment is a clean swap. The harder confound is that judges prefer longer responses, and longer responses are also more likely to contain the content a rubric asks for. A judge that rewards length will prefer the longer response in both presentations, so the swap experiment reports perfect consistency and concludes there is no position bias, while the verdict is being driven by a property unrelated to quality.

Separating length from quality requires holding one fixed: pairs matched on length, or a regression with length difference as a covariate alongside the order term. Neither is free, and the confound is not removable in general, since length and informativeness are genuinely correlated in real outputs. What is removable is the claim that a preference win rate measures quality. It measures preference, and preference has these components.

The same reasoning applies to formatting. A response with headings and bullet points is easier to scan than a dense paragraph of the same content, and a judge reading quickly will prefer the formatted one. That is a real effect on the reader experience and a spurious effect on the quality estimate, and the two cannot be separated by asking the judge, which has no independent access to the content.

What a win rate is a rate of

A pairwise win rate is a proportion, and proportions computed from a few hundred comparisons carry wide uncertainty. With 200 comparisons, a win rate near 0.5 has a standard error around 0.035, so a reported difference of three points between two systems is inside the noise. The interval arithmetic is the same as for any binomial proportion, and it is worth computing before drawing conclusions from a small judged set; the calculation is the one described in confidence intervals for evaluation sets.

The verdict also inherits whatever the judge’s notion of quality is. A judge trained to prefer helpful, concise, well-organized prose will rank systems by their ability to produce that, which is a legitimate target and not the same target as factual accuracy, safety, or instruction adherence. A win rate against a reference system therefore measures proximity to the judge’s preferences, and the reference system’s own position on those axes sets the scale. Reporting the judge, the prompt template, the order handling, and the reference system is what makes the number interpretable at all; reporting only the percentage is reporting a preference for an unstated judge’s taste.

One further design choice affects the size of the position effect: whether the judge sees both responses in one context window or evaluates each separately and then compares verdicts. A judge reading both together has the first response present while reading the second, which is the condition that produces anchoring. A judge scoring each response independently against a rubric has no such comparison available and cannot exhibit first-position bias at all, though it acquires the separate problem of scale drift between the two scorings. The single-context pairwise design is used because absolute scores are poorly calibrated across prompts, and the price of that choice is the bias the swap experiment measures.

When the judge’s verdicts are used to validate a factuality metric rather than to rank systems, the same order handling applies, and the claim-level verdicts described under atomic claim decomposition inherit the bias if the supporting evidence is presented before the claim rather than after it.