Click Features and Position Bias in Ranked Lists

Users click on results. The temptation is to treat clicks as relevance labels: a document clicked often is a document people wanted. The observation that undermines this is simple and fatal: the top result receives far more clicks than the fifth, regardless of quality, because it is seen more. Any model trained on raw clicks learns what was shown rather than what was good, and because the model then determines what is shown, the bias compounds.

The examination hypothesis and its consequence

The minimal click model splits a click into two events. The user examines a result at position $k$ with some probability that depends on the position, and, conditional on examining, clicks with a probability that depends on the document’s relevance. A click is the conjunction:

$$P(\text{click} \mid d, k) = P(\text{examined} \mid k)\cdot P(\text{click} \mid \text{examined}, d)$$

The examination probability is assumed independent of the document, which is an approximation: a result that looks irrelevant from its snippet may never be examined carefully even if it is in the first position. But the approximation is useful because it separates the nuisance factor from the quantity of interest.

The consequence for feature design is direct. A raw click count is $E[\text{clicks}] = \sum_k n_k \cdot o_k \cdot r_d$, where $n_k$ is the number of impressions at position $k$, $o_k$ is the examination probability, and $r_d$ is the document’s relevance. A count conflates the three. A ratio of clicks to impressions does not fix it either, because it is a weighted average of $o_k r_d$ over the positions the document happened to occupy, and a document that was only ever shown at rank one looks better than one that was shown at rank seven.

The usual partial fix is to compute click-through rate per position and compare documents only against documents that occupied the same position bucket. This does not remove the bias, since documents are not assigned to positions randomly, but it prevents the crudest version of the confound.

Estimating the examination curve

If $o_k$ can be estimated, the bias can be divided out. There are two practical routes.

The first uses randomized traffic. For a small fraction of impressions, results are shuffled or randomly swapped between positions. Clicks on the randomized subset are unbiased with respect to position, and comparing the click rates of documents that were randomly assigned to different positions yields an estimate of the ratio $o_j/o_k$. Randomization is the honest method and the reason it is not universal is the cost: shuffled results are worse for the users who receive them, so the randomization rate is kept small and the estimates are noisy.

The second uses a parametric form. A common assumption is that examination decays as $1/k$ or as a power law, and a single parameter is fit to aggregate data. The assumption is convenient but the fitted curve absorbs the average relevance of the documents at each position, so a search engine whose ranker is good will fit a steeper decay than one whose ranker is poor. The estimate is not a property of user behaviour alone.

def ipw_weight(position, o):
    """Inverse propensity: upweight rarely examined positions."""
    if o[position] <= 0.0:
        raise ValueError("zero examination probability")
    return 1.0 / o[position]

The weight above is the core of inverse propensity weighting: an observation at a position with low examination probability is scaled up to represent the many similar observations that would have occurred had that position been examined as often as the top. The estimator is unbiased under the assumption that clicks are missing at random conditional on position, and its variance is large precisely where the weights are large.

What inverse propensity weighting does to the loss

Training a ranker with weighted clicks changes the loss from “predict the observed labels” to “predict the labels the full-information world would have produced”. Each training instance carries a weight; a click at a position with $o_k = 0.1$ contributes ten times a click at $o_k = 1.0$.

The variance consequence is structural. The effective sample size is not the number of observations but roughly $(\sum w_i)^2 / \sum w_i^2$, which shrinks as the weight distribution spreads. Deep positions contribute many observations with large weights and large individual variance, so the estimator trades bias for variance at a rate that depends entirely on how steep the examination curve is. With a curve that assigns $o_k$ well below the top few positions, the deep observations are nearly worthless and the weighting produces noisy gradients rather than unbiased ones.

Clipping the weights, that is capping $1/o_k$ at some maximum, reduces variance at the cost of re-introducing bias at the deepest positions. The cap is a hyperparameter with no principled setting; it is chosen so that training is stable. The loss the weighted observations feed into is a separate decision, covered in Learning-to-Rank Objective Functions.

There is a second subtlety: the propensity is a property of the position, but the number of impressions at each position is not independent of the document. A document shown thousands of times at rank one is a document the current ranker likes, which correlates with its true relevance. Dividing by $o_k$ corrects for examination but not for the selection that put the document at rank one, and only randomization or a model of the current ranker’s assignment corrects for that.

Abandonment, satisfaction, and the click that is not a signal

Treating a click as a positive label assumes the user clicked because the result was relevant. Users also click to verify a suspicion, to check a price, or because the snippet was ambiguous. The result they wanted may be the one below, and they may have clicked the one above only to confirm it was wrong.

Two observable behaviours improve the signal. The first is dwell time: a click followed by a quick return to the results indicates dissatisfaction, and a click followed by a long session indicates the user found something. A threshold on dwell time converts clicks into positive and negative labels. The exact value depends on the corpus, and the choice is a heuristic.

The second is the last-click pattern: the final click in a session that ends without a reformulation is more likely to be the satisfying one than the earlier clicks. Models built on this pattern treat the whole prefix of the session as navigational noise.

Click models such as the cascade model formalize the sequential reading: the user examines results in order, stops at the first click, and never looks below it. That assumption explains why rank-one clicks dominate, and it predicts the absence of clicks below the first click rather than their presence. It also makes the strong prediction that a document below a clicked result has zero examination probability, which is false when users scan before clicking.

Clicks as features rather than labels

The distinction that matters for feature design is between using clicks to train the model and using click aggregates as input features.

Using clicks as training labels means the model learns from behaviour, with all the bias and variance issues above. Using click aggregates as features means the feature vector, whose other members are described in Query-Document Feature Extraction, contains a historically observed click rate for the query-document pair. That feature is available at query time only if the pair has been shown before, which makes it a memorization signal: it helps for head queries with abundant history and is absent for tail queries. A model trained with such a feature learns to rely on it, and its performance degrades on the tail where the feature is missing. Handling missingness explicitly, with a separate indicator feature and a default value, is necessary, and the model’s behaviour on the tail must be evaluated separately.

There is also a feedback loop to contend with. A document that ranks high accumulates clicks, and its click feature rises, which keeps it ranking high. The loop is benign if the feature is genuinely informative and the ranker is good; it is a trap if the ranker is mediocre, because the feature locks in the initial ordering. Exploration, deliberately showing some documents at positions they do not deserve, is the standard mitigation, and it is the same randomization that produces propensity estimates. The two purposes coincide, which is why exploration budgets are often shared.

Time windows and the drift the features cannot see

Click features are computed over a window, and the window length sets the trade between stability and responsiveness. A short window tracks trending queries and lets a document’s reputation rise and fall, but produces features with high variance and lets a single viral day dominate. A long window is stable, and is wrong about anything that changed.

Normalizing by impressions rather than using raw counts makes the feature comparable across documents with different exposure, which is a minimum requirement. Beyond that, the useful transformation is a smoothing that pulls sparse observations toward a prior. A document clicked once out of two impressions should not have the same click rate as one clicked a thousand times out of two thousand. The standard technique is an additive prior, equivalent to starting every document with a small number of pseudo-impressions at the global rate.

Seasonality is a failure mode that is easy to miss. Queries about a recurring event produce a burst of traffic whose clicks are unrepresentative of the rest of the year. If the feature store holds a single aggregate, the burst pollutes it for months. Maintaining the aggregate over a window and letting old observations decay avoids this at the cost of making the feature sensitive to the decay constant.

What clicks cannot tell you

A document that is never shown has no click history, and a document shown only for queries where it is a poor match has history that says nothing about the queries where it would be good. The click signal is confined to the region of the query-document space the current system already explores, which is why click features can improve a ranker’s ordering within its current reach but rarely expand it.

Clicks also do not distinguish between the two reasons a user stops: because the result was satisfying, or because the user gave up. A session that ends after a click on a mediocre result is indistinguishable from a session that ends after the perfect result, and any model that treats session end as satisfaction will learn the difference wrongly. The label construction problem is separate from the bias problem, and the ranking stack usually carries both.

The final caution is about evaluating on clicks. A ranker trained on clicks and evaluated on clicks will appear to improve even if it has only learned the bias more thoroughly, because the evaluation data has the same bias. Evaluation needs either randomized data or a separate set of judgments not derived from the behaviour being predicted; the machinery for the latter is outside this cluster, which concerns scoring models, but the requirement constrains which features are safe.