Personalization Signals and Click-Based Reranking

A ranking function that scores a document against a query alone treats every user identically. Two people typing the same query receive the same list, regardless of the fact that one searches for “mercury” as an element and the other as a planet, or that one has a long history of clicking results from a particular source. Personalization adds a second input, the user’s history and context, and the mechanism that consumes it is a form of click-based reranking layered on top of the query-only score.

What the signals are

The raw material is a stream of interactions: the queries issued, the results shown for each, and the clicks that followed, together with their positions and the time spent before the next action. Two kinds of signal are extracted from this stream. Long-term signals describe the user as a whole: the topics of documents they have clicked, the sites they favour, the queries they repeat. Short-term signals describe the current session: the queries already issued and the results already examined, which constrain what the current query is likely to mean.

The position of a click is the most contaminated signal in the set and the one that most explicitly requires a model. Craswell and others documented the position bias: a result is far more likely to be clicked when it appears at the top of the list than at the bottom, not because it is more relevant but because it is seen first and trusted. A raw click count therefore over-weights top positions. The click model corrects this by estimating, per position, the probability of being examined, and attributing a click only to relevance after discounting that examination probability.

The models Craswell compared disagree about how the examination probability and the relevance probability combine, and that disagreement is not academic, because it changes how heavily a click at a given rank counts as evidence of relevance. A cascade model assumes the user scans top to bottom and stops at the first relevant result, so a click at rank five implies the four results above were examined and found wanting. A simpler model treats examination as a position-only probability independent of what sits above. The cascade assumption makes the higher-ranked non-clicks informative in a way the position-only model does not, and the choice of model therefore changes which clicks the personalization layer treats as strong signals.

The feature vector and the rerank

The personalization layer takes the query-only ranking and its features, and augments them with user features. The resulting vector passed to the reranker might include the query-document score already computed, the user’s historical affinity for the document’s site or topic, the recency of the user’s last interaction with that source, and the match between the query’s detected intent and the user’s long-term interests. The reranker is typically trained with a learning-to-rank objective, taking the augmented feature vector and producing a new score.

The training target is itself a modelling decision. A click is not relevance, it is a proxy with the position bias above and with noise from mis-clicks, so the target is usually a derived label: a click that survives a position-bias correction, a dwell time above some threshold, or a subsequent action such as a follow-up query being issued, which indicates the click did not satisfy the need. Training on the raw click trains the system to reproduce the bias, which is a self-reinforcing failure: the more the system ranks a document high, the more it is clicked, the more the training data confirms it.

The feature vector can be written in the form the reranker consumes. Given the query-document score s, a site-affinity feature a, a topic-match feature t, and a recency feature r, the reranker computes a weighted combination whose coefficients are learned from historical interactions:

score' = w1 * s + w2 * a + w3 * t + w4 * r + b

The coefficients are the personalization itself: a user whose clicks cluster on one site gets a larger w2 in their effective model, a user whose interests are broad gets a smaller one. The model is typically shared across users, with the personalization expressed through the feature values and a small set of per-user adjustments, because a fully per-user model has too little data to estimate and too much storage to maintain at scale. The bias term b and the weights are what the learning-to-rank objective fits, and the whole expression runs as a rescoring pass over a short list of candidate documents rather than over the full index.

The cold-start and privacy bounds

Personalization fails in two places that are visible to the user. A new user with no history has no personalization to apply and receives the query-only ranking, which is the correct fallback but also the reason a personalized system behaves differently for a returning user than for a first-time one. And personalization that is too aggressive produces a filter effect: the user’s past clicks narrow the results they are offered, which narrows what they click next, a feedback loop that entrenches the existing preference at the expense of exploration.

The filter effect has a concrete mechanism. The reranker’s site-affinity feature, the w2 term, grows as the user clicks results from one source, and as it grows, that source is ranked higher, is clicked more, and the feature grows again. The loop is self-reinforcing precisely because the training target and the feature both derive from the same clicks, so a system with no explicit exploration cannot observe what it has stopped showing. The standard countermeasure is to inject a small amount of diversity into the list, either by reserving a fraction of slots for results the user has no affinity toward, or by capping the contribution of any single feature so that no one source can dominate.

The privacy constraint shapes the engineering. Because the signals are a persistent record of a person’s interests, the storage, retention, and linkage of the history are governed by design decisions that are as much about what may be kept as about what is useful. A system can degrade the signal, for example by aggregating or aging it, and the trade is always between the accuracy of the personalization and the sensitivity of what is stored. The result grouping layer and the pagination layer sit below personalization in the stack: grouping and paging operate on whatever list the reranker produces, and must remain stable even as the reranker changes the order from one user to the next.

Personalization is a reranking layer that converts a static query-only score into a per-user score using a history the engine is explicitly permitted to keep. Its two hard parts are the click model that removes position bias before a click is treated as evidence, and the training target that separates a satisfied click from a mistaken one. Both are attempts to measure relevance from behaviour that is only a noisy proxy for it.

The layer is also the most version-sensitive part of a search interface, because the reranker’s coefficients are retrained on new interaction data on a schedule, and a model update can change the ranking for the same query and the same user from one day to the next. That version dependence is invisible to the reader, who sees only that a familiar query now returns a different order, and it is a source of the kind of instability that the pagination article treats as a write-path concern but that here originates in the ranking model itself rather than in the index.