Generating Search Result Snippets with Query-Biased Summaries

A search result page does not show the matched document. It shows a fragment of it, two or three lines long, with the query terms highlighted. That fragment is the single most read piece of text on the page, and it is produced by a summarizer that must run against every document in the result set, usually in a handful of milliseconds each, with no access to anything beyond the stored document. The fragment is called a snippet, and the technique behind it is query-biased summarization.

Why the snippet cannot be the document start

The naive approach is to take the first two lines of the document and print them. That fails for exactly the reason a search engine returns the document at all: the query terms may appear anywhere, and in a long document the first two lines are frequently a title page, a table of contents, or boilerplate. A query for “cache eviction” matched against a paper whose abstract discusses eviction policy will show the paper’s author list and affiliation if the summarizer merely truncates from byte zero.

The constraint that shapes every snippet generator is therefore simple: the fragment must contain as many of the query terms as possible, in close proximity, because proximity is what lets the reader judge relevance from a two-line excerpt. A snippet that contains three query terms within a single sentence is more informative than one that contains one term in each of three scattered positions. The summarizer’s job is to find the window of the document that maximizes some combination of term coverage and term density, then to render that window.

The term-window selection step

The standard mechanism treats the document as a sequence of positions, each position a term after tokenization. For each position, the system considers a window of a fixed size, commonly on the order of 150 to 250 characters after HTML tags are stripped, and scores the window. The earliest well-known scoring function for this step comes from Tombros and Sanderson, who described a query-biased extract summarizer for question answering. Their score rewards a window for every distinct query term it contains, with a modest bonus when the term occurs multiple times within the window.

A concrete implementation works roughly as follows. The query is reduced to its distinct terms. The document is scanned once, left to right, maintaining a map from term to the list of its positions. A window of width W is slid over the positions; a window is good if it covers a prefix of the query’s distinct terms. Because scanning is linear in document length and the window width is constant, the whole document can be summarized in a single pass, which is what makes per-document cost acceptable on a result list of ten or twenty documents. More sophisticated rankers add a term-position weighting that mirrors the retrieval score, so the fragment is extracted from the same evidence the ranking engine trusted.

The score a window earns can be written out directly. Given the query’s distinct terms and the document tokenized into positions, the window centred on each position is scored as the number of distinct query terms inside it, with each repeated occurrence adding a small constant:

def window_score(tokens, center, width, query_terms):
    start = max(0, center - width // 2)
    end = min(len(tokens), center + width // 2)
    seen = set()
    score = 0
    for tok in tokens[start:end]:
        if tok in query_terms:
            score += 1.0 if tok not in seen else 0.1
            seen.add(tok)
    return score

The loop is the inner step of the single left-to-right pass; the per-document cost is the document length times the window width, which is constant for a fixed width. The repeat bonus is the parameter that trades coverage against density, rewarding a term that occurs several times in one window without letting a single repeated term swamp the count of distinct terms.

The output of this step is a start offset and an end offset into the stripped text. Everything the reader sees afterwards is a cosmetic layer on top of those two offsets.

Making the window read as a sentence

A raw character window rarely begins or ends at a sentence boundary, so most implementations expand the chosen window outward to the nearest sentence delimiters, then clip back to the display budget if the expansion runs long. This is the step where an ellipsis is inserted at one or both ends to signal truncation. The highlighted terms are then marked inline, which is why the summarizer must retain, alongside each offset, the mapping back to the original position of each query term so the renderer can wrap the exact character ranges.

There is a subtle failure mode here. When the window is expanded to sentence boundaries and then clipped, the high-scoring query terms that motivated the window can be pushed outside the final render. A careful implementation keeps the winning term cluster pinned to the centre of the display and expands around it, rather than expanding from an arbitrary start and allowing the centre of mass to drift. The difference is visible: a snippet whose highlighted terms sit at the left edge reads as a coincidence, while one whose terms sit mid-window reads as a genuine context.

What the summary is not

A query-biased snippet is extractive, never abstractive. It reproduces verbatim character ranges from the stored text and does not rewrite. This is partly a latency decision, but it is also a correctness one: a generated sentence could misstate what the document claims, and a search result that misrepresents a source is a liability even at two lines long. The verbatim rule also makes highlighting trivially correct, since the offsets are into real text and never into generated phrasing.

The extractive choice carries a real cost. The stored text of a page frequently includes navigation, copyright lines, and cookie notices, and the summarizer has no way to distinguish those from content except by heuristics such as block density. A window that lands on a footer will still be shown if it happens to contain the query terms. Document structure helps where it is available: a title, a description meta tag, or semantic sections can seed the window, which is one reason structured markup improves result presentation independently of ranking.

A second consequence of the extractive rule is that the snippet inherits whatever the document says, including statements the reader has no context for. A fragment lifted from the middle of a comparison between two systems can appear to endorse one of them when the surrounding argument does the opposite. The summarizer cannot repair this, because repair is rewriting, and the verbatim contract forbids it. The only available mitigation is the choice of window: preferring a window anchored by a sentence that contains the query terms in a self-contained clause reduces the chance that the fragment reads against the document, at the cost of a marginally lower term density.

Snippet generation is the first thing a reader interprets, and it is a compression problem as much as a retrieval problem. It compresses a document of thousands of words into two lines that must preserve enough evidence for the reader to decide without opening the page, under a latency budget that rules out any pass that is not linear. The window selection, sentence expansion, and verbatim-rendering steps each trade something: density against readability, readability against budget, and correctness against the flexibility of a generative summarizer.