Stop-Word Removal and the Phrase Queries It Breaks

A phrase query for the who returns nothing from an index that dropped the. The band, the vitamin, the man-of-war, the Bank of anything: all of these are sequences of a common word plus a rare one, and any pipeline that deletes common words without recording what it deleted has made those sequences unrepresentable. Stop-word removal is the oldest optimization in information retrieval, and it is also the one whose failure modes are least visible, because a dropped postings entry produces a smaller index rather than an error.

The argument for removal, and what has changed

Common words are common. Under Zipf’s law, the frequency of the term at rank $k$ falls roughly as $1/k$, which means a few dozen words can dominate the total number of token occurrences in a collection. Because each occurrence contributes a postings entry, the same few words dominate index size, and a classic stop list of a few hundred entries removes a large share of postings while costing almost no retrieval effectiveness on ordinary content queries.

That trade looked different when indexes were stored uncompressed. Removing the top few hundred terms shrank the term dictionary and the postings substantially, and it reduced the number of postings a query had to intersect, which mattered when intersection meant reading from disk. Two decades of compression changed the arithmetic. Delta-encoded postings with variable-byte packing make a long postings list cheap to store, and skip lists or block-max structures make it cheap to advance through one. The savings from removal are now modest, while the cost of annotating positions for the surviving terms remains.

The other half of the classical argument was that stop words carry no information for ranking. That is nearly true, and inverse document frequency already implements it: a term appearing in every document contributes almost nothing to a score. Removing it explicitly and letting IDF do it implicitly arrive at similar rankings, except that the explicit removal destroys the position information needed for phrases.

Two ways to drop a term, and what each does to positions

If a filter removes a token from the stream, it must decide what happens to the position of the next token. There are exactly two coherent choices.

The first preserves the gap. The removed token contributed an increment, and the filter adds that increment onto the next surviving token, so new york the pizza yields new at position 0, york at 1, and pizza at 3. A phrase query for "new york pizza" compiles to terms at relative positions 0, 1, 2, and the index does not satisfy it, which is correct: those words are not adjacent in the document. The price is that any phrase containing a stop word can never match a reduced index, because the query analyzer drops the same word and emits the same gap. "the who" analyzes to who, so the query no longer expresses what the user asked for.

The second option closes the gap. The filter drops the token and does not increment, so new york the pizza becomes positions 0, 1, 2. Phrases now match across removed words, which is the behaviour that makes "new york pizza" return a document that reads new york, the pizza capital. Whether that is a bug depends on the use case; for snippet generation and quoted search it usually is.

A third variant emits alternative terms at the same position, which is how synonym and decomposition filters work and which the position and offset mechanics describe in more detail. It is not a stop-word strategy, but a filter that keeps a zero-width marker where the removed term stood is effectively using it.

Queries that lose meaning, not just matches

The visible failures cluster into a few classes.

Proper nouns that begin with a common word are the largest class: the who, the police, vitamin a, bank of america, a star is born. Removal reduces each to its rare word or leaves a fragment, and no amount of position bookkeeping recovers the intended string.

Queries composed entirely of common words are the second class. to be or not to be reduces to whichever of its words the list did not contain, and after removal the surviving terms may be non-adjacent in the source text under gap preservation, or adjacent but meaningless under gap closure. Either way, the user who quoted the line gets a different query than the one typed.

Negation is the third, and the least discussed. If not and no are on the stop list, a query meaning “documents about X but not Y” loses its operator. In practice this is a reason to keep negation operators out of the removed set even when the corresponding words are frequent, since the operator carries semantics that the word alone does not.

Common grams and other recoveries

The standard fix in analyzer pipelines is to index the removed words anyway, but only in combination with their neighbours. A common-gram filter emits the single terms plus bigrams formed across stop words, so new york the pizza produces new york, the pizza, and the ordinary single tokens, with the bigram occupying the position of its first word. A phrase query containing a stop word is then rewritten into the corresponding bigram, and "the who" becomes a lookup for the two-word term. This costs dictionary entries proportional to the number of distinct stop-word bigrams, which is bounded by the vocabulary adjacent to stop words, and it is unaffected by whether the single terms themselves are removed.

The alternative is to keep the full token stream and accept the postings. Given compressed postings and IDF that already suppresses the contribution of ubiquitous terms, many modern configurations index everything and reserve stop-word filtering for the query side only, where it reduces the number of terms a scorer must handle without altering what the index contains. That combination is asymmetric by construction, and the asymmetry is safe precisely because it only affects scoring, not matching. It is not safe for phrase queries: if the query-side filter removes a word from inside a quoted phrase, the phrase changes meaning even though the index still has everything.

A stop list that is applied only to the query side does have one measurable benefit beyond scorer cost: it prevents a term like the from contributing to a boolean requirement. In a system that treats all query terms as required, every stop word in the query narrows the candidate set for no topical reason, and removing those terms widens it back. In a system that treats terms as optional that effect does not exist, because scoring already discounts the term through its document frequency. Which behaviour applies depends on the query parser, so the same stop list can be neutral in one deployment and a recall fix in another.

Choosing a list, and what its size controls

A stop list is a parameter of the pipeline and has to be identical on both sides of it. The lists in circulation differ by an order of magnitude: small lists of a few dozen of the most frequent function words, and longer lists of a few hundred entries that also drop common verbs and adverbs. Longer lists save more postings and cause more of the failures above. There is no principled cut point, because frequency is a continuous quantity and the set of words that are semantically empty in one domain (may in a date corpus, will in an estate corpus) is content in another.

The practical rule that emerges from the mechanics is narrow: either remove stop words with position gaps preserved and refuse to support phrases containing them, or index the words and let scoring decide their value. Removing words and closing the gaps produces an index whose phrase matches do not correspond to phrases, and the symptom is a ranking that looks fine on individual term queries and subtly wrong the moment anyone searches for a quotation. Term-level normalization steps such as suffix stripping interact with this choice, because a stemmer applied after removal sees a token stream whose adjacency has already been rewritten.