Information Retrieval · Cluster
Query Understanding
Tokenization, expansion, and intent.
- 01 Unicode Word Boundary Rules in Query Tokenizers Why splitting a query on whitespace mis-tokenizes apostrophes, hyphens, and scripts that do not use spaces, and how the Unicode segmentation rules decide each case.
- 02 Token Positions and Offsets in Analyzer Output How a token stream records where each term occurred, why position increments encode adjacency for phrase and proximity queries, and how offsets survive synonym expansion.
- 03 Stop-Word Removal and the Phrase Queries It Breaks Why dropping the most frequent terms from the index saves little space today, how position gaps preserve phrase structure, and which queries silently lose meaning.
- 04 Suffix-Stripping Stemmers and the Conflations They Force How the Porter and Snowball algorithms decide that a suffix may be removed, why the result is not a word, and which pairs of distinct concepts collapse together.
- 05 Synonym Expansion at Index Time Versus Query Time Why expanding a thesaurus into the index and expanding it into the query produce the same postings but different costs, and how each choice changes scoring and phrase matching.
- 06 Edit-Distance Candidate Generation for Query Spelling Correction How a corrector enumerates candidate terms within a bounded edit distance, why the noisy-channel ranking decides the winner, and when a correction should be suppressed entirely.
- 07 Query Segmentation into Phrases for Ranked Retrieval Why a query without quotes still contains phrase structure, how segmentation models recover it, and what changes when segments are scored as units instead of loose terms.
- 08 Inferring Query Intent from Click Entropy and Reformulation How the distribution of clicks across results, and the way users rewrite a failing query, distinguish navigational from exploratory intent without any labelled training data.
- 09 Pre-Retrieval Query Performance Prediction How a system estimates, before seeing any judged results, whether a query will be answered well, using score distributions, term specificity, and clarity over the collection language model.
- 10 Pseudo-Relevance Feedback and the Rocchio Drift Problem How a system rewrites a query using its own top results, why the Rocchio weights and RM3 interpolation decide the outcome, and how feedback amplifies an initial ranking error.