Topic domain
Information Retrieval
Finding documents and ranking what matters.
- Articles
- 80
- Clusters
- 8
- Foundational
- 24
- Intermediate
- 32
- Advanced
- 24
Indexing
10Inverted indexes and compression.
- Postings Lists as Sorted DocID Arrays
- Term Dictionary Lookup Structures for Inverted Indexes
- Delta Encoding and Variable-Byte Postings
- Bit Packing and Frame-of-Reference Blocks
- Document-at-a-Time and Term-at-a-Time Evaluation
- Front Coding the Term Dictionary
Ranking
10Scoring models and feature design.
- BM25 Term Saturation and Document Length Normalization
- Disjunction-Max Combination for Multi-Field Scoring
- Score Normalization and Rank Fusion Across Retrievers
- Static Rank Priors and Their Effect on Ranking
- Query-Document Feature Extraction for Learned Ranking
- Click Features and Position Bias in Ranked Lists
Tokenization, expansion, and intent.
- Unicode Word Boundary Rules in Query Tokenizers
- Token Positions and Offsets in Analyzer Output
- Stop-Word Removal and the Phrase Queries It Breaks
- Suffix-Stripping Stemmers and the Conflations They Force
- Synonym Expansion at Index Time Versus Query Time
- Edit-Distance Candidate Generation for Query Spelling Correction
Relevance judgments and ranking metrics.
- How Relevance Judgments Are Encoded in a qrels File
- Precision and Recall at a Cutoff: Why the Cutoff Changes the Metric
- Average Precision and the Mean Over Topics
- Graded Gains and the NDCG Discount
- Reciprocal Rank and Rank-Biased Precision
- Pooling Depth and the Unjudged Tail
Embeddings and approximate nearest neighbors.
- Cosine Similarity and the Geometry of Normalized Embedding Vectors
- Distance Concentration and Why Tree Indexes Collapse in High Dimensions
- Random Hyperplane LSH and the Arithmetic of Banding
- Coarse Quantization with Inverted Files and the nprobe Setting
- Product Quantization and the Asymmetric Distance Computation
- HNSW Layer Assignment and the Greedy Descent
Crawling
10Scheduling, politeness, and freshness.
- The URL Frontier: Priority Queues Behind Per-Host Politeness
- Matching Paths in robots.txt: Longest Rule, Not First Rule
- Per-Host Request Rate Limits and the Cost of a Fixed Delay
- URL Normalization as a Set-Equality Problem
- Conditional GET and the Revalidation Economy
- Estimating Change Rates from Interval-Censored Fetches
Index updates and query latency.
- The Document Add Path into an Inverted Index
- Segment-Based Indexing and the Merge Operation
- Deletion via Tombstones and the Delete Bitvector
- Near-Real-Time Refresh and the Commit Boundary
- Query-Time Cost and the Conjunctive Fast Path
- Query and Filter Caches in a Search Engine
Search UX
10Result presentation and interaction.
- Generating Search Result Snippets with Query-Biased Summaries
- Relevance Feedback Through Query Expansion and Term Reweighting
- Faceted Navigation and the Counts on Each Filter
- Spelling Correction and the Did-You-Mean Suggestion
- Query Autocomplete and the Prefix Trie Behind It
- Pagination, Ranking Stability, and the Shuffling Result Set
All Information Retrieval articles
- Oct 11, 2026 Bit Packing and Frame-of-Reference Blocks How choosing a bit width per block of postings beats byte-aligned codes, what the block maximum costs in wasted bits, and why patched exceptions sometimes beat both.
- Oct 6, 2026 Synonym Expansion at Index Time Versus Query Time Why expanding a thesaurus into the index and expanding it into the query produce the same postings but different costs, and how each choice changes scoring and phrase matching.
- Sep 23, 2026 Elias-Fano Encoding for Monotone Sequences How splitting each integer into a high and low part stores a sorted sequence at near-optimal space while keeping random access, and what a select operation costs.
- Sep 11, 2026 Pooling Depth and the Unjudged Tail How pooled judgments are built from system submissions, why bias grows with pool depth and novelty, and how to measure the reliability of a partial pool.
- Sep 10, 2026 Searcher Snapshots and Lock-Free Concurrent Readers How a search engine publishes a new index version to running queries without pausing them, using copy-on-write searcher snapshots and reference counting.
- Aug 18, 2026 URL Normalization as a Set-Equality Problem Which transformations are safe when deciding two URLs identify the same resource, why scheme and host case rules differ from path rules, and how over-normalization merges distinct pages.
- Aug 10, 2026 Query and Filter Caches in a Search Engine Which parts of a query can be cached without corrupting results, how a filter cache keys on the filter itself rather than the query, and why invalidation is the hard part.
- Aug 6, 2026 Token Positions and Offsets in Analyzer Output How a token stream records where each term occurred, why position increments encode adjacency for phrase and proximity queries, and how offsets survive synonym expansion.
- May 17, 2026 Query-Time Cost and the Conjunctive Fast Path What dominates the latency of a term-based query, how a conjunction of rare and common terms is evaluated, and where the unavoidable costs of scoring live.
- Apr 27, 2026 Score Calibration for Blended Ranking Why a ranker optimized for ordering produces scores with no absolute meaning, how to fit a monotone calibration, and what breaks when the query mix shifts.
- Apr 20, 2026 Stop-Word Removal and the Phrase Queries It Breaks Why dropping the most frequent terms from the index saves little space today, how position gaps preserve phrase structure, and which queries silently lose meaning.
- Apr 11, 2026 Diversity-Aware Ranking and Subtopics How ambiguous queries are covered by selecting results that span subtopics, what maximal marginal relevance maximizes, and why diversity costs relevance.
- Apr 6, 2026 Static Rank Priors and Their Effect on Ranking How document-level signals that ignore the query accumulate into a prior, how that prior is combined with query-dependent scores, and why it can dominate a weak match.
- Mar 24, 2026 Distance Concentration and Why Tree Indexes Collapse in High Dimensions Why bounding-box and ball-tree pruning stops helping as dimensionality grows, what the concentration effect does to nearest-neighbor contrast, and which index families survive it.
- Mar 23, 2026 Near-Real-Time Refresh and the Commit Boundary How an index makes newly added documents searchable within seconds, and the distinction between committing to the filesystem cache and committing to durable storage.
- Feb 26, 2026 Crawl Order as Resource Allocation Under a Fixed Budget How to decide which of the ten million known URLs to fetch first, why importance and freshness compete for the same request, and what ordering rules can and cannot promise.
- Feb 6, 2026 Coarse Quantization with Inverted Files and the nprobe Setting How an inverted-file index reduces a full scan to a few cluster scans, what the centroid training step optimizes, and why the nprobe parameter buys recall at a nearly linear price in work.
- Jan 8, 2026 Cosine Similarity and the Geometry of Normalized Embedding Vectors What the cosine score actually measures, why length normalization turns it into a plain dot product, and when that normalization destroys information the model encoded.
- Nov 24, 2025 How Relevance Judgments Are Encoded in a qrels File The four-field judgment record, the legacy iteration column, and why the choice of a binary or graded scale bounds what any later metric can express.
- Nov 19, 2025 Edit-Distance Candidate Generation for Query Spelling Correction How a corrector enumerates candidate terms within a bounded edit distance, why the noisy-channel ranking decides the winner, and when a correction should be suppressed entirely.
- Nov 19, 2025 Reciprocal Rank and Rank-Biased Precision Two measures built from a model of the user rather than from a fixed cutoff, and the patience parameter that decides where they place their weight.
- Nov 17, 2025 Tombstones, Merges, and the Cost of Keeping a Vector Index Current Why deleting from a proximity graph is harder than inserting, how segment-and-merge maintenance bounds the damage, and when a rebuild is cheaper than incremental repair.
- Nov 5, 2025 Front Coding the Term Dictionary How sharing prefixes between adjacent sorted terms compresses the dictionary, what block size has to do with random access, and where the savings actually come from.
- Oct 8, 2025 Judgment Drift and Assessor Agreement Relevance is not a stable property: label sets shift with the assessor, the interface, and the year. How kappa quantifies that disagreement and how to report metrics in light of it.
- Sep 30, 2025 Conditional GET and the Revalidation Economy How ETag and Last-Modified turn a refetch into a question, what 304 responses actually save, and why strong and weak validators answer different questions.
- Sep 19, 2025 Spelling Correction and the Did-You-Mean Suggestion How a search engine decides a query is misspelled, generates candidate corrections, and chooses whether to show them or to run the query as typed.
- Sep 18, 2025 Query-Document Feature Extraction for Learned Ranking How match statistics, proximity windows, and coverage ratios are computed from position lists, and what each feature measures that a bag-of-words score cannot.
- Jul 25, 2025 How Many Topics a Benchmark Needs Why the standard error of a mean metric is dominated by topic-to-topic variance, and how resampling shows when a query set is too small to separate two systems.
- Jun 28, 2025 Query Segmentation into Phrases for Ranked Retrieval Why a query without quotes still contains phrase structure, how segmentation models recover it, and what changes when segments are scored as units instead of loose terms.
- Jun 25, 2025 Generating Search Result Snippets with Query-Biased Summaries How a result list turns a matched document into a two-line fragment that shows the query terms in context, and what the summarizer has to give up to do it in milliseconds.
- Jun 20, 2025 Unicode Word Boundary Rules in Query Tokenizers Why splitting a query on whitespace mis-tokenizes apostrophes, hyphens, and scripts that do not use spaces, and how the Unicode segmentation rules decide each case.
- May 23, 2025 Score Normalization and Rank Fusion Across Retrievers Why scores from different retrievers cannot be compared, how score distributions are normalized, and what reciprocal rank fusion gives up to avoid the problem.
- Apr 22, 2025 Query Autocomplete and the Prefix Trie Behind It How the dropdown under a search box is generated from a prefix match, how the candidates are ranked, and why the index must answer partial-query lookups in single-digit milliseconds.
- Mar 29, 2025 Disk-Resident Graph Indexes and the Memory Bandwidth Wall What changes when the vector index no longer fits in RAM: page-oriented adjacency storage, the cost of a random read per hop, and why compression and caching decide throughput.
- Mar 26, 2025 The URL Frontier: Priority Queues Behind Per-Host Politeness How a crawler satisfies two conflicting orderings at once, using a randomized priority front end and a back end of per-host queues keyed by ready time.
- Feb 3, 2025 Over-Fetch and Exact Reranking in Two-Stage Vector Retrieval Why an approximate candidate list must be wider than the final result set, how to size the over-fetch from the approximation error, and where the rerank stage changes the answer.
- Jan 4, 2025 Pagination, Ranking Stability, and the Shuffling Result Set How a search result list is split into pages without re-ranking the whole set, why stable pagination and live index updates conflict, and what the page offset actually costs.
- Jan 3, 2025 Delta Encoding and Variable-Byte Postings How storing document identifier differences instead of absolute identifiers shrinks a postings list, when the vByte continuation-bit scheme wins over fixed-width storage, and where it stops paying.
- Dec 26, 2024 Postings Lists as Sorted DocID Arrays Why an inverted index keeps document identifiers in ascending order, how that ordering turns conjunction into a merge of cursors, and what the layout costs when lists grow long.
- Dec 15, 2024 Write-Ahead Logging and Crash Recovery in Search How a search index survives a crash without losing recently added documents, by logging mutations to a sequential write-ahead log and replaying it after restart.
- Dec 13, 2024 Adaptive Per-Host Rate Control from Latency and Error Signals How a crawler can raise and lower its request rate per host using response latency and rejection codes, and why the control loop is asymmetric and needs damping.
- Nov 26, 2024 Interleaving as an Online Evaluation Method Team-draft and balanced interleaving merge two rankings into one list, so that clicks on the result can be used to compare systems without relevance judgments.
- Nov 4, 2024 Structured Result Cards and the Schema That Feeds Them How a search interface renders a recipe, event, or product as a structured card rather than a text snippet, and how the structured data gets from a page into the index.
- Oct 9, 2024 HNSW Layer Assignment and the Greedy Descent How a hierarchical proximity graph assigns levels by an exponential draw, why the top layers act as a coarse skip list, and what ef actually bounds during the search.
- Sep 25, 2024 Product Quantization and the Asymmetric Distance Computation How splitting a vector into subspaces and coding each against a small codebook replaces a distance computation with table lookups, and where the reconstruction error shows up in the ranking.
- Aug 21, 2024 Estimating Change Rates from Interval-Censored Fetches Why a page observed to change once in ten fetches does not have a ten percent change rate, and how a Poisson model with a Gamma prior turns sparse observations into a revisit interval.
- Jul 9, 2024 Click Features and Position Bias in Ranked Lists Why a click at rank one means something different from a click at rank nine, how examination models correct for it, and what inverse propensity weighting costs.
- Jun 18, 2024 Result Grouping and Near-Duplicate Collapsing How a result list collapses many near-identical documents into a single entry with a cluster, and how the engine detects the duplicates fast enough to do it on every query.
- Jun 15, 2024 Pseudo-Relevance Feedback and the Rocchio Drift Problem How a system rewrites a query using its own top results, why the Rocchio weights and RM3 interpolation decide the outcome, and how feedback amplifies an initial ranking error.
- Jun 6, 2024 Term Dictionary Lookup Structures for Inverted Indexes How an index maps a query term to its postings offset, why the dictionary is kept in memory in a different layout from the postings, and what each structure trades for lookup speed.
- May 30, 2024 Faceted Navigation and the Counts on Each Filter How a search interface narrows a result set by attribute facets, and where the per-facet document counts come from without a full re-query.
- May 10, 2024 Golomb-Rice Coding for Postings Gaps How splitting a gap into a quotient and remainder against a tuned divisor makes an optimal code for geometric distributions, and how the divisor is chosen without knowing the data.
- May 8, 2024 Per-Host Request Rate Limits and the Cost of a Fixed Delay Why a constant delay between requests is the simplest defensible politeness policy, how concurrency multiplies the rate a server actually sees, and where the model fails.
- Apr 12, 2024 Shard Routing and Replica Consistency in Distributed Search How a search index is split across many machines, how a query fans out to shards and merges their results, and how replicas stay consistent with their primary shards.
- Mar 30, 2024 The Document Add Path into an Inverted Index How an incoming document is tokenized, assigned a document id, and appended to the postings lists it matches, and what the ordering invariants on those lists cost.
- Mar 28, 2024 Detecting Crawler Traps by Budget per Host How infinite URL spaces, calendar generators, and session identifiers make a crawl diverge, and the per-host accounting that stops it without losing real pages.
- Mar 25, 2024 Average Precision and the Mean Over Topics How average precision sums precision at every relevant rank, why the mean over topics is a questionable estimator, and what significance testing reveals about it.
- Mar 5, 2024 Segment-Based Indexing and the Merge Operation Why search indexes are built as many small immutable segments rather than one mutable structure, and how the merge step reclaims dead postings and bounds segment count.
- Feb 16, 2024 Personalization Signals and Click-Based Reranking How a search engine folds a user history and click behaviour into a ranking that was previously query-only, and what the feature vectors and the training targets actually look like.
- Jan 5, 2024 Zero-Result Queries and Automatic Query Relaxation How a search interface responds when no document matches, the sequence of constraint relaxations it can try in order, and how it decides which relaxation to show.
- Dec 29, 2023 Positional Indexes and Phrase Verification Why phrase queries need positions and not just identifiers, how a positional postings list stores them compactly, and the cost of exact phrase matching.
- Dec 12, 2023 Graded Gains and the NDCG Discount How discounted cumulative gain combines a gain mapping and a rank discount, why the logarithmic discount was chosen, and what normalisation does and does not fix.
- Dec 6, 2023 Random Hyperplane LSH and the Arithmetic of Banding How one random plane per bit turns cosine similarity into collision probability, and how band and row counts convert that probability into a practical recall curve with a sharp threshold.
- Nov 23, 2023 Precision and Recall at a Cutoff: Why the Cutoff Changes the Metric How a rank cutoff turns set-based measures into per-prefix measures, why precision at small k is unstable across topics, and what R-precision fixes.
- Nov 15, 2023 LambdaMART and Metric-Driven Gradient Boosting How lambda gradients weight document pairs by their effect on NDCG, why the resulting tree ensemble is trained on pseudo-residuals, and where the heuristic gradient fails.
- Sep 9, 2023 Inferring Query Intent from Click Entropy and Reformulation How the distribution of clicks across results, and the way users rewrite a failing query, distinguish navigational from exploratory intent without any labelled training data.
- Sep 6, 2023 Sub-topic Judgments and Diversity Metrics How aspect-level qrels make novelty and coverage measurable, and what the two knobs in alpha-nDCG and intent-aware measures actually control.
- Sep 2, 2023 Pre-Retrieval Query Performance Prediction How a system estimates, before seeing any judged results, whether a query will be answered well, using score distributions, term specificity, and clarity over the collection language model.
- Aug 15, 2023 Metadata Filtering Inside a Vector Index and the Selectivity Collapse Why post-filtering silently returns fewer results than requested, how pre-filtering degrades graph traversal, and what the filter selectivity threshold does to each index family.
- Aug 8, 2023 Partitioning a Crawl Across Machines Without Duplicate Fetches Why host-based partitioning is the only split that preserves politeness, how a frontier survives reassignment, and what happens when one partition dies.
- Jul 5, 2023 Relevance Feedback Through Query Expansion and Term Reweighting How a search system uses a user marking results as relevant to rewrite the next query, and why the classic Rocchio and BM25 expansion methods diverge in what they trust.
- Mar 3, 2023 Deletion via Tombstones and the Delete Bitvector Why a search index does not remove a document immediately, how a tombstone marks a document dead, and what a query does when it encounters one.
- Feb 23, 2023 Suffix-Stripping Stemmers and the Conflations They Force How the Porter and Snowball algorithms decide that a suffix may be removed, why the result is not a word, and which pairs of distinct concepts collapse together.
- Feb 17, 2023 Merge Scheduling and Latency-Bounded Throttling How a search engine decides which segments to merge and when, and how it keeps merge I/O from spiking query latency while still reclaiming space fast enough.
- Jan 16, 2023 Document-at-a-Time and Term-at-a-Time Evaluation Two ways to turn postings lists into ranked results: accumulate per document, or merge per term. What each order costs in memory, latency, and early termination.
- Dec 27, 2022 Impact-Ordered Postings and Early Termination How reordering postings by score contribution instead of document identifier enables early termination, and what it does to merge semantics, memory, and result quality.
- Jul 25, 2022 BM25 Term Saturation and Document Length Normalization How the BM25 scoring function bounds the reward for repeated query terms, what the k1 and b parameters actually control, and where the model breaks down.
- Apr 25, 2022 Pointwise, Pairwise, and Listwise Training Objectives What each learning-to-rank formulation optimizes, why pairwise ranking losses match the evaluation better than regression, and what listwise gradients buy.
- Jan 30, 2022 Matching Paths in robots.txt: Longest Rule, Not First Rule How a robots.txt file is split into groups, why the winning rule is chosen by octet count rather than by order, and what the protocol requires when the file cannot be fetched.
- Nov 26, 2021 Disjunction-Max Combination for Multi-Field Scoring Why scoring each field separately and summing produces runaway scores, and how max-plus-tiebreaker combination keeps field evidence bounded.