Tag
tokenization
- Sep 19, 2026 Inverted Index Construction and Query Planning for Log Search How tokenized log records become delta-encoded posting lists, how a time-partitioned index prunes work, and when a scan beats an index lookup. observability
- Aug 6, 2026 Token Positions and Offsets in Analyzer Output How a token stream records where each term occurred, why position increments encode adjacency for phrase and proximity queries, and how offsets survive synonym expansion. information retrieval
- Nov 16, 2025 Maximal Munch and the Boundary Between Tokens How a lexer decides where one token ends, why the longest-match rule is separate from the grammar, and which token sequences it produces that the source text does not appear to contain. compilers
- Jun 20, 2025 Unicode Word Boundary Rules in Query Tokenizers Why splitting a query on whitespace mis-tokenizes apostrophes, hyphens, and scripts that do not use spaces, and how the Unicode segmentation rules decide each case. information retrieval
- Feb 24, 2025 Ranking Documentation Search When the Query Is an Error String Documentation queries are lookups, not explorations. Identifier tokenisation, field weighting, and version duplication decide whether the entry a reader needs appears at all. software engineering
- Mar 30, 2024 The Document Add Path into an Inverted Index How an incoming document is tokenized, assigned a document id, and appended to the postings lists it matches, and what the ordering invariants on those lists cost. information retrieval
- Feb 1, 2024 Why Perplexity Is Not Comparable Across Tokenizers Perplexity measures average surprise per token, so the vocabulary decides the unit. Converting to bits per byte removes the unit but not every confound. inference systems
- Feb 23, 2023 Suffix-Stripping Stemmers and the Conflations They Force How the Porter and Snowball algorithms decide that a suffix may be removed, why the result is not a word, and which pairs of distinct concepts collapse together. information retrieval