Tag
text-analysis
- Aug 6, 2026 Token Positions and Offsets in Analyzer Output How a token stream records where each term occurred, why position increments encode adjacency for phrase and proximity queries, and how offsets survive synonym expansion. information retrieval
- Jun 20, 2025 Unicode Word Boundary Rules in Query Tokenizers Why splitting a query on whitespace mis-tokenizes apostrophes, hyphens, and scripts that do not use spaces, and how the Unicode segmentation rules decide each case. information retrieval