Unicode Word Boundary Rules in Query Tokenizers

A query arrives as a string. Before any index can be consulted, that string has to become a sequence of terms, and the boundaries between those terms determine what the index is even able to match. Splitting on whitespace is the obvious first attempt, and it is wrong for a large fraction of real queries: don't keeps its apostrophe attached, e-mail and e mail become different queries, 3.14 may or may not survive as a unit, and a query in Chinese arrives without a single space in it.

The Unicode Consortium specifies boundary behaviour in a separate annex from the character tables, because boundaries are a property of adjacent character pairs rather than of characters individually. The rules there are short enough to read in one sitting and have consequences that reach all the way into phrase matching.

Boundaries are decided by pairs, not by delimiters

A tokenizer that scans for delimiter characters still has to decide, per occurrence, whether the delimiter delimits, and that decision needs the surrounding context. An apostrophe inside don't joins; an apostrophe in 'quoted text' separates. Rather than special-case each punctuation mark, UAX #29 assigns each code point one of a small set of word-break properties and then applies an ordered list of rules to every adjacent pair. The properties that matter are ALetter, Numeric, Katakana, ExtendNumLet, MidLetter, MidNum, MidNumLet, plus the ignorable Extend, Format, and ZWJ.

The rules are evaluated in numbered order and the first match decides; the final rule, numbered WB999, is the catch-all “break between any two characters not covered above”. Ordering is what makes the table work. One early rule states that Extend and Format characters attach to whatever precedes them, which is how a combining accent or a zero-width joiner does not create a boundary. U+0027 APOSTROPHE carries the Single_Quote property and U+2019 RIGHT SINGLE QUOTATION MARK carries MidNumLet; both belong to the group of quote characters that the rule joining a letter to a following quote consults, and that rule only fires when a letter follows. So can't and can’t each hold together, while students' ends with a boundary before the apostrophe and a trailing apostrophe becomes a segment of its own.

This is why a character-level property lookup is not enough on its own: the same apostrophe joins in one position and separates in another, and only the neighbour context distinguishes them.

The cases that actually show up in logs

Hyphenated compounds are the most consequential. U+002D HYPHEN-MINUS does not have a joining property in the word-break table, so e-mail becomes two words, e, mail, while e mail becomes two words as well. The two queries converge, which is usually the desired outcome, but only if the document side tokenizes the same way. A tokenizer that keeps e-mail whole on the index side and splits it on the query side produces a query that can never match.

Numeric punctuation is the second. The period and comma carry MidNum-family properties, so 1,000 and 3.14 stay single segments when digits sit on both sides. Acronyms such as U.S. behave the other way: the periods sit between letters, so the rule joins them and the whole string becomes one token. A query for U.S. will therefore not match a document containing US unless some normalization step folds them together, and vice versa.

The third case is punctuation-adjacent tokens that carry meaning as units. Programming queries like C++ and C# are routinely split into C, +, +. ExtendNumLet covers underscore, which is why user_id survives as one token and user-id does not. There is no rule that can recover C++ from first principles, because the language name and the arithmetic operator are the same character sequence; analyzers handle it with a hand-maintained exception list or by preserving symbol runs attached to a preceding letter.

Scripts that do not put spaces between words

Chinese and Japanese text has no inter-word spaces, so no character-delimiter approach can work. UAX #29 does not solve this: Han characters have no joining property, and the catch-all rule places a boundary between every adjacent pair. Segmenting 北京大学 under the word rules yields four single-character segments, which is a defensible decomposition but not a word segmentation.

Two strategies are in common use. Dictionary-based segmentation applies a maximum-match or dynamic-programming model over the Han run, using a lexicon plus character-frequency statistics, and emits words. Bigram indexing takes the other route: it emits every overlapping pair of characters at index time and rewrites the query into the same overlapping pairs, with the pairs combined conjunctively and their positions preserved so that adjacency can still be enforced. Bigrams need no lexicon, tolerate unknown words, and cost roughly twice the postings of an idealized word segmentation, since an $n$-character run yields $n-1$ bigrams. The trade between the two is the same one that query segmentation makes for space-delimited languages, applied at the character level.

Thai, Lao, and Khmer are in the same position for a different reason: they have no spaces and no widely agreed word boundaries, so analyzers ship dictionaries. Japanese adds three scripts in one sentence, and the same maximum-match approach applies with the additional problem that katakana runs are a distinct word-break property, joining across the run rather than per character.

Normalize before cutting, and pick one form

Segmentation operates on code points, so the text must be normalized first or identical-looking queries produce different token streams. NFKC folds compatibility variants: fullwidth Latin letters such as U+FF21, the fullwidth A, become A; the ligature fi becomes fi; superscript digits become plain digits. NFC instead composes canonical sequences, which keeps é as one code point but does not fold width or ligatures. An index built with one form and queried with the other silently loses matches.

Case handling has a similar seam. Lowercasing İ and case-folding it are not the same operation, and the mapping of ß under full case folding is ss, which changes the length of the string and therefore the offset arithmetic used for highlighting. Turkish locales define a different lowercase for I, so a locale-independent fold has to be chosen deliberately rather than inherited from the platform default.

A minimal implementation and where it stops

The following approximates the letter-adjacent cases without claiming conformance. It demonstrates the shape of the problem: each decision needs both neighbours.

import re, unicodedata

APOSTROPHES = {"\u0027", "\u2019"}          # U+0027, U+2019

def split_terms(text: str) -> list[str]:
    text = unicodedata.normalize("NFKC", text).casefold()
    terms, buf = [], []
    for i, ch in enumerate(text):
        prev = text[i - 1] if i else ""
        nxt = text[i + 1] if i + 1 < len(text) else ""
        if ch in APOSTROPHES and prev.isalpha() and nxt.isalpha():
            buf.append(ch)                       # can't, can’t
        elif ch == "-" and prev.isalpha() and nxt.isalpha():
            terms.append("".join(buf)); buf = []  # e-mail -> e, mail
        elif ch.isspace() or unicodedata.category(ch).startswith("P"):
            if buf: terms.append("".join(buf)); buf = []
        else:
            buf.append(ch)
    if buf: terms.append("".join(buf))
    return terms

Run it on C++ and the trailing plus signs vanish as punctuation; run it on don't and the token survives. Both outcomes were decided by three lines of context logic, not by the character class alone, which is the point the full rule table generalizes.

A conformant implementation uses a boundary iterator driven by the property table, which is what ICU’s word-break iterator provides and what most search libraries delegate to. Writing a correct table from scratch is possible but the table changes with each Unicode release as properties are added for newly encoded scripts.

What the boundaries cost downstream

Every boundary that is inserted becomes a position increment in the token stream. A phrase query is a set of terms with consecutive positions, so a tokenizer that drops punctuation without recording a gap will happily match e mail against email; one that records a one-position gap for the dropped hyphen will not. The choices made here are inherited by token positions and offsets and cannot be repaired later.

The failure mode to watch for is asymmetry. A boundary rule applied at index time but not at query time, or a normalization form applied in only one of the two paths, produces a system that returns nothing for queries that ought to work and gives no signal about why. Running the same analyzer over a sample of indexed text and over a sample of logged queries, then diffing the resulting vocabularies, surfaces most of these mismatches in a few minutes of inspection.