Token Positions and Offsets in Analyzer Output

An inverted index that stores only “term $t$ appears in document $d$” supports single-term queries and nothing else. The moment a user quotes a phrase, or a scorer needs to know whether two terms were adjacent, the index must also record where each term occurred. That second channel of information is carried by two numbers per token occurrence: an ordinal position, which counts tokens in sequence, and a pair of character offsets into the original text, which point back at the source.

Positions and offsets serve different consumers, and conflating them is the source of most phrase-matching bugs. Positions drive matching. Offsets drive highlighting, snippet generation, and any post-hoc verification that a match really corresponds to the text that was indexed.

Positions are increments, not indices

The natural way to store a position is an absolute token index: token 0, token 1, token 2. Analysis pipelines do not do this, and the reason is that positions must survive filters that delete or insert tokens. A stream of absolute indices cannot express insertions without rewriting every subsequent index.

Instead each token carries a position increment, the distance from the previous token. The first token has increment 1 by convention. A filter that removes a stop word sets the next surviving token’s increment to 1 plus the removed token’s increment, so the gap in the sequence is preserved rather than closed. A filter that emits two terms for one input (a synonym, a stemming variant) emits them at the same effective position with the second carrying increment 0.

The stream is therefore a run-length encoding of positions, and any consumer can reconstruct absolute positions by accumulating increments. A phrase query "new york pizza" compiled against an index where the was removed still requires the analyzer to have recorded that new and york were adjacent with increments 1 and 1 respectively, and that whatever followed carried the correct accumulated offset.

The distinction matters concretely when a stop filter is applied on the query side but not the index side, or the reverse. If the index recorded new york the pizza as positions 0,1,2,3 and the query is analyzed with stop removal to new(1) york(1) pizza(1), the phrase matches a three-term adjacency that does not exist in the document. A proximity check will reject it if the query carries a slop constraint; an exact phrase will produce a false positive. Either both sides use the same stop-word set, or the position gap must be preserved by the filter on both sides.

Multi-valued positions and the slop parameter

A token at position $p$ with a zero increment means two terms claim the same slot. Stemmers do not normally do this; synonym filters and word-decomposition filters do. Decomposing Wi-Fi into wifi and wi fi produces an alternative representation for the same source span, and phrase slop is computed against position differences, so alternatives at one position are treated interchangeably.

Slop is measured in positions, not tokens. A phrase query with slop 2 permits the terms to appear in order with up to two extra positions between them, or, under the transposition-aware interpretation used by some implementations, permits reordering at a cost. The exact semantics differ: some engines count a swap as a cost of 2, others as 1. This is not a detail that can be guessed from the query syntax; it is a property of the phrase scorer, and inconsistent understanding of it shows up as near-miss phrase results that no one can explain.

Storing positions multiplies index size. For each posting, the position list adds one integer per occurrence, and those integers are usually delta-encoded and variable-byte packed, so a term occurring frequently costs roughly one byte per occurrence on top of its document identifier entry. Once a term occurs more than a few times per document, the position bytes exceed the docid bytes for that term, which is why position data is the largest single component of many positional indexes. That ratio is the trade: phrases and proximity against space.

Offsets are half-open character ranges

The offset pair is a start and an end, both measured in code units of the original string, conventionally half-open: [start, end) covers characters start through end - 1. Half-open ranges concatenate cleanly and empty ranges are expressible, which is why slicing APIs use them.

The unit is a trap. In Java and JavaScript the natural unit is UTF-16 code units; in Python 3 and Rust it is Unicode scalar values or bytes. A single emoji outside the Basic Multilingual Plane occupies two UTF-16 code units but one scalar value. A tokenizer that reports scalar-value offsets into a string that the highlighter then slices by UTF-16 index will clip the last character of some tokens or, worse, split a surrogate pair and produce an invalid string. The offsets are correct with respect to the analysis pipeline and wrong with respect to the consumer, and nothing in the token stream declares its unit.

The second trap is that offsets must refer to the original input, not to intermediate normalized text. Lowercasing can change length, as case folding ß to ss does; NFKC changes length when it expands fi to fi or composes a base and a combining mark into one code point. If normalization happens before offset assignment, the reported offsets index a string that no longer exists. The correction is either to normalize with a character mapping table that tracks provenance, or to compute offsets on the original and let the filters carry them forward unchanged.

Carrying offsets through filters

Every filter in the chain is a transformation of a stream, and it must decide what to do with the offset pair when it changes the term text. The consistent rule is that an offset describes the source span responsible for the token, so a synonym emitted for a source token inherits that token’s start and end, while a token assembled from several source tokens (a shingle, a concatenation of new and york) takes the union of their ranges.

class Token:
    __slots__ = ("term", "start", "end", "pos_inc")
    def __init__(self, term, start, end, pos_inc=1):
        self.term, self.start, self.end, self.pos_inc = term, start, end, pos_inc

def add_synonyms(tokens, thesaurus):
    out = []
    for t in tokens:
        out.append(Token(t.term, t.start, t.end, t.pos_inc))
        for syn in thesaurus.get(t.term, ()):
            # same source span, same position: increment 0
            out.append(Token(syn, t.start, t.end, 0))
    return out

With that stream, a highlighter can render every matched synonym over the same source characters, which is what a reader expects. If the synonym filter instead assigned zero-width offsets or invented ranges, highlighting a synonym match would underline nothing or underline the wrong span.

Term-centric filters such as a stemmer preserve offsets trivially because they replace text without changing span. Token-deleting filters preserve them by not emitting. The only genuinely awkward operations are ones that merge across a gap, and those need an explicit rule rather than an inherited default.

Where the position channel becomes the bottleneck

Positions and offsets are optional per field. A field used only for ranking, never for phrases, can be indexed without them and saves the position bytes entirely. A field used for display can store offsets only. Splitting a schema this way is the standard way to keep an index small without giving up phrase support where it is needed.

The cost of getting it wrong is asymmetric. Missing positions are detectable at query time, as a phrase query that errors or silently degrades to an AND. Wrong increments are not detectable: the query runs, returns results, and some of them are wrong in a way that only shows up when someone reads the returned documents and notices the words were three sentences apart.

When the phrase machinery does fail on a legitimate phrase, the first thing to inspect is the token stream dumped for both the document and the query. Boundary decisions from Unicode word segmentation determine token count, stop-word dropping determines increments, and either mismatch is visible in two lines of dump output. The next layer up, synonym expansion at query time, is where zero increments are introduced and where phrase queries quietly become more permissive than the syntax suggests.