Ranking Documentation Search When the Query Is an Error String

A general web search engine and a documentation search box receive different queries and should not be scored the same way. The web query is short, ambiguous, and exploratory; the documentation query is often a verbatim string copied out of a stack trace, a symbol name with punctuation in it, or a page title the reader half remembers. The reader is not browsing, they are locating, and a result list that puts the right page third is a failure rather than a partial success.

The mechanics of a search index are the same in both cases, which is why the difference is easy to miss. A crawler or build step produces documents; an analyzer turns each into tokens; an inverted index maps each token to the documents containing it, with positions and counts; a scorer ranks the candidates for a query. Every point in that pipeline where documentation text differs from prose is a point where a default tuned for prose misbehaves.

The analyzer decides what a document contains

Before any ranking, text is split into terms and normalised. The usual steps are case folding, word segmentation, optional stopword removal, and stemming. Each is a decision with consequences, and the consequences are more visible on technical text than on natural language.

Case folding is close to mandatory, but it destroys one distinction that matters: an error code written in upper case is a different kind of token from the same letters in a sentence. Keeping both the folded term and a case-preserving term, or detecting the code pattern at query time and issuing an exact-match clause, preserves the signal.

Segmentation is where identifiers break. A prose analyzer treats getUserName as one word, because it contains no spaces and no punctuation. A reader searching for user name then misses the function, and a reader searching for the function as written hits a term that appears in exactly one place in the corpus, which is not necessarily the place that documents it. The fix is to emit both forms: the original token and its parts, split at underscores, dots, and lower-to-upper case boundaries.

import re

def tokens(text):
    for raw in re.findall(r'[A-Za-z0-9_]+', text):
        yield raw.lower()
        for part in re.split(r'_|(?<=[a-z0-9])(?=[A-Z])', raw):
            if part and part.lower() != raw.lower():
                yield part.lower()

Stopword removal is the step that most often goes wrong on technical corpora. A list built for English prose contains in, for, to, no, not, and off, and every one of those appears in identifiers, option names, and error messages. Removing them makes log in unfindable and makes not unsearchable, which is a serious loss in a corpus where a negation changes the meaning of an instruction. The standard mitigation is to apply the stopword list per field: prose body yes, code and headings no.

Stemming has a similar profile. It helps when a reader searches mapping for a page about map, and hurts when it conflates terms the corpus treats as distinct, such as indexes and indices when both are used deliberately. It also mangles identifiers that survived tokenisation. A conservative stemmer, or lemmatisation, reduces the damage; a synonym list maintained alongside the terminology rules is often the better investment, and it has the side effect of making terminology discipline pay off twice.

Error strings add one more problem: they contain punctuation, paths, and quoted values that vary between occurrences. A message like connect ECONNREFUSED 127.0.0.1:5432 should be searchable by its stable fragment, and the analyzer has to keep ECONNREFUSED intact while treating the address as low-value. Splitting on every non-alphanumeric character produces 127, 0, 0, 1, and 5432; four of those are noise, and the last is a number a reader might search for on its own.

Scoring: BM25, fields, and what the reader is looking for

The default scorer in most open-source engines is a variant of Okapi BM25. For each query term, it combines three quantities: how rare the term is across the corpus (inverse document frequency), how often it occurs in this document, and how long the document is relative to the average. The saturation is controlled by two parameters, typically k1 near 1.2 and b near 0.75, and the length normalisation is what stops a long page from outranking a short one merely by containing more words.

The formula matters less than the field structure. A documentation page has several regions with very different signal: the title, the headings, the body, and code blocks. A term in the title is a stronger claim about the page’s subject than the same term in a paragraph. Field-aware scoring computes a per-field term frequency, normalises each field by its own length, and combines them with weights.

Those weights encode editorial judgement and should be chosen by looking at failures. A page whose title matches the query exactly but ranks below a page that mentions the term in a footnote is a weighting bug. A symbol’s own reference page losing to a tutorial that mentions the symbol once is the same bug in a different place. The weights are also the cheapest lever available: they can be changed without re-analysing the corpus if the index stores per-field frequencies.

Headings deserve separate treatment rather than being folded into body text, because a documentation heading is close to a semantic label. ## Retry policy and backoff is a statement about what the section contains, and a query for backoff should reach it even if the body never repeats the word.

Four intents, and why one ranking cannot serve them

Documentation queries cluster into four kinds, distinguishable from surface features with reasonable accuracy.

Navigational queries name a page the reader already knows exists: a product name, a guide title. These are best served by a strong title match and by a top result that is the canonical page rather than a changelog entry mentioning it.

Symbol lookups are a single identifier, often with punctuation: --dry-run, os.path.join, Retry-After. These want exact and phrase matching ahead of fuzzy matching, and they want the reference entry above everything else.

Troubleshooting queries paste an error string or a sentence from a message. These often match nothing verbatim, because messages are templated and the reader’s copy has interpolated values. They need the stable fragment to match, and they need results to include the page that documents the error code, which is why a registry with one entry per code is also a search asset.

Conceptual queries are the ones prose ranking handles well: “how does the scheduler pick a worker”. These benefit from body text, from synonyms, and from letting a longer explanatory page win.

A single scorer with fixed weights serves the conceptual case and underserves the other three. The practical arrangement is a small classifier over query features — presence of punctuation, presence of an exact identifier pattern, length in tokens, presence of a quoted phrase — that selects a boost profile or an exact-match clause. The classifier does not need to be accurate in the abstract; it needs to route the obvious cases, and the fallback is the general profile.

Version duplication is a ranking problem disguised as an indexing one

Documentation sites that keep several releases online index the same page several times. A query for a configuration key then returns ten near-identical results, one per version, and the reader has to work out which one applies. The ranking is not wrong; the corpus contains ten documents that match equally well.

Three mitigations exist and they compose. Index the version as a filterable field and default the filter to the current stable release, with an explicit control to widen it. Collapse results by canonical page identity so that one result represents all versions, with the version shown as a property of the result. And weight the current release above older ones, so that a version-agnostic query surfaces the maintained documentation first.

The first is the only one that changes what the reader sees by default, and it has a failure mode worth stating: a reader on an older release searches for a symbol that exists only in their version, the default filter hides it, and the result set is empty. A search that finds nothing is a worse outcome than a search that returns the wrong version, so the filter should widen automatically when the filtered result set is empty.

Judging relevance without inventing numbers

Search quality cannot be assessed by looking at one query. The usable method is a judgment set: a list of queries, each with the pages that a person who knows the documentation considers correct, graded if possible into “answers the question” and “related”. With a few dozen such queries, two changes can be compared by a rank-sensitive metric such as mean reciprocal rank, which rewards putting the first correct result early, or normalised discounted cumulative gain, which handles graded relevance across the whole list.

The judgment set is also the only defence against tuning to the last complaint. A change that fixes one query and breaks four others is invisible without the set, and the four broken ones are the ones nobody reports.

Query logs supply the other half: queries with no clicks, queries followed immediately by a reformulation, and queries that return zero results. The last is the most actionable, because a zero-result query is a term the corpus does not contain, which is either a synonym gap or a missing page. Both are content findings rather than ranking findings, and they are the cheapest kind of documentation defect to fix once they are visible.