Diagnostic Spans and the Encoding Problem

A finding is only useful if it points at the code that caused it. That sounds trivial until the source contains a non-ASCII identifier, a tab, or a CRLF line ending. Then the analyzer’s internal offset, the editor’s column, and the terminal’s idea of a character position can all disagree, and the squiggle lands three characters to the left of the problem.

A position is a pair of numbers with a unit

There is no unit-free position in a text file. The candidates are byte offset, Unicode code point index, UTF-16 code unit index, and grapheme cluster index. They coincide only for ASCII text without tabs, which is why bugs in this area survive testing on English identifiers.

Most compilers and analyzers store byte offsets into the source as read from disk. A byte offset is compact, supports slicing without decoding, and is stable under any transformation that preserves the byte sequence. It is also the natural unit for a parser written in C, C++, Rust, or Go, all of which index strings by byte. Rust’s compiler, for example, represents spans as byte ranges into a UTF-8 source file.

Editors are the complication. The Language Server Protocol specifies that positions are expressed as a line number plus a character offset, and the default unit for that offset is UTF-16 code units, because the protocol grew out of editors whose string APIs used UTF-16. A character outside the Basic Multilingual Plane, such as an emoji or a rarely used CJK ideograph, occupies two UTF-16 code units and four UTF-8 bytes. An analyzer that reports a byte column to a client expecting UTF-16 columns will place the diagnostic too far to the right on exactly those lines.

The protocol has since added negotiation for UTF-8 and UTF-32 positions, but the default remains UTF-16, so an implementation that does not negotiate must convert. The conversion is not a simple division: it requires decoding the line up to the byte offset and counting code units.

Ranges, line breaks, and the zero-width case

A span is usually a half-open range: start offset inclusive, end offset exclusive. Half-open ranges compose without the off-by-one errors of inclusive ends, and the length is just the difference. The convention has to be documented because a diagnostic that underlines one character too many is a visible defect.

Line breaks are the next trap. A line index maps offsets to line numbers, and it must agree with the editor about what a line is. \n, \r\n, and a lone \r have all been line terminators in real files. If the analyzer counts only \n, a file with \r\n still gets the right line count, but the column of the last character on each line may include the \r. A byte-order mark at the start of the file shifts every offset on line zero if it is not stripped or accounted for.

Zero-width spans are legitimate and necessary. An insertion point, such as a missing import that should be added, has start equal to end. Many diagnostic formats allow it; some renderers drop it; a few crash. The analyzer should carry an explicit flag or a minimum width for rendering rather than silently widening the span, because widening changes which text the user sees highlighted.

Tabs are a separate axis from encoding. A tab is one code point and one byte, but it occupies a variable number of columns depending on the tab stop. If the analyzer reports columns in code points, it is consistent; if it reports display columns, it must apply the tab width the editor uses, and that width is a client setting. The safe choice is to report offsets in a defined unit and let the client render, rather than to guess at display width.

def build_line_index(text: str):
    # Offsets are Python string indices (code points) here; a byte-based
    # implementation would scan the encoded buffer instead.
    starts = [0]
    for i, ch in enumerate(text):
        if ch == "\n":
            starts.append(i + 1)
    return starts

def utf16_column(line: str, cp_index: int) -> int:
    return len(line[:cp_index].encode("utf-16-le")) // 2

The example makes the unit explicit in the function name. That is the habit that prevents the class of bug: every conversion function says which unit it consumes and which it produces, and no span crosses a boundary without a named conversion.

Spans survive transformations badly

An analyzer rarely reports on the file it read. It reports on a tree that has been desugared, on a macro-expanded translation unit, or on a bundle produced by a build step. Each transformation moves the text, and the span must be mapped back.

Macro expansion is the hardest common case. A finding inside an expanded macro may have no single location in the original source; it has an invocation site and a definition site. Clang models this with a chain of expansion locations rather than one span, so a diagnostic can say “in expansion of macro X” and point at both. A tool that stores only the final buffer offset points into a synthetic buffer the user has never seen.

Generators and bundlers have the same problem with a different mechanism. A minified bundle maps positions through a source map back to original files. A linter that runs on generated output without consulting the map produces findings on line 1, column 40,000 of a file nobody edits. The map is also lossy: names are mangled, and some constructs have no original location.

Desugaring inside a single compiler is subtler. If the parser lowers a ??= b into a conditional assignment, a finding on the lowered node must still point at the original operator, not at the synthetic conditional. The usual solution is to attach the original span to every synthesized node at creation time, so the tree carries provenance even where the syntax does not.

Suppression is part of the diagnostic model

A suppression comment is a directive in the source that tells the analyzer not to report a particular finding. It is a diagnostic-adjacent construct with its own grammar: a marker, a rule identifier or a wildcard, sometimes a justification, and a scope (this line, the next line, the whole file). Because it lives in comments, it is only visible if the lexer kept trivia, as discussed in Why Linters Parse.

Three design questions arise. First, matching: does the directive apply to the line it is on, the following line, or the enclosing declaration? Each choice produces surprises at boundaries, and the answer must be documented. Second, granularity: suppressing a whole file is cheap for the user and expensive for the codebase, because it removes every future finding of that rule in that file, including ones introduced later. Third, validation: an unused suppression is itself a finding in some tools, since it usually means the underlying issue was fixed and the directive is now dead weight.

Suppression counts are a useful health metric, but they should not be treated as a fix. A suppression records that a true positive was accepted, not that the analysis improved. The accumulation of such records is the subject of The Structure of False Positives.

Interchange formats force the ambiguity into the open

When results leave the tool, they are serialized. SARIF, the OASIS standard for static analysis results, represents a location as a physical location with an artifact location and a region, where the region has start line, start column, end line, and end column. The standard defines columns in characters, and implementations differ on whether a character means a code point or a UTF-16 unit. That difference is exactly the ambiguity described above, and it is why a consumer that renders SARIF must know the producer’s convention.

SARIF also carries a partialFingerprints property, which lets a result be tracked across runs even when line numbers shift. The fingerprint is derived from stable content, such as the enclosing function name and a normalized message, rather than from the position. Without it, every insertion above a finding looks like a new finding, and every baseline comparison becomes noise.

The practical rule is to keep one canonical internal representation, convert at the boundary, and test the boundary with text that contains a non-BMP character, a tab, and a CRLF line ending. Those three inputs catch nearly every unit confusion before a user sees a misplaced underline.