Character Encoding Determination Before the Parser Runs

The tokenizer described in How the HTML Tokenizer and Tree Builder Recover from Malformed Markup consumes characters, but a network response delivers bytes. Something has to choose a decoder before the first < is examined, and that choice cannot be deferred: the bytes of the tag itself depend on it. The HTML parsing algorithm therefore begins with an encoding sniffing step that runs over the raw byte stream and consults, in order, a byte order mark, a user override, metadata supplied by the transport layer, and a scan of the first 1024 bytes.

Getting this wrong produces a specific class of failure that no amount of markup repair can fix. A document served as UTF-8 but declared as windows-1252 decodes each multi-byte sequence into two, three, or four separate characters, and the resulting text is mojibake rather than an error. The parser then builds a perfectly valid tree of the wrong characters.

The precedence order

Four sources dominate the decision, in a fixed order, and the first that yields an encoding wins.

The byte order mark comes first. A stream beginning with EF BB BF is UTF-8, FF FE is UTF-16LE, and FE FF is UTF-16BE. The BOM is consumed and does not appear in the document. Its precedence over every other signal is deliberate: a file that was saved as UTF-8 with a BOM carries evidence about its own encoding that is stronger than a server misconfiguration.

A user override comes next in the specification’s ordering, which is why changing the character encoding in a browser’s page menu wins over both the header and the meta element.

The transport layer follows. For an HTTP response, a charset parameter on the Content-Type header supplies the encoding. The value is treated as a label and matched against the Encoding Standard’s label table, which means iso-8859-1, latin1, and cp1252 all resolve to the same encoding, windows-1252, and utf8 resolves to UTF-8. There is no strictness at this step: an unknown label is ignored rather than rejected.

The document’s own declaration comes last, found by prescanning the first 1024 bytes for a meta element with a charset attribute, or with an http-equiv="content-type" attribute whose content contains a charset parameter. If none of these sources produces an encoding, the implementation’s fallback applies.

The 1024-byte prescan

The prescan exists because the parser cannot tokenize before it knows the encoding, and the declaration is inside the document it is trying to decode. The resolution is a small, separate scanner that operates on bytes and understands just enough HTML to find a meta tag: it looks for <, then meta in any letter case, then attributes, and it skips over comments so that a commented-out meta is not honoured.

The window is exactly 1024 bytes. A declaration that begins after that offset is invisible to the prescan, and the document decodes with whatever the earlier sources provided. In practice the limit only bites on documents with a very large comment or an inline script block ahead of the meta, and the remedy is to move the declaration to the top of the head, before anything that could push it past the boundary.

The prescan is also deliberately loose about syntax. Attribute quoting, whitespace around =, and unquoted values are all handled, because real documents contain all of them. The scanner is not the tokenizer and does not apply the tokenizer’s error rules; it accepts sequences the tokenizer would treat as parse errors, which is why a malformed meta can still influence decoding.

Why a declaration cannot say UTF-16

The encoding named by a meta element has to be one that can encode the ASCII characters used to write the declaration itself. UTF-16 fails that requirement: the bytes of <meta charset="utf-16"> would themselves have to be interpreted as UTF-16 before the declaration could be read, which is circular.

The specification resolves the circularity by treating a meta declaration of UTF-16LE or UTF-16BE as a declaration of UTF-8. A document genuinely encoded as UTF-16 must be identified by a BOM or by the transport layer. This is one of the few places where the declared value is silently overridden rather than ignored, and it is a source of confusion for documents converted to UTF-16 by an editor that also rewrote the meta tag.

The same constraint is why the declaration must be ASCII-compatible in general. An encoding that does not share ASCII’s byte values for the ASCII range cannot be named inside a document that is being decoded with it.

Changing the encoding after decoding started

A meta declaration that the prescan missed is still seen by the tokenizer, and it can trigger the change-the-encoding algorithm. When the encoding it names differs from the one in use, the engine discards the partially built tree and restarts parsing from the beginning of the byte stream with the new decoder.

The cost is a full reparse, and it is why the prescan exists at all: the common case is handled without one. Engines are conservative about the switch. A declaration that conflicts with a BOM is ignored, and a change that would alter the interpretation of bytes already consumed forces a fresh parse of the whole stream rather than a patch to the existing tree.

The specification models the ranking with a confidence flag rather than leaving it to convention. A BOM, a user override, and a transport-layer encoding are all certain; a prescan result is tentative. A meta declaration is only allowed to change the encoding while the confidence is tentative, so a Content-Type header outranks a later meta declaration by rule. The flag becomes certain once the encoding is settled, which is what prevents a second declaration later in the document from starting another cycle.

What happens to bytes that do not decode

Every decoder in the Encoding Standard is total: it maps every byte sequence to a sequence of code points, substituting U+FFFD REPLACEMENT CHARACTER for anything invalid. A truncated multi-byte sequence at the end of a stream, an overlong encoding, or a surrogate half all become one or more replacement characters rather than raising an error.

How many replacement characters appear depends on how the decoder groups the invalid bytes. A single invalid byte produces one U+FFFD; a lead byte followed by a wrong continuation byte may produce one for the maximal subpart that was consumed and another for the byte that broke the sequence. The exact grouping is specified per encoding in the Encoding Standard, and code that counts replacement characters to infer the original bytes is relying on an implementation detail that is specified but not intuitive.

TextDecoder exposes the same decoders to script. By default it substitutes U+FFFD, which matches the parser’s behaviour. Passing fatal: true makes it throw a TypeError on the first invalid sequence instead, which is the appropriate mode for validating input before storing it. The ignoreBOM option controls whether a leading BOM is stripped or preserved as U+FEFF; the default preserves the standard behaviour of stripping it.

const decoder = new TextDecoder('utf-8', { fatal: true });
const bytes = await (await fetch('/data/notes.txt')).arrayBuffer();

try {
  const text = decoder.decode(bytes);
  console.log(text.length, decoder.encoding);
} catch (error) {
  // invalid byte sequence: reject rather than store replacement characters
}

Decoding outside the document parser

Script that fetches text has to make the same decision the parser makes, and the platforms differ in a way that surprises people. Response.text() decodes the body as UTF-8 unconditionally, following the Fetch Standard, and ignores any charset parameter on the response’s Content-Type. A resource served as windows-1252 and read with text() produces replacement characters for every non-ASCII byte.

TextDecoder with an explicit label is the correct tool when the encoding is known, and reading the Content-Type header to extract the label is the correct way to learn it. A resource that is genuinely UTF-8 needs neither step, which is the argument for serving UTF-8 everywhere and declaring it in both the header and the document.

The document itself follows the same logic at a different layer. document.characterSet reports the encoding the parser settled on, and it is a read-only observation of a decision that was made before parsing began. It is not changed by later manipulation, and it does not reflect the encoding of a string, since strings in the DOM are sequences of UTF-16 code units regardless of the source encoding.

Choosing an encoding for new content

The specification recommends UTF-8 as the fallback and every engine supports it, which makes the decision straightforward for new documents: serve UTF-8 in the Content-Type header, declare it in a meta element at the very start of the head, and avoid a BOM unless the file is consumed by tools that require one. The BOM adds three bytes and, in a document assembled by concatenation, can end up in the middle of the stream where it is no longer stripped and becomes a zero-width character in the text.

Legacy encodings remain supported because the web contains documents in them, not because they are a reasonable choice. The label table in the Encoding Standard is the authoritative list of what any given name resolves to, and it is worth checking before assuming that iso-8859-1 and windows-1252 differ, since they do not.