Text-to-Speech Synthesis Under the Hood

The speech engine is the component the screen reader calls last, and it is the one whose latency the user feels most directly. It takes a document of marked-up text and produces an audio stream. Between those two ends sit three distinct problems, each with its own algorithms: figuring out what the text says, turning words into sounds, and deciding the pitch and timing. These are usually described as text normalization, phonemic conversion, and prosody, and a screen reader’s speech quality is the product of all three.

Text normalization decides what to say

The input is rarely words. It is a mix of numbers, dates, abbreviations, currency amounts, and symbols, and each has several possible readings. The token “1/2” could be “one half”, “January second”, or “one of two”. “Dr” could be “doctor” or “drive”. The normalization stage resolves these into a sequence of pronounceable words, and it does so using context that is not available to a simple dictionary lookup.

The hard cases are systematic. A number read in isolation uses one reading, and the same number as a date uses another, and the system distinguishes them by the surrounding tokens and by punctuation. Ordinals, Roman numerals, and unit suffixes each add a rule. Abbreviations are the most context-dependent: a token’s expansion depends on whether it is followed by a period, a name, or nothing, and a wrong guess produces a word that sounds like the symbol rather than the meaning. The normalization stage is where a screen reader’s “verbosity” setting operates, because whether a user hears “one thousand” or “one zero zero zero” for “1000” is a normalization choice, not a synthesis one.

Normalization also decides what not to say. Some punctuation produces a pause, some is silent, and some, like a quotation mark, produces nothing at all but changes how a later symbol is read. The stage outputs text with embedded pronunciation and pause directives, which is exactly the shape SSML provides for representing the result.

Phonemic conversion maps words to sounds

The normalized words must become a sequence of phonemes, the discrete sound units of the language. There are two classic ways to do this, and real engines combine them.

The first is a pronunciation dictionary. Each word in the lexicon has a stored phonemic transcription, written in a notation derived from the International Phonetic Alphabet. The engine looks up the word and emits the transcription. This is exact for every word in the dictionary but silent for every word outside it, and the dictionary can never be complete because languages coin and borrow words continuously.

The second is grapheme-to-phoneme rules, often called letter-to-sound rules. These map spelling patterns to sounds heuristically, handling words the dictionary lacks. Rule-based systems succeed on regular languages and fail on irregular ones; English, with its famously inconsistent spelling, is the canonical stress test, and a rule system alone produces mispronunciations on common words. Real engines run the dictionary first and fall back to rules, so the failure mode is predictable: a word the dictionary does not know gets the rule-based guess, which is right for most new words and wrong in the familiar irregular cases.

The phoneme stream is where accent and voice identity enter. The same phoneme sequence can be realized differently by different voices, and the mapping from an abstract phoneme to an actual sound is the voice’s own business. This is why a screen reader can offer multiple voices that share the same pronunciation engine: the dictionary and rules are common, and the acoustic realization differs.

Prosody sets pitch, timing, and pauses

Prosody is what makes the output sound like speech rather than a list of words. It has three main controls. Pitch, or fundamental frequency, rises and falls across a phrase; the fall at the end of a declarative sentence and the rise at a question are the two most recognizable patterns. Duration stretches and compresses syllables to mark stress. Pauses segment the stream into phrases, and their placement is the difference between a sentence read correctly and one read as a run-on.

SSML exposes these controls directly. A document can set the rate, the pitch contour, and the pause lengths, and can mark specific spans for emphasis or for a different voice. This is the mechanism by which the screen reader communicates its own decisions to the engine: the reader has already decided, during the announcement pipeline, where a clause ends and where the emphasis falls, and it encodes those decisions as SSML rather than leaving them to the engine’s defaults.

The prosody stage is also where latency is traded for quality. A cheaper engine uses fixed, precomputed prosody and produces audio quickly. A more elaborate engine computes prosody from a syntactic parse, which takes longer but sounds more natural. The screen reader, which must interrupt speech on every focus change, tends to prefer engines that start producing audio quickly, because a high-quality engine that takes half a second to begin is useless when the user has already moved on.

The latency chain and interruption

The reader’s need to interrupt speech shapes the engine’s whole architecture. The engine cannot buffer an entire sentence before emitting anything, because the user may cancel it halfway. It must be streaming, producing a prefix of audio as soon as the first phonemes are ready. The time from SSML input to the first audio sample is the metric the reader cares about most, and it is bounded below by the normalization and dictionary lookups, which cannot run before the input arrives.

Some readers reduce this by pre-synthesizing. Common announcements, like the role names “button” and “heading”, are short and frequent enough that the reader can synthesize them once and cache the audio, then splice the cached segments around the variable part. This is why a screen reader can announce a name quickly even when a cold start of the engine would be slow: the fixed scaffolding is already rendered, and only the novel part costs time. The cache is a direct optimization of the interruption constraint, and it trades memory for latency in the usual way.

Failure modes

Normalization failure produces the wrong word: a date read as a number, an abbreviation expanded into the wrong part of speech. Phonemic failure produces a mispronunciation, concentrated on the words the dictionary lacks. Prosody failure produces speech that is comprehensible but hard to follow, because the phrase boundaries are wrong. And latency failure produces the most user-visible symptom of all: speech that trails the user’s actions, which the interruption mechanism can mask but never fully hide.

The engine is the last stage in a chain that began with an event on the accessibility tree. Each earlier stage produced a string or a markup decision, and the engine is where those decisions finally become sound. It is tempting to treat it as a black box, but its three internal stages each have failure modes that are distinguishable from the ones upstream, and knowing which stage failed is what tells a developer whether the fix belongs in the markup or in the dictionary.