Cue Timing in Captions and the Problem of Describing What Is Not Said

A caption is a text overlay whose lifetime is decided by the audio it transcribes, and that constraint is more binding than it appears. Speech arrives at a rate the speaker chose; the caption has to be on screen long enough to be read at a rate the viewer chose, and those two rates are not the same. Every captioning decision is a negotiation between fidelity to the speech and the time available to read it, and the file format is mostly a set of places to record the outcome of that negotiation.

The cue as a timed box

WebVTT is the format the track element consumes. A file is a sequence of cues, each with an identifier, a start and end timestamp, optional settings, and a payload. Timestamps are hh:mm:ss.mmm with hours optional, and the separator is a period rather than a colon — the parser is strict about this, and a comma in the millisecond position is a syntax error that causes the cue to be skipped rather than approximated.

Cue settings control placement independently of the text: line positions the cue on the vertical axis as a percentage or a line number, position places it horizontally as a percentage, size sets its width as a percentage of the video, and align sets the anchor within that box. The defaults put a cue centred, at the bottom, spanning the full width. region refers to a named region defined elsewhere in the file, which is the mechanism for pinning captions to a fixed area so that they do not jump as cue lengths change.

WEBVTT

REGION
id:lower-third
width:80%
lines:2
regionanchor:50%,100%
viewportanchor:50%,92%

00:00:04.000 --> 00:00:07.200 region:lower-third
The parser rejects a comma in the
millisecond field.

00:00:07.400 --> 00:00:09.000 align:start position:10%
Not a warning. The cue is dropped.

The two cues above also illustrate a spacing decision. A cue that ends at 7.200 and another that begins at 7.400 leaves a 200 ms gap, which reads as a pause and is generally preferable to a hard cut between two sentences. A gap of zero is acceptable between two lines of the same speaker but reads as a flicker when the speaker changes, because the eye registers the blank frame.

Reading rate sets the character budget

The binding constraint is characters per second. Published captioning guidance converges on a band rather than a single number: targets commonly cited for adult audiences fall between roughly 15 and 20 characters per second, with individual publishers fixing their own value and lowering it for children’s content or for material with dense vocabulary. The figure encodes how fast people read continuous prose while also watching, and it is an empirical rule rather than a normative limit.

The budget is computed per cue, and it is unforgiving of long cues. A six-second cue at 16 characters per second holds about 96 characters, which is two lines of roughly 48 characters. A three-second cue holds 48 characters, which is one line. The consequence is that captioning a fast speaker requires either splitting into more cues, shortening the text by removing filler, or accepting a rate above the target. The usual resolution is to edit: remove false starts, repeated words, and discourse markers that carry no information, and preserve the words that do.

Line length has a separate limit. Professional practice caps a caption line at around 32 to 42 characters, and two lines per cue is the maximum, because a third line forces the eye to travel further and the reader loses the connection to the image. A cue with three lines of 40 characters is 120 characters, which at a 7.5 second duration is exactly 16 characters per second — but the third line will be read last and may be cut off by the cue’s end.

Wrapping interacts with this. The break should fall at a phrase boundary rather than at a fixed column, because a line that ends mid-clause forces a re-read. A cue whose first line is a complete clause and whose second line is a subordinate clause reads faster than one split at an arbitrary word.

Describing sound and speaker identity

Captions are not only a transcription of dialogue. Non-speech audio that carries meaning has to be represented: a door slamming, a phone ringing, a musical sting that signals a joke. The convention is to bracket or parenthesise these, in a style distinct from dialogue, so that the reader knows the text is not being spoken.

Speaker identity is the harder problem when more than one person talks. Three strategies exist and they trade off differently. A speaker label in brackets at the start of each cue is unambiguous and costs characters on every cue. A change of cue alignment or position per speaker uses space instead of characters but requires the video to have room and the reader to learn the convention. A change of text color per speaker is the cheapest in characters and the least reliable: it depends on color discrimination and fails under forced-colors substitution and for readers who cannot distinguish the hues, which is the same collapse that redundant encoding addresses elsewhere. Most broadcast practice uses labels or positioning for this reason.

Audio description and the gap problem

Audio description is a second narration track that describes visual information the dialogue does not cover. Its constraint is not reading rate but available silence. The description must fit between the existing audio events, and the existing audio was not written with gaps for it.

Two production models exist. In the extended model, the video is paused or slowed to make room: the description plays, the picture holds, and the programme resumes. This preserves the description’s natural pacing and costs runtime, typically a few percent for dialogue-light material and more for dense visual content. In the standard model, the description is squeezed into the natural gaps, which requires writing to a tight word budget and often means describing only the most load-bearing visual facts.

The word budget is derived the same way as the caption budget. A four-second gap at a narration rate of roughly 150 words per minute, which is 2.5 words per second, holds about ten words. That is enough for “She opens the letter and reads.” It is not enough to describe a costume, a room, and an expression. Prioritisation is therefore structural: the description carries what a listener needs to follow the narrative, and leaves decorative detail out.

The track element’s kind attribute distinguishes captions from descriptions, and the descriptions track is expected to be a separate audio resource rather than text, since its content is narration rather than a text overlay. A text track of kind descriptions exists for the case where the description is itself text, which is useful for search and for readers who want the visual information in writing, but it does not replace the audio track for a listener who cannot see the picture.

Where synchronisation breaks

The most common failure is drift between the caption file and the media. If the captions were authored against one cut and the media was re-encoded with a different start offset, every cue is displaced by the same amount. A constant offset is easy to detect and easy to correct; a variable drift is not, and it appears when the media is played at a non-standard rate, since most players shift cue times by the playback rate but some do not.

Live captioning has the inverse problem: the text arrives after the speech, by a delay that depends on the transcription path, and the cues are necessarily approximate. The lag is a property of the transcription path, not of the format, and the format’s job is to represent whatever timing that path produces without adding its own.

Placement failures are visual rather than temporal. A cue positioned in the lower third overlaps burned-in graphics, lower-third titles, or the speaker’s own mouth region, which is where viewers look. A cue that spans the full width covers a wide shot’s content. The region mechanism exists so that a caption file can declare a stable area once rather than repeating settings on every cue, and a file that omits it will produce captions that shift position as cue lengths change, which is more distracting than a fixed imperfect placement.

Finally, the text itself has to be legible against the picture. Captions are usually rendered with an outline or a background box precisely because the backdrop is uncontrolled, which is the video case of text over imagery. A caption style that relies on a translucent box over a bright scene fails the same way a headline over a photograph does, and for the same reason.