Mapping RTP Clocks to Wall Time for Cross-Stream Synchronization

An RTP stream’s timestamps are relative to a random per-stream offset, at a codec-specific rate. Nothing in a media packet says what wall-clock instant its samples represent. Two streams that arrive on separate ports with unrelated SSRCs and unrelated timestamp bases cannot be aligned by comparing their headers, no matter how carefully the receiver inspects them.

Lip sync is the requirement that forces a solution. Audio and video of the same speaker must be presented within a tolerance that is asymmetric: audio arriving ahead of video is noticed at smaller offsets than audio lagging behind, and the accepted window is roughly tens of milliseconds in one direction and somewhat wider in the other. Within that window the mapping between the two streams’ timelines must be accurate, and it must stay accurate as the session runs.

The correspondence carried in a sender report

A sender report resolves the ambiguity by stating, for one instant, both an NTP timestamp and the RTP timestamp of the media that was being captured at that instant. The NTP field is 64 bits: 32 bits of seconds since 1900 and 32 bits of fractional seconds, giving a resolution of about 233 picoseconds and a range that runs out in 2036.

That single pair of numbers defines an affine map. For a stream with clock rate r, an RTP timestamp t corresponds to the NTP instant:

ntp_time = ntp_from_report + (t - rtp_from_report) / r

with the subtraction performed modulo 2^32 on the RTP side, so that timestamps from before the report instant are handled by the same expression. A receiver that has this map for audio and another for video can convert both to a common NTP timeline and compare.

The resolution of the map is limited by the RTP timestamp’s granularity, not by NTP’s. At 48 kHz, one tick is about 20.8 microseconds; at 90 kHz, about 11.1 microseconds. Both are far finer than the lip sync tolerance, so the mapping is not the limiting factor. The limiting factors are the accuracy of the sender’s own clock and the stability of the offset over time.

Clock offset and why the naive subtraction is wrong

A receiver cannot use the NTP value in a sender report directly as a local time. The sender’s clock and the receiver’s clock differ by an offset that is unknown and, on consumer devices, can be large: unsynchronized hosts commonly differ by seconds, and a device that has just booted may have a clock that is wrong by minutes.

What the receiver needs is not the absolute instant but the relationship between the two streams’ timelines, and that relationship can be established without knowing either clock’s offset. If audio and video sender reports both state their correspondences in the same sender’s clock, then converting both to that clock and comparing yields the correct relative alignment, regardless of how far that clock is from the receiver’s.

The condition is that both streams come from the same sender and the same clock. When they do not, as when audio and video are terminated by different servers, the two NTP bases are different clocks with different offsets, and the comparison is wrong by that difference. Fixing it requires an external reference: RFC 7273 defines signalling that identifies the clock source a stream’s timestamps are referenced to, so that a receiver can know whether two streams share a clock or need independent synchronization. Without such signalling, a receiver has no way to determine this from the packets.

Drift, and the difference between offset and rate

Two clocks that agree at one instant will not agree later. Oscillator tolerances on commodity hardware are measured in parts per million, and a 50 ppm difference accumulates 180 milliseconds per hour. For lip sync, that is far beyond tolerance within a single long session, which is why a mapping established from one sender report is insufficient.

The fix is to keep collecting correspondences and to fit a line rather than a point. Each sender report gives one (NTP, RTP) pair; several over time reveal both the offset and the rate difference between the sender’s clock and the receiver’s. The rate estimate is what allows the receiver to extrapolate between reports, and the offset estimate is what allows it to place a packet on the local timeline.

Fitting needs care because the samples are noisy. The NTP value is exact as stated by the sender, but the receiver’s own timestamp of when it arrived includes network delay and scheduling jitter, which is why the arrival time cannot simply be compared with the stated NTP time. The stated correspondence is between two sender-side clocks, and it is exact with respect to the sender; the uncertainty enters only when the receiver converts to its own clock, and that conversion is what drift estimation is for.

A workable approach discards samples whose implied offset deviates far from the current estimate, on the grounds that they were delayed unusually, and applies a slow filter to the rest. The filter’s time constant is the trade: a long constant yields a stable estimate that is slow to notice a genuine clock step, and a short one tracks steps but is noisy.

Playout scheduling across two streams

Once both streams map to a common timeline, the receiver can schedule them. The scheduling decision is not simply “play audio and video when their mapped instants arrive,” because each stream has its own buffering. Audio may be held 60 ms to absorb jitter; video may be held 100 ms to wait for complete frames. If each is played as soon as its own buffer allows, the relative alignment is destroyed by the difference in buffer depths.

The correct construction is to choose one stream as the reference, compute its playout delay, and then delay the other stream by the amount needed to preserve the mapped offset. In practice the audio stream is the reference because audio glitches are more noticeable than video ones, and because audio’s buffer tends to be smaller and its deadlines tighter. Video is then delayed to match, which usually means adding latency to video rather than removing it from audio.

This is a case where the synchronization requirement and the latency requirement conflict directly. Aligning video to audio may require holding video frames longer than video’s own jitter tolerance demands, adding tens of milliseconds. The alternative, letting video run ahead, breaks lip sync. There is no way to satisfy both; the standard resolution is to accept the added latency on the stream that has slack, which is almost always video.

When the mapping shifts, because the sender’s clock stepped or because a report revealed a corrected offset, the receiver must decide whether to jump or to slew. A jump produces an audible or visible discontinuity; a slew spreads the correction over time, which for audio means resampling by a tiny ratio and for video means adjusting the frame hold. Slewing is preferred for small corrections and jumping for large ones, with the threshold set by what the media can hide.

Failure modes in the mapping

The first is a sender that emits reports too infrequently. RFC 3550 sets the nominal report interval and scales it with session bandwidth and participant count, so a large session yields sparse reports from each sender. A receiver with two reports an hour apart cannot estimate drift; it can only assume the clocks agree, which they will not.

The second is a sender whose NTP clock is not monotonic. A device that adjusts its clock during the session, by a step or by a gradual correction, invalidates the correspondence: the RTP timestamp continues at the media clock rate while the NTP value jumps. A receiver that fits a line across the discontinuity produces a mapping that is wrong for every packet after it. Detecting this requires watching for a residual jump in the fit, which is the same signal as a network delay spike and must be distinguished from it.

The third is a stream that restarts. A new SSRC, or a new random timestamp base with the same SSRC, breaks the mapping entirely, and the receiver must discard its accumulated correspondence and start again. Until a new sender report arrives, the stream cannot be aligned with anything, which means a visible loss of sync for at least one report interval after any restart. Senders that change encoding parameters mid-session without changing the timestamp base avoid this; ones that reinitialize the media clock do not.

The fourth is the NTP era rollover. The 32-bit seconds field wraps in 2036, and implementations that treat the field as an absolute time rather than as a value to be interpreted within an era will misplace timestamps by 136 years. Since only differences matter for synchronization, the wrap is harmless if the arithmetic is done modulo 2^32 on the seconds field, and catastrophic if it is done in floating point without care.

Where the mapping stops being useful

The mapping aligns streams that share a clock. It cannot align streams that do not, and it cannot compensate for a receiver whose own output device has a different latency for audio and video. The last of these is outside RTP entirely: a display pipeline that adds a frame of buffering, or an audio path with a deep hardware buffer, offsets the presentation of one stream relative to the other after the timestamps have done their job. Correcting it requires measuring the output path, which no protocol field provides.

What the mapping does provide is a common reference that every stage can use, and the discipline of keeping that reference explicit is what makes the rest tractable. The individual clock rates that feed into it are covered in why RTP timestamps do not count seconds, and the buffer depths that must be reconciled across streams are covered in sizing and adapting the jitter buffer.