Sizing and Adapting the Jitter Buffer

Every packet that arrives at a receiver is either played or discarded. A jitter buffer is the component that decides, and its entire job is to convert a stream with variable interarrival times into one with constant intervals. The conversion cannot create information. It can only hold packets longer, and holding longer costs latency.

The buffer sits between the transport and the decoder. Packets are pushed into it as they arrive, keyed by sequence number, and the decoder pulls them on a clock. The design question is how far behind the arrival stream the pull clock should run. That offset is the playout delay, and it is the only parameter that matters.

The minimum delay is not the average delay

The intuitive sizing rule, set the delay to the average network delay, fails immediately. If the playout instant for packet n is the average arrival time plus a fixed offset, then roughly half of all packets arrive after their playout instant and must be discarded. The buffer needs enough slack that the late tail of the delay distribution fits inside it, which means sizing against a high percentile rather than a mean.

Suppose delay is a random variable with mean m and standard deviation s. To have a given packet arrive in time, the delay target must exceed the actual delay for that packet. Choosing the target at m + 3s covers most of a normal distribution but says nothing about a heavy-tailed one, which is what network delay usually is: a mixture of a narrow base and occasional excursions when queues build. This is why the interarrival jitter estimate from RTCP, which is a mean absolute deviation of the delay derivative, is a useful signal but not a sufficient one. It describes typical variation and understates the tail.

The practical consequence: a buffer sized from average jitter produces audible or visible glitches whenever a queue builds, and those glitches cluster, because the queue that caused one packet to be late will cause the next several to be late too. A buffer sized from the observed maximum over a recent window handles the cluster but is vulnerable to a single extreme outlier inflating the target for the next several minutes.

Fixed, adaptive, and the case for both

A fixed delay is simple and predictable. The target is set once at session start, typically a few hundred milliseconds for consumer networks, and never changes. Its virtue is that the playout timeline never shifts, so there are no discontinuities in output; its cost is that the delay is either too large for a clean network or too small for a congested one, and there is no feedback to correct it.

An adaptive buffer changes the target during the call. The reason adaptivity is necessary is that network conditions change on a timescale of seconds, while a session lasts minutes. A call that starts on a clean network and encounters congestion twenty seconds in must either increase its delay or accept losses.

Two adaptation strategies exist, and the difference is audible.

The first adjusts the delay continuously by small amounts, resampling or stretching the media to absorb the change. A ten-millisecond increase spread over a second of audio is imperceptible. This is the approach used in audio, where time-scale modification can insert or remove a few milliseconds without changing pitch.

The second adjusts in discrete jumps at packet boundaries, accepting a gap or a skip. This is what video does, because video frames cannot be partially stretched. A delay increase means holding the current frame longer; a decrease means skipping to a later frame. Both are visible if large, so video adaptation moves in small steps and accepts that the effective delay lags the target by a frame or two.

The trade-off in both cases is the same axis: responsiveness of the delay estimate against stability of the playout timeline. A fast-adapting buffer tracks a changing network but produces frequent timeline shifts, each of which is a potential glitch. A slow-adapting buffer produces a smooth timeline but is late to react, and pays for it in losses during the transition.

A workable estimator

The estimator below keeps a windowed maximum of the observed delay and uses a two-speed filter: increases are applied quickly because arriving late is worse than arriving early, and decreases are applied slowly because a decrease that turns out to be premature causes a glitch immediately.

#include <stdint.h>
#include <stddef.h>

#define WINDOW 200   /* packets observed before re-evaluating the target */

struct jb {
    int64_t  delay[WINDOW];   /* arrival - playout-clock, in microseconds */
    size_t   n;               /* number of samples currently in the window */
    size_t   pos;             /* circular write position */
    int64_t  target;          /* current playout delay, microseconds */
};

/* Percentile of the window, by simple selection. Called once per window. */
static int64_t window_percentile(const struct jb *b, int pct)
{
    int64_t tmp[WINDOW];
    size_t  i, j;
    for (i = 0; i < b->n; i++)
        tmp[i] = b->delay[i];
    /* insertion sort: the window is small and this runs rarely */
    for (i = 1; i < b->n; i++) {
        int64_t v = tmp[i];
        for (j = i; j > 0 && tmp[j - 1] > v; j--)
            tmp[j] = tmp[j - 1];
        tmp[j] = v;
    }
    return tmp[(b->n - 1) * (size_t)pct / 100];
}

void jb_update(struct jb *b, int64_t observed_delay_us)
{
    b->delay[b->pos] = observed_delay_us;
    b->pos = (b->pos + 1) % WINDOW;
    if (b->n < WINDOW)
        b->n++;

    if (b->n < WINDOW)
        return;

    int64_t want = window_percentile(b, 95);
    if (want > b->target)
        b->target += (want - b->target) / 2;     /* rise fast */
    else
        b->target -= (b->target - want) / 16;    /* fall slowly */
}

The 95th percentile over a 200-packet window is a defensible default for audio: at 50 packets per second that is four seconds of history, which covers a congestion episode without reacting to a single spike. The asymmetric gains are the important part. A buffer that decreases its delay as eagerly as it increases will oscillate, and each oscillation costs a glitch.

Video needs a different window because packet rates differ by orders of magnitude. At 30 frames per second with a few packets per frame, a 200-packet window covers a second or two, which is too short to distinguish a transient from a regime change. Sizing the window in time rather than in packets keeps the behaviour consistent across formats.

What happens when a packet misses its deadline

Once the playout clock passes a packet’s scheduled instant, the packet is gone. There is no late delivery; the decoder has already produced the output for that interval using whatever concealment the codec offers. For audio, that means repeating the last pitch period with a fade, or generating noise with the right spectral envelope. For video, it means repeating the previous frame or, if the lost data is a reference frame, showing artifacts until the next independently decodable frame arrives.

This is the strongest argument for keeping the buffer slightly larger than the estimate suggests. The cost of an extra 20 ms of latency is imperceptible in a conversation; the cost of a 20 ms gap is a click. The asymmetry of consequences justifies asymmetry in the error: over-provisioning latency by a frame is cheaper than under-provisioning by a frame.

The exception is interactive audio with echo cancellation or with tight conversational turn-taking, where total latency matters. There, the buffer is only one term in a sum that includes codec delay, packetization delay, network delay, and the receiving device’s output latency. Budgeting requires accounting for all of them, and a buffer that takes a large share of a small budget is a poor trade even if it eliminates glitches.

Reordering is not loss until the deadline passes

A packet that arrives out of order is indistinguishable from a packet that is delayed, until enough time passes. The buffer must therefore keep a hole open rather than declaring loss immediately. RFC 4737 formalizes reordering metrics, and the practical rule is that a sequence number gap is provisional: the packet may still arrive, and the buffer’s capacity to wait is exactly its playout delay.

The interaction with the RTCP loss counters is subtle. A packet that is late but arrives before its deadline is counted as received by the RTP layer and as late by the buffer. A packet that misses the deadline is counted as received by the RTP layer, because it did arrive, but as lost by the application. The two loss statistics diverge, and the divergence is the buffer’s contribution to quality: it converts transport loss into delay, and delay into a smaller number of actual gaps. The report fields that carry those counters are described in reading loss and round-trip time from RTCP.

Where the buffer cannot help is with a burst longer than its depth. A one-second outage against a 100 ms buffer loses everything in that second regardless of adaptation, because the packets that would have filled the gap are simply absent. Adaptive buffering handles the variance of delay, not the absence of data. Distinguishing the two cases matters for the sender: a burst loss is a signal to reduce the sending rate, while high jitter with no loss is a signal to nothing at all. The feedback mechanisms that express these distinct conditions are covered in RTCP feedback messages and retransmission.