Fragmenting Encoded Frames to Fit the Path MTU

A single H.264 keyframe at high resolution can be hundreds of kilobytes. A datagram that large cannot cross the Internet: the path MTU is typically 1500 bytes on Ethernet, and IPv6 requires every link to carry at least 1280 bytes, with no fragmentation performed by routers. The encoded frame must therefore be split before transmission, and where that split happens determines whether a lost fragment costs a few milliseconds of concealment or an entire frame.

IP-level fragmentation exists and can carry a large UDP datagram, but it is the wrong tool for media. A fragmented datagram is reassembled only when every fragment arrives, so one lost fragment destroys the entire original datagram, which in this case is the whole frame. There is also no way for the receiver to know which parts of the frame it is missing, because the fragments carry no codec-level structure. RFC 8085 advises applications to avoid IP fragmentation for these reasons.

The alternative is fragmentation inside the RTP payload format, where each fragment is a full RTP packet with its own sequence number. Loss of one fragment loses part of one frame, which the decoder may be able to conceal, and the receiver can tell exactly which bytes of the frame are missing.

The unit that must stay whole

Before fragmenting, a payload format defines a unit of independence. In H.264, RFC 6184 calls it a NAL unit: a self-contained syntax element that the decoder can parse without reference to the bytes of other NAL units. A slice is a NAL unit; a sequence parameter set is a NAL unit. Fragmentation happens at NAL unit boundaries, and each NAL unit is carried by one or more RTP packets.

The mapping has three modes. In single NAL unit mode, each packet carries exactly one NAL unit and no fragmentation header. In STAP-A mode, several small NAL units are aggregated into one packet, which reduces header overhead for parameter sets and small slices. In FU-A mode, one NAL unit is fragmented across several packets. The mode is chosen per packet, so a stream can aggregate small units and fragment large ones in the same frame.

The payload’s first byte distinguishes the modes by the NAL unit type it carries. Types 1 through 23 indicate a single NAL unit whose header is the payload’s first byte. Type 24 is STAP-A, 28 is FU-A. A receiver reads one byte and knows how to interpret the rest.

What the fragmentation header carries

An FU-A packet replaces the NAL unit header with a two-byte fragmentation header. The first byte repeats the original NAL unit’s forbidden-zero bit and nal_ref_idc, then sets the type to 28. The second byte has a start bit, an end bit, a reserved bit, and the original NAL unit type in its low five bits.

The original type has to be carried because the decoder needs to know whether it is receiving a slice, a parameter set, or something else before it has the whole unit. The start and end bits let the receiver delimit the unit: the fragment with the start bit begins a new unit, the one with the end bit completes it. Fragments in between carry neither.

All fragments of one NAL unit share the RTP timestamp, because they describe the same sampling instant, and all but the last have the marker bit clear. This is the only signal that ties the fragments together at the RTP level; the sequence numbers are consecutive, but consecutive sequence numbers also occur between different NAL units, so the receiver relies on the start and end bits to know where one unit ends and the next begins.

The cost is two bytes per fragment on top of the twelve-byte RTP header, plus the UDP and IP headers. With a 1500-byte MTU and 20-byte IPv4 plus 8-byte UDP headers, the RTP payload capacity is 1460 bytes, and the FU-A header reduces the usable NAL fragment to 1458 bytes.

Choosing a packet size below the MTU

Using the full MTU is a poor default. The path MTU is not known in advance, tunnels and VPNs reduce it, and a packet that exceeds it is either fragmented by an intermediate router (defeating the purpose) or dropped with an ICMP message that may never arrive. RFC 8085 recommends that applications use a conservative datagram size and, where possible, discover the path MTU.

The common practice for RTP is to target roughly 1200 bytes of payload, which leaves room for the RTP, UDP, and IP headers inside a 1280-byte IPv6 minimum and inside the smaller MTUs imposed by some tunnels. The specific number matters less than the principle: stay comfortably below the smallest plausible MTU on the path, because the penalty for exceeding it is a lost packet, and the benefit of the last few hundred bytes is a marginal reduction in header overhead. The per-packet header cost itself is dominated by the fields described in the RTP fixed header.

The trade-off is quantifiable. At 1200 bytes of payload per packet, a 100 kB frame needs about 84 packets; at 1400 bytes it needs 72. The saving is 12 packets, or 12 times the per-packet header cost, which at 40 bytes per packet is under 500 bytes, roughly half a percent of the frame. Against that, a single path with a 1280-byte MTU would drop every 1400-byte packet, costing the whole frame.

Loss behaviour and the case for smaller slices

When a fragment is lost, the NAL unit it belonged to cannot be reconstructed, and the decoder discards the fragments it did receive. The frame may still be partially decodable if the encoder split it into multiple slices: each slice is an independent NAL unit, so losing one slice’s fragments damages a region of the picture rather than all of it.

This is why encoders targeting lossy networks use multiple slices per frame. A frame split into eight slices loses at most one slice per lost fragment group, and the error is confined to a horizontal band. The cost is coding efficiency: each slice carries its own header and resets some prediction context, so more slices means more bits for the same picture quality. The number of slices is a direct trade of compression efficiency against loss resilience, and it is set by the encoder, not by the transport.

A receiver’s response to a lost fragment is dictated by the payload format’s rules and by the codec’s error propagation. For H.264, a lost slice means the pixels it covers are unknown, and any later frame that predicts from those pixels inherits the error until an intra-coded frame refreshes the region. This is the mechanism behind the smeared, blocky artifacts that persist after a brief glitch: the error is not the lost frame, it is the propagation through the prediction chain.

Reassembly without unbounded state

The receiver’s fragment reassembler must hold partial NAL units until either the end fragment arrives or the unit is declared lost. Both paths need bounded state, because a stream that loses end fragments would otherwise accumulate memory indefinitely.

#include <stdint.h>
#include <stdlib.h>
#include <string.h>

struct frag {
    uint8_t *buf;
    size_t   len;
    size_t   cap;
    int      started;      /* saw a start fragment */
    uint32_t timestamp;    /* fragments must share this */
};

/* Returns 1 when a complete NAL unit is available in f->buf, 0 if more needed,
 * -1 if the fragment cannot belong to the unit under construction. */
int frag_push(struct frag *f, uint32_t ts, int start, int end,
              const uint8_t *data, size_t n)
{
    if (start) {
        f->len = 0;
        f->started = 1;
        f->timestamp = ts;
    } else if (!f->started || ts != f->timestamp) {
        return -1;                 /* orphan fragment or timestamp change */
    }

    if (f->len + n > f->cap) {
        size_t want = f->cap ? f->cap * 2 : 4096;
        while (want < f->len + n)
            want *= 2;
        uint8_t *p = realloc(f->buf, want);
        if (!p)
            return -1;
        f->buf = p;
        f->cap = want;
    }

    memcpy(f->buf + f->len, data, n);
    f->len += n;

    if (end) {
        f->started = 0;
        return 1;
    }
    return 0;
}

The timestamp check is the load-bearing line. A start fragment resets the buffer, so an orphan continuation fragment from an earlier unit cannot corrupt the current one; but a continuation whose timestamp differs from the one being assembled indicates the start fragment was lost, and the partial data must be discarded rather than concatenated with unrelated bytes.

There is a second guard needed in practice: a maximum unit size. A malicious or broken sender can emit start fragments forever, and the realloc above grows without bound. Limiting the buffer to a multiple of the largest plausible encoded frame converts that into a dropped unit instead of an exhausted process.

Why the RTP marker bit matters more here

For a fragmented frame, the marker bit on the final fragment is what tells the receiver the frame is complete. A receiver that begins decoding as soon as it has a start fragment and some continuation data will produce a partial picture and then a corrected one, which is worse than waiting a few milliseconds for the whole frame. The buffer’s playout deadline usually dominates this decision, but the marker bit is the only in-band signal that the frame has actually ended.

This is also why relays must not reorder or coalesce fragments. A middlebox that merges two RTP packets into one, or that drops the final fragment of a frame because its own buffer filled, breaks the marker semantics without any field indicating that it did so. The format assumes the transport preserves packet boundaries, and that assumption is worth stating explicitly because it is the one that NAT traversal and tunneling most often violate. The delay tolerance that decides how long to wait for a missing fragment is the jitter buffer’s, discussed in sizing and adapting the jitter buffer.