Opus: Two Codecs Behind One Payload Type

A single payload type in RTP must name a format whose parameters the receiver can use without negotiation per packet. Opus is unusual in how much it packs under one identifier: a codec that switches internally between two fundamentally different coding methods, at frame sizes from 2.5 ms to 60 ms, at bitrates from around 6 kbit/s to above 500 kbit/s, and between mono and stereo, all without any change to the RTP payload type or the session description.

That flexibility is why Opus is the default choice for interactive audio in WebRTC and in most modern conferencing systems. It is also why implementations misconfigure it: the codec’s parameters interact in ways that are not obvious from the API surface.

Two coding methods with opposite assumptions

SILK descends from Skype’s speech codec and is built for speech. It models the vocal tract as a linear predictive filter excited by a pitch-synchronous residual, and it codes the residual with a small number of pulses per frame. At low bitrates this is efficient because speech has strong short-term predictability. The method degrades gracefully when the bitrate falls, losing spectral detail before losing intelligibility.

CELT takes the opposite approach. It transforms overlapping blocks of samples, codes the resulting spectrum with a band-energy envelope plus normalized coefficients, and reconstructs with attention to the temporal envelope. It makes no speech-specific assumption, so it handles music and non-speech audio well, and its transform structure means it has algorithmic delay tied to the block size rather than to a pitch analysis window.

The two methods are not layered; they run in different frequency regions when used together. In hybrid mode, the lower band up to roughly 8 kHz is coded by SILK and the remaining band by CELT. That split follows from what each method does well: the pitch-predictive approach captures the harmonic structure of voiced speech in the low band, while the transform approach captures the broader spectrum above. The receiver decodes both and sums them.

The consequence for configuration is that bitrate allocation between the two halves is a real decision. At a fixed total bitrate, giving more to the low band improves speech intelligibility under noise, and giving more to the high band improves the impression of presence and the handling of music. The codec’s own bitrate controller does this allocation, which means the application’s influence is limited to the total and to the bandwidth hint.

Frame size sets the latency floor

Opus supports frame durations of 2.5, 5, 10, 20, 40, and 60 ms. The duration is chosen per packet by the encoder, and the receiver learns it from the packet’s table-of-contents byte rather than from the RTP header. A packet may contain more than one frame, up to a total of 120 ms of audio per packet.

The frame size determines the algorithmic delay floor. A 2.5 ms frame is the minimum and gives the lowest latency, at the cost of more packets per second and therefore more header overhead. At 20 ms frames with 48 kHz audio, each packet carries 960 samples; at 2.5 ms, 120 samples. The RTP, UDP, and IP headers cost roughly 40 bytes per packet uncompressed, so the same bitrate expressed in 2.5 ms packets spends far more of the link on headers than in 20 ms packets.

Longer frames reduce overhead and improve coding efficiency, because a longer window gives the transform better frequency resolution. They also increase the delay contribution: a 60 ms frame cannot begin playback until 60 ms of audio has been captured. For a conversational call, 20 ms is the common compromise, and 60 ms is used mainly for music or for links where header overhead dominates.

The frame duration interacts with packet loss concealment. A lost 20 ms packet is concealed over 20 ms; a lost 60 ms packet over 60 ms, and concealment quality degrades with duration because the pitch continuation becomes less plausible. Long frames therefore make individual losses more audible, even though they reduce the number of loss events for the same loss rate.

The configuration surface in a session description

An Opus payload type in SDP carries format parameters that constrain what the encoder may do, expressed as a=fmtp attributes. The ones that matter:

• maxplaybackrate tells the encoder the highest sample rate the receiver will use, so it can avoid coding bandwidth that will be discarded.
• sprop-maxcapturerate describes the sender’s capture rate, letting a receiver know whether to expect content above 8 kHz.
• stereo indicates whether the sender may use stereo; a receiver that omits it should be sent mono.
• cbr requests constant bitrate rather than the default variable rate.
• useinbandfec enables the codec’s in-band forward error correction, where each packet carries a low-bitrate copy of the previous frame’s signal.

In-band FEC is worth understanding because it is not what the name suggests. It is not parity over the packet; it is a redundant encoding of the previous frame embedded in the current one. A receiver that loses packet n can reconstruct an approximation of it from the FEC data in packet n+1, at reduced quality. This works only when losses are isolated: if n and n+1 are both lost, the redundancy for n is gone too. The mechanism converts random loss into degraded audio rather than silence, and it costs bitrate continuously whether or not any loss occurs.

Why the timestamp must match the frame contents

RFC 7587 constrains the relationship between the RTP timestamp and the audio in the packet. The timestamp advances by the number of samples at 48 kHz, regardless of the codec’s internal rate. Opus always operates on a 48 kHz timeline for RTP purposes even when the content is narrowband, so a 20 ms packet always advances the timestamp by 960.

The payload’s table of contents gives the frame count and each frame’s duration, and the sum must equal the RTP timestamp step. A sender that pads or truncates without adjusting both produces a stream where the receiver’s playout clock drifts against the content. Because Opus tolerates a range of frame sizes, a mismatch is not always immediately audible; it accumulates.

The encoder’s own sample rate handling is a separate matter. An encoder fed 16 kHz input resamples internally to whatever it needs and codes the band it has. The RTP timestamp still advances at 48 kHz, because that is the payload format’s clock. Confusing the input rate with the RTP clock rate is a common source of double-speed or half-speed playback.

Bitrate control and its interaction with the network

Opus has three rate control modes: constant bitrate, constrained variable bitrate, and unconstrained variable bitrate. Variable bitrate lets the encoder spend more bits on complex passages, which improves quality at the same average rate but produces bursty output. On a network with shallow queues, bursts are absorbed; on one with a congested bottleneck, they can fill a queue and cause delay for the stream itself and for anything sharing the link.

The sender’s rate should follow the available bandwidth estimate, which for RTP-based systems comes from receiver reports and from transport feedback. The coupling is indirect: the bandwidth estimate is a property of the transport, the encoder bitrate is a property of the codec, and something in between must translate one into the other, typically with hysteresis so the encoder does not oscillate. The feedback mechanisms that supply the estimate are covered in RTCP feedback messages and retransmission.

Where Opus differs from older codecs is the width of its useful range. A single configuration can run from about 6 kbit/s for narrowband speech to several hundred kbit/s for stereo music, which means a session can start at one rate and move by an order of magnitude without renegotiation. That range is why the codec displaced format-specific alternatives: one implementation, one payload type, and a bitrate dial.

Misconfiguration that survives testing

The most common is a mismatch between what the sender assumes and what the receiver declared. If the receiver’s session description sets maxplaybackrate to 8000 and the sender ignores it, the receiver will discard the high band, wasting bits and, in a hybrid stream, potentially discarding content the low band does not carry. The audio is still intelligible, so the error goes unnoticed.

The second is misuse of in-band FEC as a substitute for a jitter buffer. FEC reconstructs a lost frame only if the following packet arrives; it does nothing for a burst. A receiver configured for FEC and a shallow buffer will lose the reconstruction along with the original.

The third is frame size selection driven only by latency. Choosing 2.5 ms frames to minimize delay multiplies the packet rate by eight compared with 20 ms frames, and on a link where per-packet overhead matters, the resulting bandwidth increase can push the stream into congestion. Latency and overhead pull in opposite directions here, and the right point depends on the link, which the codec cannot know. The buffer that absorbs the resulting jitter is discussed in sizing and adapting the jitter buffer.