Simulcast and Scalable Video: Choosing Which Layer to Send

A video sender that must adapt to a receiver’s available bandwidth has two structural options. It can encode the picture several times at different resolutions and let the receiver pick one, or it can encode once in a layered form where a base layer plus enhancement layers reconstruct progressively finer pictures. The first is simulcast, the second is scalable video coding. Both solve the same problem and both pay for adaptivity in bits.

The problem is concrete. A sender cannot know the available bandwidth precisely; it has an estimate from transport feedback that lags and is noisy. A single encoding at a fixed quality either wastes bandwidth when the path is good or exceeds it when the path degrades, and re-encoding to a new resolution mid-stream costs a keyframe and a visible quality step. Layering moves the adaptation decision to the receiver and to the packet level, where it can happen within a frame.

Simulcast: independent encodings, independent streams

In simulcast, the sender runs several encoders over the same input, typically at different spatial resolutions, and sends each as its own RTP stream with its own SSRC. RFC 8853 describes how these streams are signalled in SDP: a single media description with multiple a=simulcast alternatives, or multiple media descriptions grouped together, depending on whether the streams share a transport.

The receiver selects one stream and discards the others. Because the streams are independent, switching is instantaneous from the network’s perspective: the receiver stops decoding one SSRC and starts decoding another, and the only cost is that the new stream’s next frame must be independently decodable, which in practice means waiting for or requesting a keyframe.

The costs are at the sender. Encoding n resolutions costs roughly n times the encoding compute, and sending all of them costs the sum of their bitrates on the sender’s uplink. A three-layer simulcast with 1080p, 720p, and 360p might consume several megabits per second of uplink for a call where any single receiver uses a fraction of that. The saving is that no receiver ever receives more than it needs, and that the layers can be routed by a server that forwards only the selected one to each participant, so the uplink cost is paid once and the downlink cost is paid per receiver at the rate it can absorb.

The trade is therefore favourable when there is a server in the middle and unfavourable in a direct peer-to-peer call, where the sender’s uplink carries every layer regardless. This is why simulcast is the dominant design in conferencing systems and rare in two-party calls.

Scalable coding: one bitstream, nested subsets

Scalable video coding produces a single bitstream whose prefixes are themselves decodable. A base layer codes a low resolution or low frame rate picture; enhancement layers add spatial detail, temporal frames, or quality refinement. A receiver that decodes only the base layer gets a valid picture, and the enhancement layers it receives improve it.

H.264/SVC, specified in RFC 6190 for RTP carriage, defines three axes. Temporal scalability splits frames into layers by dependency, so a receiver can decode 15 fps from a stream that nominally carries 30. Spatial scalability codes the same scene at multiple resolutions, with enhancement layers predicting from the upsampled lower layer. Quality scalability refines the residual of a lower layer to improve fidelity at the same resolution.

The RTP payload format carries the layer information in a priority field: a 6-bit priority identifier, a discardable flag, a 3-bit dependency identifier, a 4-bit quality identifier, and a 3-bit temporal identifier, packed into the NAL unit header extension. The dependency identifier is the important one: a receiver can discard every packet whose dependency identifier exceeds what it can handle, and the remaining packets still form a decodable stream. The discardable flag tells a middlebox which packets to drop first when it must shed load.

That last property is the real argument for SVC in a network with selective forwarding. A router or media server that understands the priority field can drop enhancement packets under congestion without parsing the video syntax, and the receiver degrades smoothly rather than losing whole frames. With simulcast, the equivalent operation is dropping an entire stream, which is coarser but requires no payload inspection.

The costs are at the encoder and in coding efficiency. Layered prediction is less efficient than independent encoding at the same rate, because enhancement layers must predict from a base layer that was itself coded under constraints. The gap is format-dependent and generally small at high bitrates and larger at low ones. Decoder complexity is also higher, and hardware support for SVC decoding has historically been spottier than for baseline H.264.

Switching costs and what the receiver must track

Both mechanisms share a hidden cost: decoder state. A decoder that switches to a different spatial layer needs the reference frames of that layer, and those are not present if it has been decoding another layer. The switch therefore requires an intra refresh, which is expensive.

SVC reduces the cost by construction. A receiver moving from base plus one enhancement layer to base only needs no refresh at all; it simply stops feeding enhancement packets to the decoder, and the base layer continues. Moving upward requires waiting for the enhancement layer’s next refresh point, which the encoder schedules periodically. The dependency structure tells the receiver exactly which packets it must have before the switch is safe.

Simulcast has no such structure. The streams are independent, so switching between them means the new stream’s decoder needs a keyframe, and until one arrives the receiver either shows nothing or shows the last frame of the old stream frozen. A sender that responds to a layer switch request by encoding an immediate keyframe pays a bandwidth spike; one that waits for the scheduled keyframe makes the switch visible for up to the keyframe interval.

This asymmetry is the main reason SVC is theoretically superior and simulcast is practically dominant. The theoretical gain of SVC is offset by the fact that simulcast’s independent streams are trivially routable, trivially encrypted per stream, and trivially supported by every decoder that handles ordinary H.264.

Signalling and the limits of negotiation

Layer availability must be communicated before a receiver can request a layer. For simulcast, the session description lists the alternatives, and the receiver’s initial choice is made at session setup; later changes happen through feedback rather than through renegotiation. For SVC, the payload’s priority field is present in every packet, so a receiver can discover the layer structure from the stream itself.

The feedback messages that request a layer change belong to the codec control family defined in RFC 5104. A receiver can request a temporary or permanent layer increase or decrease, and a sender can acknowledge. The message carries the target layer and a flag indicating whether the request is for a single frame or for the remainder of the session, which lets a receiver probe upward without committing.

The limitation is that the sender’s response is advisory. Nothing forces a sender to honour a layer request, and nothing tells the receiver when the switch has actually taken effect beyond observing the incoming packets. A receiver that requests a lower layer and continues receiving the higher one has no way to distinguish a slow sender from a lost request, because feedback is unacknowledged in the RTP sense. The feedback timing rules that bound how often such requests can be made are the same ones described in early RTCP feedback.

Choosing between them

The decision comes down to where the adaptation must happen. If a server forwards to many receivers, simulcast’s independent streams let the server select per receiver with no decoding, and the sender’s uplink cost is amortized across the audience. If the path between sender and receiver is direct, or if the network can drop packets selectively based on priority, SVC’s single bitstream spends uplink bandwidth proportional to the highest layer actually needed rather than the sum of all layers.

There is a middle position that both mechanisms support: simulcast for the coarse spatial choice and temporal layering within each stream for the fine choice. A receiver that cannot sustain 30 fps can drop to every other frame of the stream it has selected, using the temporal identifier or the equivalent structure, without a layer switch. That combination is what deployed systems converge on, because it puts the expensive decision at the coarsest granularity where it happens rarely, and the cheap decision at the finest where it happens constantly.

What neither mechanism solves is the sender’s estimate problem. Both depend on knowing which layer to send or which to select, and that knowledge comes from transport feedback that lags the network. A receiver that switches down after a congestion event has already lost frames; one that switches down early wastes capacity. The delay in that loop, not the layering scheme, usually determines how good the adaptation feels. The bandwidth estimate itself is derived from reports described in transport-wide feedback and congestion, and the playout buffering that absorbs the resulting jitter is the subject of jitter buffer adaptation.