Early RTCP Feedback: Negative Acknowledgements, Picture Loss, and Transport Congestion

Periodic receiver reports describe an interval: how much was lost, how much the delay varied. They cannot express a single event, such as a packet needed for a frame that must be decoded within eight milliseconds. The gap between the interval view and the event view is what the feedback profiles exist to close.

RFC 4585 defines an RTP profile, RTP/AVPF, in which a participant may send a feedback packet immediately when an event occurs, rather than waiting for the regular report timer. The constraint that makes this safe is an allowance that accrues over time: each participant earns credit at a rate derived from the session bandwidth and the participant count, and may spend it on early feedback only when the credit exists. A receiver that fires a message on every lost packet exhausts its allowance and is then forced to wait, which preserves the property that control traffic stays within its allocation.

The packet structure shared by all feedback messages

A feedback packet is an RTCP packet of type 205, transport layer feedback, or type 206, payload-specific feedback. Its body is a list of feedback control information blocks, and the RTCP length field bounds the whole list. Each block begins with a common header: the SSRC of the participant sending the feedback, then the SSRC of the media source the feedback is about. The payload type field of the enclosing RTCP packet, not the block, selects a format identifier that determines how the rest of the block is parsed. This indirection is what lets new feedback formats be defined without new RTCP packet types, and it is why a receiver that does not recognize a format identifier must skip the block rather than guess.

The generic negative acknowledgement format, defined in RFC 4585, packs a list of 16-bit packet identifiers into each block. Each identifier is the sequence number of a packet the receiver wants resent, and the block carries as many as fit. A receiver that has lost several packets from a frame lists them all in one block rather than sending one message per loss, which is what keeps the feedback within the token budget.

NACK, PLI, and FIR ask for different things

A negative acknowledgement asks for specific packets. It works because the sender keeps a short window of recently sent packets, typically a second or two, and can retransmit from that window. The receiver must decide how long to wait for the retransmission before giving up, and that decision is bounded by the playout deadline: a retransmitted packet that arrives after its deadline is worthless.

Picture loss indication, defined in RFC 4585 for video, asks for something different. It says the receiver’s decoder has lost state and needs a new independently decodable frame. The receiver does not ask for specific packets, because the lost data is far enough back that retransmission is pointless or the packets are no longer in the sender’s window. A full intra request, defined in RFC 5104, is the explicit form of the same request, distinguished so that a sender can apply policy: PLI is a decoder-level complaint, FIR is a request for a specific refresh.

The distinction matters because the cost of satisfying them differs by orders of magnitude. Retransmitting a packet costs one packet’s worth of bandwidth. Encoding an intra frame costs many times a normal frame, because intra coding cannot exploit temporal prediction. A sender that treats every PLI as a request for an immediate keyframe will produce bandwidth spikes exactly when the network is already struggling, which is why the sender usually defers the refresh to the next scheduled intra frame or throttles how often it honours the request.

Temporal and spatial layer requests fall in the same family. A receiver that cannot keep up with a high-resolution simulcast layer can ask to be moved to a lower one, or can ask for a temporary layer refresh, without a full session renegotiation. The layering itself is covered in simulcast and scalable layer selection.

Transport-wide feedback and the bandwidth estimate

RFC 8888 defines a different kind of feedback: the receiver reports, for a window of recently received packets, which ones arrived and when, identified by a transport-wide sequence number rather than by the media sequence number. The distinction is that a transport-wide counter spans all streams sharing the transport, so the sender gets one coherent view of what the path delivered rather than separate, partially overlapping views per stream.

From that report the sender derives an estimate of available bandwidth. The standard approach, described in RFC 8888’s discussion, is to measure the arrival rate of the acknowledged packets and to compare it against the rate at which they were sent. When the arrival rate falls below the send rate and one-way delay is increasing, the path is queueing, and the estimate drops. When the arrival rate matches the send rate and delay is flat, the estimate can rise cautiously.

The key property is that the estimate is derived from delivery timing, not from loss. Loss-based estimation reacts only after a queue has already overflowed, which for real-time media means the delay spike has already destroyed the call’s quality. Delay-based estimation reacts to the queue building, before it overflows, at the cost of being noisier and of confusing a delay increase caused by a routing change with one caused by congestion. The trade is explicit: earlier signal, less certain interpretation.

One-way delay cannot be measured from timestamps alone without synchronized clocks, which is why the comparison uses the trend of delay rather than its absolute value. A constant offset between sender and receiver clocks cancels out in a difference, so a rising trend is visible even when the two clocks disagree by seconds.

The timing rules and why they exist

Early feedback is governed by a deterministic schedule rather than by a simple rate limit, because a rate limit alone allows all participants to fire simultaneously after a quiet period. RFC 4585 defines a virtual time function that advances at a rate proportional to the number of participants and the session bandwidth, and allows a participant to send when the virtual time has advanced past its last transmission plus an interval that depends on the feedback type’s priority.

Feedback messages are not all equal in urgency, and the profile assigns each a priority that determines how much the virtual time must advance before it may be sent. A message the profile treats as immediate can go out as soon as credit exists; one treated as early waits for a smaller advance than a regular report would require. A keyframe request belongs in the early class rather than the immediate one, because it is not so urgent that it justifies spending the whole allowance.

The result is a system where the total feedback rate scales down as the session grows, which is the same property the regular report interval has. A two-party call can send feedback frequently; a session with a hundred participants cannot, and the mechanism degrades by making each participant’s messages less frequent rather than by making the aggregate unbounded.

What retransmission costs and when it is worth it

Retransmission in RTP is not a repair mechanism in the TCP sense. There is no guarantee, no ordered delivery, and no penalty for the sender ignoring the request. The receiver decides whether to ask based on its deadline, and the sender decides whether to answer based on its buffer and its bandwidth estimate.

The economics are worth stating concretely. A retransmitted packet consumes bandwidth a second time and arrives one round trip later than the original would have. For a 20 ms audio frame, a round trip of 40 ms means the retransmission arrives after two frames’ worth of time has passed; it is useful only if the jitter buffer’s depth exceeds that. For a video frame in a multi-fragment packet, retransmitting one fragment may be worth it if the frame’s deadline is still ahead, and pointless if the deadline has passed.

This is why audio and video use different policies. Audio packets are small, so retransmission is cheap, and the buffer is often deep enough to absorb a round trip; audio senders commonly answer NACKs within their window. Video fragments are larger, retransmission is expensive, and the deadline is tight; video senders more often respond to loss with a keyframe request or with nothing at all, letting the error propagate until the next intra refresh. The error propagation itself is a property of the codec’s prediction structure, discussed in fragmenting encoded frames to fit the path MTU.

Silent failures in the feedback loop

A receiver that NACKs packets it has already received, because it mis-tracks its own loss state, causes duplicate retransmissions that consume bandwidth without improving anything. A sender that retransmits unconditionally, without checking its bandwidth estimate, turns a loss event into a congestion event. A receiver that sends PLI too often causes a stream of keyframes, each larger than a normal frame, which is the worst possible response to a congested link.

The subtlest failure is feedback that arrives but is ignored because the sender cannot correlate it. The media source SSRC in the feedback block must match the SSRC the sender is actually using; a sender that has changed its SSRC, or a relay that rewrites it, leaves the receiver’s feedback addressed to a stream that no longer exists. The receiver sees its requests go unanswered and concludes the network is worse than it is. Because there is no acknowledgement of feedback, the failure is silent on both ends.