Multipath TCP and the Data-Level Sequence Space

A connection that uses several network paths at once cannot use the ordinary TCP sequence number for both purposes it serves. That number is simultaneously a position in the byte stream and a position in one path’s transmission order, and once there is more than one path the two diverge: the same byte stream is spread across subflows that have independent losses, independent round-trip times, and independent windows.

Multipath TCP resolves this by keeping two counters and defining a mapping between them. The design is worth studying even outside MPTCP, because the split it makes is the same split any multipath transport must make, and the costs it accepts are the costs that follow from the split.

Two counters with different widths

The connection-level counter is the data sequence number, which numbers the byte stream exactly as the counter described in Sequence Numbers and the Sliding Window Arithmetic does, but with a wider field. The option can carry 32 or 64 bits of it, and the initial value is derived from the key exchanged in the handshake, so both ends agree on where the stream starts without a separate negotiation. The width matters because a 32-bit space wraps in about 34 seconds at 1 Gbit/s, and a wrapping counter at the connection level would reintroduce exactly the ambiguity between old and new bytes that the extra level exists to remove.

The subflow-level counter is ordinary TCP’s 32-bit sequence number, carried in the subflow’s own header and acknowledged by the subflow’s own ACKs. It numbers bytes in transmission order on that one path. Two subflows carrying the same connection have unrelated sequence spaces, unrelated initial sequence numbers, and unrelated retransmission state.

The mapping between them is carried explicitly. A segment sent on a subflow carries a data sequence signal option that states the connection-level acknowledgement, the data sequence number of the first payload byte, the subflow sequence number of that same byte, and the number of bytes the mapping covers. The receiver uses it to place the bytes at the right position in the connection’s stream and to acknowledge them at both levels.

Data Sequence Signal (DSS) option:
  Kind, Length, Subtype, flags (A: Data ACK present, M: mapping present,
                                a: Data ACK is 8 octets, m: DSN is 8 octets)
  Data ACK                    4 or 8 octets
  Data Sequence Number        4 or 8 octets
  Subflow Sequence Number     4 octets
  Data-Level Length           2 octets
  Checksum                    2 octets, only if negotiated

The option is variable length twice over. Flag bits decide whether the acknowledgement and the mapping are present at all, and separate flag bits decide whether each is four or eight octets. The maximum size is 28 octets, which consumes more than half of the 40-octet TCP option area. Parsing is therefore conditional on flags that an attacker controls, and the parser is the part of the stack most exposed to hostile input.

The mapping covers a range rather than a single packet, which is the reason for the explicit length field. One mapping can describe several segments, and the specification requires it to do so because segmentation or coalescing may happen below the layer that writes the option, in a network card or a middlebox. A per-packet mapping would break as soon as those devices regrouped the bytes.

The data sequence number is absolute while the subflow sequence number is relative to that subflow’s initial sequence number. The asymmetry is deliberate: a firewall that rewrites the initial sequence number of a subflow would otherwise corrupt the mapping, since the receiver would compute a different subflow numbering than the sender intended. The data-level number is protected by the handshake and is not rewritten by middleboxes, so it can serve as the anchor.

Why the mapping is signalled rather than inferred

An alternative design would derive the mapping from the subflow sequence number and a single offset established at connection setup. That works as long as every subflow carries the stream in order and never retransmits across paths. Neither holds.

Retransmission across paths is the reason the mapping must be explicit. A segment lost on one subflow can be retransmitted on another, and when it is, the retransmission carries a data sequence number that belongs to the original position in the stream and a subflow sequence number that belongs to the new path’s space. The receiver cannot infer the pairing from either number alone. The mapping option states it.

The same mechanism allows a sender to split a stream at arbitrary points. A sender with a large window on a fast path and a small window on a slow path can assign bytes to each in whatever proportion the scheduler chooses, and the receiver reassembles by data sequence number regardless of the order of arrival. Without an explicit mapping, the split would have to be at fixed boundaries known in advance.

The mapping also makes the data-level acknowledgement meaningful. The receiver’s data-level ACK is a cumulative acknowledgement over the connection’s stream, so it can advance only when there are no gaps at the data level. A gap that exists because a segment is still in flight on a slow subflow blocks the data-level ACK even though every byte below it has arrived. This is head-of-line blocking, and MPTCP does not eliminate it; it moves it from the subflow to the connection.

Head-of-line blocking, restated

The promise of multipath is that a loss on one path does not stall the connection. The promise is partially kept. A loss on one subflow does not stop the other subflows from transmitting, and their segments continue to arrive and be placed in the receive buffer. What stops is the delivery of data to the application, because the stream must be delivered in order and the missing bytes sit at the head.

The distinction matters for the buffer. The receiver must hold every out-of-order byte until the gap is filled, and the gap is filled by a retransmission whose delay is governed by the slow path’s round-trip time, not the fast path’s. A connection using a 20 ms path and a 200 ms path therefore has a reordering delay of up to 200 ms at the data level whenever the slow path loses a segment, even though the fast path is delivering normally.

The receive window, whose relationship to the sender’s buffer limits is set out in The Effective Window: Congestion Window, Receive Window, and Buffer Limits, must be sized to cover this. The data-level window has to hold the bandwidth-delay product of the sum of the paths plus the reordering delay of the slowest, which is larger than the window either path alone would need. A receiver that sizes its buffer for one path’s bandwidth-delay product will drop data when the paths have different delays, and the drops will look like congestion at the data level.

The alternative to in-order delivery is per-stream sequencing, where each stream is independent and a gap in one does not block another. That is what QUIC does with its streams, and the comparison is instructive: MPTCP keeps the single-stream contract that applications written against TCP expect, and pays for it with head-of-line blocking that QUIC’s design avoids by not having a single ordered stream at all.

Coupled congestion control

Each subflow runs its own congestion window, and the windows must be coupled or the connection is unfair to single-path traffic.

Consider a connection with two subflows on paths that share a bottleneck. If each subflow runs ordinary TCP congestion control independently, each probes for capacity and each increases until it sees loss, so the connection as a whole consumes roughly twice the bandwidth a single-path connection would. On a shared link, a multipath connection would take a disproportionate share.

RFC 6356 defines the coupling. The increase applied to a subflow’s window is scaled by a factor that depends on the windows of all subflows, so that the connection’s total increase per round trip is approximately what a single TCP connection’s would be on the best of its paths. The effect is that a multipath connection targets, in aggregate, the throughput of a single connection on its best path rather than the sum of its paths, and it does not take more than its share of a shared bottleneck.

The trade-off is that the connection is not designed to use the sum of the paths’ capacities when it is one flow among many. Two 100 Mbit/s paths that are already carrying other traffic should not be expected to yield 200 Mbit/s under coupled control: the aggregate is targeted at what a single flow would get on the best path. The exception is the case where the multipath flow is itself the dominant load and therefore changes the loss rates it observes; with two idle paths, RFC 6356 notes that the aggregate can approach the sum. The restraint is the price of not being a bad neighbour, and it is the reason multipath deployments are more often about resilience than about aggregate throughput.

The failure modes that are specific to multipath

Three failure modes have no analogue in single-path TCP.

The mapping can be inconsistent. A sender that retransmits a segment on a different subflow must send a mapping that states the same data sequence number. If the receiver has already accepted the original transmission, it now has two copies of the same bytes arriving under different subflow sequence numbers. The receiver must discard the duplicate at the data level, which requires a data-level record of the ranges it has already accepted, not just the cumulative Data ACK, or the duplicate is delivered twice. That record is proportional to the receive window, which is why an MPTCP receiver needs more memory per connection than a plain TCP receiver does.

A subflow can fail without the connection failing. A subflow that stops delivering is detected by its own retransmission timer, and the connection continues on the remaining subflows. But the data that was in flight on the failed subflow is now unacknowledged at the data level, and the sender must reinject it on another path. Reinjection is a retransmission with a fresh mapping, and it interacts with the congestion control of the receiving subflow: the data is new to that subflow’s sequence space but old at the data level, so the subflow’s congestion window must accommodate it without treating it as a new stream.

The data-level ACK can stall. A data-level ACK that does not advance causes the sender’s retransmission timer at the data level to fire, and the sender must decide which subflow to retransmit on. Choosing the fastest path is correct for latency and wrong for congestion, because it adds load to the path already carrying the most data. The scheduler’s decision is a policy with no single correct answer, and implementations expose it as a pluggable component.

Where the design lands

The two-counter split is the minimum any multipath transport needs. On every segment that carries data it costs eight bytes for the data sequence number, four for the subflow sequence number, four or eight for the data-level acknowledgement, and the length and option header around them. It buys the ability to spread a stream across paths with unrelated sequence spaces, to retransmit across paths, and to acknowledge at a level above the path.

The parts that remain hard are not the sequencing. They are the scheduler that decides which path gets which bytes, the coupling that keeps the aggregate fair, and the receiver buffer that must hold the reordering delay of the slowest path. Those are the subjects of the transport’s ongoing engineering, and they are why multipath is deployed for resilience more often than for throughput.