Tunnel Encapsulation Overhead and Path MTU Discovery

Path MTU discovery works because the sender learns the smallest MTU on the path and shrinks its packets to fit. The mechanism depends on a router that cannot forward an oversized packet sending back a message naming the size it could have forwarded. Put a tunnel in the middle and both halves of that arrangement change: the message describes the outer packet, while the sender is sizing the inner one, and the two are not the same packet.

Counting the bytes

Every encapsulation scheme adds a fixed, known amount. The figures are properties of the header formats, not measurements.

For VXLAN over IPv4: 14 bytes of outer Ethernet, 20 of outer IPv4, 8 of UDP, 8 of VXLAN, the last of which is described byte by byte in the VXLAN header layout. Fifty bytes. Over IPv6 the outer header is 40 bytes rather than 20, giving 70.

For Geneve the base header is 8 bytes of UDP plus 8 of Geneve header, so the same 50 or 70 before any options; each option adds a 4-byte header plus a value rounded up to a multiple of 4.

For IP-in-IP the outer header is 20 or 40 bytes with no UDP layer, so 34 or 54 including the outer Ethernet.

For GRE the outer stack is 14 + 20 + 4 for the basic header, plus 4 more if a key is present, plus 4 if a sequence number is present, so 38 to 46 over IPv4.

For an MPLS label stack each label is 4 bytes; a two-label stack adds 8 bytes between the Ethernet and IP headers.

The relevant comparison is not which is smallest but whether the total fits the underlay’s link MTU. A physical link configured at 1500 bytes and carrying 50 bytes of VXLAN overhead leaves 1450 bytes for the inner frame. A guest that believes its MTU is 1500 will emit a 1500-byte frame, the endpoint will encapsulate it into a 1550-byte packet, and the underlay will either drop it or fragment it.

Three ways to resolve the mismatch, and what each costs

The first option is to reduce the inner MTU. The endpoint advertises a smaller MTU to its guests, either through the DHCP lease or through router advertisements. This makes the inner network’s MTU honest and keeps every packet inside the underlay’s limit. The cost is that every inner packet carries slightly less payload: an inner MTU of 1450 rather than 1500 removes fifty bytes from the largest frame, which is one part in thirty, and the loss falls entirely on bulk transfers. The benefit is that nothing is ever fragmented and no discovery is needed.

The second option is to raise the underlay MTU. If the physical fabric carries 1600-byte or 9000-byte frames, a 50-byte overhead is absorbed with room to spare. This is why data centre fabrics are commonly built with jumbo frames. It removes the inner reduction entirely but requires every link and every device on the underlay path to agree, since a single 1500-byte link in the path reintroduces the problem. It also raises the per-packet transmission time on slower links and increases the burst size that buffers must absorb.

The third option is fragmentation, and it is the one that behaves worst. If the outer packet is fragmented by the underlay, the receiver must reassemble before decapsulating, because the VXLAN header and the inner frame are not independently meaningful. Reassembly at the tunnel endpoint means the endpoint needs buffers sized by the number of concurrent fragmented packets, and a lost fragment discards the whole inner frame even though the inner frame’s own delivery might have tolerated a smaller unit. IPv6 forbids routers from fragmenting at all, which removes this option on an IPv6 underlay.

RFC 7348 forbids the tunnel endpoint itself from fragmenting an encapsulated packet, allows an intermediate router to fragment it, and permits the receiving endpoint to discard such fragments silently. Its recommendation is the second option: the physical infrastructure’s MTU should be set to a value that accommodates the encapsulated frame, with path MTU discovery as a permitted alternative. The prohibition on endpoint fragmentation is what makes the third option unattractive in practice, because the party that would have to reassemble is the party that is allowed to throw the fragment away.

Where the underlay MTU cannot be raised, the reduction of the inner MTU is what remains, and it is the only option that requires no cooperation from the underlay.

Why ICMP errors do not reach the right sender

Suppose the inner MTU is not reduced, a 1500-byte inner frame is encapsulated, and an underlay router on the path has a 1400-byte link. The router cannot forward a 1550-byte packet, so it emits an ICMP “fragmentation needed” message. That message is addressed to the outer source, which is the tunnel endpoint’s underlay address, and it quotes the outer packet’s headers. The endpoint receives it and now knows that some underlay link is 1400 bytes, but the endpoint is not the sender of the inner packet, and it has no way to map the quoted outer five-tuple back to a specific inner flow when the encapsulation does not preserve that identity. All traffic between the two endpoints shares one outer five-tuple unless the source port is varied per flow.

What the endpoint can do is shrink the inner MTU for the whole tunnel, which is a global reaction to a per-path condition. If the underlay is multipathed and only one of several paths has the smaller link, the endpoint now penalizes every flow, including those that never take that path. If the smaller link is transient, the reduction persists.

An endpoint can also act as a relay: it receives the ICMP error, recognizes it as pertaining to a packet it encapsulated, and generates a corresponding ICMP error back into the overlay addressed to the inner source. That requires the endpoint to have retained enough state to identify the inner packet, which in practice means keeping a record of recent encapsulations or relying on the outer source port encoding a flow hash. Where the encapsulation carries no per-flow information, the relay is impossible and the error is silently dropped. UDP source port entropy and underlay ECMP describes the encoding that makes this possible and where it stops being reliable.

IPv6 makes this sharper. RFC 8201 forbids routers from fragmenting and requires the sender to perform discovery, and the ICMPv6 “packet too big” message is the only signal. An IPv6 underlay that drops an oversized outer packet without generating that message leaves the sender with no information at all, and the resulting behaviour is a connection that establishes and then stalls on the first full-size segment. This is the classic black-hole symptom: small packets such as handshakes and acknowledgements pass, and the first data segment disappears. On an inner TCP connection the sender retries and eventually gives up, and nothing in the inner network reports an MTU problem because the inner network never saw one.

The MSS clamp as a partial answer

TCP’s maximum segment size is negotiated in the SYN exchange and is derived from the MTU of the interface the SYN was sent on. A tunnel endpoint that rewrites the MSS option in passing SYNs to a value that fits the reduced MTU solves the problem for TCP without touching the guest’s configured MTU. The clamp is a small, stateless edit: find the option, replace the value, fix the checksum.

It is partial in two ways. It does nothing for UDP, which has no negotiation, so a UDP application sending full-size datagrams still overflows. And it does nothing for traffic that arrives already established, since the MSS is fixed at handshake time; a tunnel that appears on the path later does not retroactively change it. The clamp is therefore a safety net for the common case, not a substitute for a correct inner MTU.

Where the numbers have to be agreed

The failure mode that matters operationally is disagreement, not the specific figure. Four places hold an opinion about the MTU: the guest’s interface configuration, the endpoint’s advertised inner MTU, the underlay link MTU, and the MSS clamp. If any of them is larger than the sum the others permit, some traffic will be dropped, and the drop will be silent because the outer packet is discarded by a device that has no route back to the inner sender.

The practical rule is that the inner MTU equals the smallest underlay link MTU minus the largest encapsulation overhead any endpoint on that virtual network may apply. The “largest” qualifier matters because a fabric may run more than one encapsulation, and a packet that takes the more expensive one needs the smaller inner MTU. Computing the figure once and configuring it everywhere is simpler than discovering which of the four places was wrong after a transfer stalls.