Cloud Infrastructure · Cloud Networking
MTU and Fragmentation Across a VPC Boundary
Why the usable MTU inside a virtual network is smaller than the physical link MTU, how encapsulation and translation each consume bytes, and what happens when a packet that needs fragmenting is dropped instead.
The maximum transmission unit of an interface is a number a host believes about its own link. In a virtual network that number is a policy choice made by the provider, and it is almost never the same as the MTU of the physical links underneath. Every layer of encapsulation between the two subtracts bytes, and every translation point that cannot fragment must drop instead. The resulting failures are distinctive: a connection that completes its handshake and then stalls the first time it sends a full-size segment.
The number the interface reports
An instance’s interface reports an MTU, and the guest operating system sizes its packets to fit. Common values inside a virtual network are 1500, matching conventional Ethernet, and 9001, the jumbo frame size. The provider documents which value applies, because it is not discoverable by negotiation: the interface simply states it, and the guest takes it at face value.
The reported value is a promise about the largest IP packet the fabric will carry from that interface to any destination inside the same virtual network. It is not a promise about destinations outside, where the path may include links with a smaller MTU, and it is not a promise about paths that traverse encapsulation. Both of those cases are handled by path MTU discovery, which depends on receiving ICMP messages that are frequently filtered, and which interacts with every translation point on the path, as internet gateway, NAT, and source address translation explains for the NAT case.
The relationship between the reported MTU and the physical MTU is the source of most of the confusion. If the physical fabric carries 9000-byte frames and the overlay adds 50 bytes of encapsulation, the largest inner packet that fits is 8950, so the interface cannot honestly report 9000 unless the fabric is willing to fragment the encapsulated packet. Providers therefore choose a reported MTU with headroom for their own encapsulation, which is why a virtual interface may report 1500 on a fabric that carries 9000-byte frames and why it may report 9001 only when the tenant’s traffic stays inside a region whose underlay supports it.
Where the bytes go
The arithmetic is worth doing explicitly, because each term is a place where a design can lose a packet.
An Ethernet header with an 802.1Q tag is 18 bytes, and the frame check sequence is 4, so a 1500-byte IP packet becomes a 1522-byte frame. If the frame is then encapsulated for an overlay, an outer Ethernet header, an outer IP header, a UDP header, and the encapsulation header are added in front, and the whole inner frame becomes the payload. For VXLAN the outer additions are 14 bytes of Ethernet, 20 of IPv4, 8 of UDP, and 8 of VXLAN, which is 50 bytes; the inner frame’s own 14-byte Ethernet header and 4-byte FCS are part of the payload but the FCS is usually not carried, so the effective overhead on the wire is around 50 bytes plus whatever the inner frame carries.
A tunnel that adds 50 bytes to a 1500-byte packet needs a path MTU of at least 1550. If the underlay carries 1500-byte frames, the encapsulated packet does not fit, and the choice is to fragment the outer packet or to reduce the inner MTU. Fragmenting the outer packet is legal for IPv4 and forbidden for IPv6, where only the source may fragment; a router that receives an IPv6 packet too large for the next link must drop it and return an ICMPv6 Packet Too Big message. RFC 8201 defines the discovery procedure that depends on those messages.
| Component | Bytes |
|---|---|
| Inner IPv4 header | 20 |
| Inner TCP header with timestamps | 32 |
| Payload | up to MTU minus 52 |
| Outer Ethernet | 14 |
| Outer IPv4 | 20 |
| Outer UDP | 8 |
| VXLAN | 8 |
For a 1500-byte inner MTU, the largest TCP payload is 1448 bytes with timestamps, and the encapsulated frame on the wire is 1550 bytes plus the outer Ethernet overhead. A design that assumes the payload can be 1460 bytes will produce packets that need fragmenting at the tunnel.
The two ways to fail
There are two distinct failure modes and they look different.
The first is silent discard. A packet larger than the next hop’s MTU arrives at a device that cannot fragment it, either because the protocol forbids it or because the device is configured not to. The device drops the packet and, if it is well behaved, returns an ICMP message indicating the size it can accept. If the message is filtered, the sender retries with the same size, and the connection stalls permanently. Because the handshake uses small packets, the connection establishes normally and then hangs on the first data segment that fills the window.
The second is fragmentation, which works but is expensive. A fragmented packet is reassembled at the destination, which must hold all fragments, and a lost fragment loses the whole packet while the surviving fragments consume buffer space until a timer expires. Fragmentation also breaks stateless filters that do not track fragment offsets: a non-initial fragment carries no transport header, so a rule that matches on ports cannot evaluate it. That is why ACLs commonly have a rule for fragments as a separate case, and why the usual guidance is to avoid fragmentation rather than to handle it.
The third case, which is a variant of the first, is fragmentation in the underlay that is invisible to the tenant. If the fabric fragments an encapsulated packet, the fragments are reassembled by the tunnel endpoint, and the tenant sees nothing. This works, and it costs the fabric reassembly buffers at every tunnel endpoint, which is why providers prefer to avoid it and why the reported MTU is chosen to make it unnecessary.
Discovery and its dependency on ICMP
Path MTU discovery works by probing upward. The sender starts with a conservative size or with the interface MTU, sets the Don’t Fragment bit, and waits for either an acknowledgement or an ICMP message saying the packet was too large and naming the size that fits. The sender then reduces its size and retries. For IPv4 the mechanism is defined in RFC 1191; for IPv6 the equivalent is in RFC 8201 and is mandatory rather than optional, since routers cannot fragment.
The dependency is total: without the ICMP message, the sender never learns. Any filter that drops ICMP, any NAT that fails to translate the quoted header, and any middlebox that discards unknown ICMP types breaks discovery for every flow through it. This is the reason the interaction between filtering and MTU is a design concern rather than an implementation detail, and it is why security groups and the stateful connection tracker spends effort associating ICMP errors with the flows that caused them.
# Ask the kernel what it believes, then probe with a specific size.
ip link show eth0 | grep mtu
ping -M do -s 1472 10.20.4.17 # 1472 + 8 ICMP + 20 IP = 1500
# A reply means the path carries 1500; "message too long" means it does not.The -s 1472 value is the payload that makes the total IP packet exactly 1500 bytes with a 20-byte IPv4 header and an 8-byte ICMP header. Using -M do sets Don’t Fragment, so an oversized probe is refused locally or reported by the first hop rather than being fragmented silently. Probing upward from a smaller size and stopping at the last success gives the path MTU, subject to the caveat that the path can change between probes.
Clamping at the boundary
Where discovery cannot be relied on, the alternative is to reduce the size the endpoints choose before any packet is sent. The mechanism is maximum segment size clamping: a device on the path rewrites the MSS option in the SYN and SYN-ACK packets to a value that fits the smallest MTU on the path. The endpoints then never send a segment larger than that, and no packet ever needs fragmenting.
Clamping has limits. It applies only to TCP, because only TCP has an MSS option; UDP has no equivalent and a large datagram will still need fragmenting or will be dropped. It requires the clamping device to see the handshake, which means it must be on the path in both directions, and a device that sees only one direction can rewrite only one side’s option. It also interacts badly with middleboxes that verify checksums, since rewriting the option changes the TCP checksum and requires the device to recompute it.
The practical pattern in a virtual network is to set the interface MTU low enough that the guest never produces a packet the fabric cannot carry, and to rely on discovery only for paths that leave the network. Setting it too low costs throughput, since the per-packet overhead is fixed and a smaller payload means more packets for the same bytes; setting it too high produces the stall. The value the provider reports is chosen to be the largest that works without fragmentation inside the network, which makes it the correct default and a bad value to override upward.
Jumbo frames and the regional boundary
Jumbo frames reduce the per-packet overhead and are worth using where they are supported end to end. A 9001-byte MTU carries roughly six times the payload of a 1500-byte MTU per packet, which reduces the packet rate for a given throughput by the same factor. The benefit is real and the risk is that support is not universal: a path that includes a link with a 1500-byte MTU will drop jumbo packets, and discovery may not report it if ICMP is filtered.
The boundary is usually regional or at a gateway. Traffic that stays within a zone or a region may be carried on a jumbo-capable fabric, while traffic that crosses to another region or leaves through an internet gateway is carried on links with conventional MTUs. A design that sets a jumbo MTU on an interface and sends traffic across such a boundary depends on discovery working, and where it does not, the failure is a stall on the first large transfer rather than a slow connection. Because the failure is intermittent and depends on the destination, it is frequently misattributed to the application.
The defensible default is the value the provider reports for the interface’s own network, with ICMP permitted along every path where discovery is needed, and with an explicit Don’t Fragment probe used to confirm the path rather than a successful connection being taken as proof. A connection that completes a handshake proves only that packets smaller than the smallest MTU on the path can traverse it.