Fragmentation, the MTU Floor, and Path MTU Discovery Black Holes

Every link has a maximum frame payload, and the IP layer above it inherits that number as its maximum transmission unit. Ethernet’s is 1500 bytes. A path across several links has the minimum of the MTUs along it, and that minimum is the largest datagram that can traverse the path without being split. The protocols handle the mismatch between what the sender produced and what the path can carry in two ways, one of which is deprecated and one of which has a well-known failure mode.

The split, byte for byte

When a router must fragment an IPv4 datagram, it divides the payload into pieces whose lengths, plus the header, fit the outgoing MTU. The header is copied to each piece with three fields changed: Total Length becomes the size of that piece, the fragment offset becomes the position of the piece’s first payload byte divided by eight, and the more-fragments flag is set on every piece except the last.

The division by eight is the constraint that shapes everything else. The offset field is thirteen bits and counts eight-byte units, so every fragment except the last must have a payload length that is a multiple of eight. A router choosing fragment sizes computes the largest multiple of eight that fits under the MTU after subtracting the header, and uses that for all but the final piece.

Consider a 4000-byte datagram, of which 3980 bytes are payload, crossing a link with a 1500-byte MTU. The first fragment carries a 20-byte header and 1480 bytes of payload, at offset zero, with more-fragments set. The second carries 1480 bytes at offset 185, since 1480 divided by 8 is 185. The third carries the remaining 1020 bytes at offset 370, with more-fragments clear. Reassembly at the destination uses the identification field to group the three and the offsets to place them.

Reassembly is receiver-side work, and it is expensive

Fragments are reassembled only at the final destination. Intermediate routers forward each fragment independently, which means a fragment can take a different path from its siblings, and the destination must tolerate arrival out of order. The receiver holds each incomplete datagram in a buffer until either all pieces arrive or a timer expires. RFC 791 sets that timer from the remaining time-to-live of the fragments and suggests an initial value of 15 seconds; RFC 1122 rejects the derivation, because gateways treat the field as a hop count rather than an elapsed time, requires a fixed reassembly timeout, recommends one between 60 and 120 seconds, and requires that the incomplete datagram be discarded and an ICMP Time Exceeded message of code 1 sent to the source when the first fragment had arrived.

The buffering is the cost. A receiver must reserve memory for datagrams it may never complete, and the amount is bounded only by policy, not by the protocol. Implementations cap the number of concurrent reassemblies and the total memory, which means a flood of first fragments with no successors exhausts the buffer and causes legitimate traffic to be dropped. This is a denial-of-service vector that exists in the protocol design rather than in any particular implementation, and the standard mitigation is to avoid fragmenting at all.

A second cost is that a fragment carries no transport header after the first piece. A firewall or a NAT that must inspect the transport port can only do so on the first fragment, so subsequent fragments either pass uninspected or are dropped. Both choices are bad, and the usual answer is to drop fragments that cannot be inspected, which breaks legitimate large datagrams. The quoting rules that ICMP errors follow have the same limitation, as ICMP errors as a feedback channel describes.

The don’t-fragment bit and the ICMP that reports the MTU

The alternative to fragmenting is to refuse. A sender can set the don’t-fragment bit, meaning the datagram must reach the destination intact or not at all. A router that cannot forward it because the outgoing MTU is too small must discard it and return ICMP Destination Unreachable with code 4, “fragmentation needed and DF set”. RFC 1191 extends that message: the low sixteen bits of the otherwise unused four-byte field in the ICMP header carry the MTU of the next hop, so the sender learns the actual limit rather than merely that one exists.

Path MTU discovery is the loop built on that message. The sender starts with the MTU of its own link, sends datagrams with the don’t-fragment bit set, and on receiving a code 4 message lowers its estimate to the advertised value and retries. Because the advertised value is the MTU of the hop that rejected the packet, each message moves the estimate down to a size that hop is known to carry, which may still be too large for a later hop. The sender converges after one step per distinct MTU on the path that rejects a probe, so a path whose bottleneck is the first hop costs a single round trip and a path with several successively smaller hops costs one per hop.

The mechanism’s weakness is total dependence on the error message arriving. A firewall that drops all ICMP, a policy that rate-limits it, or a middlebox that strips it produces a black hole: the sender keeps sending datagrams that the path silently discards, and the connection stalls after the handshake succeeds. The handshake succeeds because the initial packets are small; the stall appears when the first full-sized data packet is sent. That signature, a working connection that freezes on the first large transfer, is the classic symptom.

# Fragment offsets a router computes for a 4000-byte datagram (3980 payload)
# crossing a link whose MTU is 1500, header length 20.
HDR, MTU, PAYLOAD = 20, 1500, 3980
max_piece = (MTU - HDR) // 8 * 8      # 1480: largest multiple of 8 that fits
off = 0
while off < PAYLOAD:
    size = min(max_piece, PAYLOAD - off)
    more = off + size < PAYLOAD
    print(f"offset={off // 8:5d} len={size:5d} total={size + HDR} more={more}")
    off += size

The output is three fragments: 1480 bytes at offset 0, 1480 at offset 185, and 1020 at offset 370, the last with the more-fragments flag clear. Every offset is a multiple of eight because max_piece was rounded down to one.

The lower bound of 1280 in IPv6 is not arbitrary. Every link is required to carry at least 1280 bytes, so a sender that keeps its packets at or below that size never needs discovery and never needs to fragment. IPv4’s corresponding floor is 68 bytes, a value small enough that no modern link enforces it. RFC 8899, which generalises discovery to datagram transports that have no fragmentation of their own to fall back on, takes the same position: probe upward from a conservative base size rather than assuming any particular path can carry a full-size datagram.

IPv6 removes router fragmentation

IPv6 changes the division of responsibility. Routers do not fragment; only the source may, and only by inserting a Fragment extension header. RFC 8200 requires every link to support at least 1280 bytes, and requires a router that cannot forward an oversized packet to return ICMPv6 Packet Too Big with the MTU of the next hop. The message is mandatory rather than optional, and the sender is required to react to it.

The design intent is that fragmentation becomes an endpoint concern that can be avoided entirely by discovering the path MTU. The design assumes the ICMPv6 message is delivered. Where it is filtered, IPv6 fails in the same way IPv4 does, except that there is no fallback: an IPv4 sender can clear the don’t-fragment bit and let routers split the packet, while an IPv6 sender has no such option. That makes ICMP filtering strictly more damaging in IPv6, and it is why the operating guidance for IPv6 firewalls treats Packet Too Big as a message that must be permitted.

The floor also sets what a receiver must accept. RFC 8200 requires a node to accept a fragmented packet that reassembles to 1500 octets, and to accept unfragmented packets as large as the MTU of each link it is directly attached to; links with a configurable MTU are to be set to at least 1280, with 1500 recommended so that an encapsulated packet still fits. The IPv4 numbers are smaller. RFC 1122 requires a host to reassemble at least 576 bytes, and 576 is the largest datagram any IPv4 host is obliged to accept, fragmented or not. A router that forwards a packet larger than the next hop’s MTU without sending Packet Too Big breaks discovery silently.

Tunnels and the MTU they hide

Encapsulation adds a header that the underlay must carry but the overlay’s MTU calculation does not see. An IP-in-IP tunnel adds at least twenty bytes, so a 1500-byte packet becomes 1520 and no longer fits the 1500-byte underlay. The tunnel endpoint must either fragment the outer packet, which is legal in IPv4 and not in IPv6, or set the inner packet’s size limit so that encapsulation fits.

The usual fix is to reduce the MTU of the tunnel interface by the encapsulation overhead and to clamp the TCP maximum segment size for connections crossing it. Clamping rewrites the MSS option in the SYN, which is a header rewrite performed by a device that is not an endpoint. It works and it breaks the end-to-end principle: a sender that never learns the real path MTU keeps producing segments sized for a path it is not using.

The failure mode when clamping is absent or misconfigured is again a black hole, but a subtler one. Small packets pass, large ones vanish, and the boundary is the tunnel’s effective MTU rather than any link’s. Diagnosing it requires knowing the encapsulation overhead of every tunnel on the path, which is the reason operators reduce the MTU on tunnel interfaces by a fixed margin rather than computing it exactly. The same reduction has to be reflected in the forwarding table’s notion of the interface, since a tunnel is an interface like any other and its MTU is what the lookup in on-link determination and next-hop resolution hands to the output path.

Choosing between fragmenting and discovering

The trade is between a mechanism that always works and one that works better when it works. Fragmentation guarantees delivery of a datagram of any size the sender can produce, at the cost of receiver memory, loss of transport-header visibility, and reassembly ambiguity. Discovery avoids all three, at the cost of a hard dependency on an error message that crosses administrative boundaries.

Modern practice prefers discovery and treats fragmentation as a failure to be avoided, which is why the MSS is clamped and the MTU reduced rather than relying on routers to split. The residual risk is concentrated at exactly the boundaries where policy is applied, and the way to check for it is to send a probe with the don’t-fragment bit set at the expected size and see whether it arrives, which is the same test the discovery loop performs.