Link MTU, Jumbo Frames, and Where the Path Breaks

The 1500-byte payload limit is not a property of Ethernet’s frame format. It is a number chosen when the standard was written, and the format can carry considerably more. What prevents larger frames is not physics but agreement: every device on the path must accept the same size, and a switch that does not will drop the frame rather than report a problem. The resulting failure mode, where small packets work and large ones disappear, is easy to overlook because it depends on packet size.

Where 1500 came from

The choice was a compromise between two costs. A larger maximum frame amortises the 18 bytes of header and trailer over more payload, raising goodput; a smaller one bounds the time a station must wait to transmit on a shared medium, since a station cannot begin until the current frame ends. At 10 Mbit/s, a 1500-byte payload occupies the wire for about 1.2 milliseconds, and that figure was judged acceptable as a worst-case access delay.

The number also had to fit the receive buffers of contemporary controllers. A 1500-byte limit with a few kilobytes of buffering per port was affordable in the hardware of the time. None of these constraints bind today: switched links are point-to-point, so no station waits for another’s frame, and buffers are measured in megabytes. The number persists because it is what everything expects, and changing it requires coordinated configuration.

The 1500 bytes refer to the payload, not the frame. A maximum-length frame is 1518 bytes on the wire, or 1522 with a VLAN tag, or 1526 with two. Those extra bytes matter when configuring a device that specifies its limit in frame bytes rather than payload bytes, and they are the reason a nominal “1518” setting is not the same as “1500 MTU” on a port that must accept tagged frames.

What an MTU actually constrains

The interface MTU bounds the largest payload the link layer will carry, and everything above it must respect that bound. For IPv4, a datagram larger than the outgoing interface’s MTU must be fragmented by the sender, unless the don’t-fragment flag is set, in which case the sender must return an error. For IPv6, routers never fragment; only the source may, and only when it has no other option, so a datagram that exceeds a downstream link’s MTU is dropped and an ICMPv6 Packet Too Big message is returned.

The fragmentation cost is not just the extra headers. A fragmented datagram is reassembled only at the final destination, so a single lost fragment loses the whole datagram, and the loss rate of a datagram becomes roughly the loss rate of a fragment times the fragment count. Splitting a 9000-byte datagram into 1500-byte fragments produces seven fragments, since each carries at most 1480 bytes of data after its 20-byte IPv4 header, so the exposure is multiplied by seven. This is the strongest argument for larger MTUs: fewer fragments, and less per-packet header processing in the stack.

The counter-argument is latency. A frame occupies the transmitter for its full serialisation time, so a large frame blocks the link for that long. At 1 Gbit/s, a 1500-byte frame takes about 12 microseconds; a 9000-byte frame takes about 72. On a link shared by latency-sensitive traffic and bulk transfer, the larger frame adds up to 60 microseconds of queueing delay to every packet waiting behind it. At 10 Gbit/s the figures shrink by ten and the argument weakens; at 100 Mbit/s they grow and it strengthens. The trade is bandwidth efficiency against worst-case delay, and its direction depends on link speed and traffic mix.

Jumbo frames and the absence of a standard

Frames with payloads above 1500 are called jumbo frames. There is no IEEE standard for them at the sizes commonly deployed; the de facto convention is a 9000-byte payload, giving a 9018-byte frame, large enough to amortise the header substantially and small enough for the buffer memory of common interfaces.

Because there is no standard, there is no guarantee that two devices configured for jumbo frames agree on the maximum. A path may support 9000 at one hop and 4088 at another, and a frame sized for the larger will be dropped by the smaller. Vendors document their own maximums, and the only reliable approach is to configure the smallest value on the path, or to verify end to end before relying on the larger size.

The failure mode is asymmetric and silent. A frame larger than a receiving port’s limit is dropped with a counter incremented, but no ICMP message is sent, because the link layer does not produce one and the IP layer never sees the frame. The sender’s retransmission produces another oversized frame, which is dropped again. A TCP connection in this state stalls on large segments and proceeds on small ones, so it appears to work for interactive traffic and fail for transfers, which is easy to mistake for an application problem.

A related and more confusing case is a path where the MTU is reduced in one direction only. If the forward path supports 9000 and the return path supports 1500, requests succeed and responses fail, or the reverse, depending on which direction carries the bulk data. Diagnosing this requires measuring the MTU in both directions separately.

Discovering the path MTU

IPv4’s mechanism is defined in RFC 1191: send a datagram with the don’t-fragment bit set, and when a router cannot forward it, the router returns an ICMP Fragmentation Needed message carrying the next-hop MTU. The sender reduces its estimate and retries. The process converges in a few round trips and is then cached per destination.

The mechanism fails when the ICMP message does not arrive. A firewall that drops all ICMP, a router configured not to generate the messages, or a middlebox that strips them, all produce the same result: the sender never learns, keeps sending oversized datagrams, and the connection stalls. This is the classic black hole, and its signature is a connection that establishes, transfers a little data, and then hangs.

The workarounds are heuristic. Sending a probe sized just under the assumed MTU and watching for acknowledgement can detect the stall; reducing the segment size unconditionally avoids it at the cost of efficiency. Modern stacks implement a packetisation-layer discovery that does not depend on ICMP at all: RFC 8899 describes probing at the transport layer, where the sender can infer the path MTU from which probe sizes are acknowledged. That approach works through ICMP-blocking paths because it uses the transport’s own acknowledgements as the signal.

IPv6 makes the ICMP dependency explicit. Routers do not fragment, so a packet too large for a downstream link is always dropped with a Packet Too Big message, and a network that filters that message breaks IPv6 for any path with a smaller MTU than the sender assumed. The minimum IPv6 MTU that every link must support is 1280 bytes, so a sender that limits itself to that size is always safe, at a cost in efficiency.

Configuring a consistent MTU

The settings that must agree are more numerous than they appear. Each interface has an MTU; each switch port has a maximum frame size, sometimes expressed as payload and sometimes as frame bytes; a VLAN tag adds four bytes and a stacked tag eight; and a tunnel or overlay adds its own encapsulation, which reduces the payload available to the inner frame. A path with a 9000-byte physical MTU and a 50-byte tunnel header has 8950 bytes for the inner packet, and a sender that assumes 9000 will be wrong.

Layer of overhead	Bytes
Ethernet header and trailer	18
One VLAN tag	4
Two tags	8

The practical rule is that the MTU of an encapsulated path is the physical MTU minus the encapsulation overhead, and every device that terminates the encapsulation must be configured with the reduced value. An interface that carries a tunnel and is left at the physical MTU will emit frames that exceed the path once encapsulated, and the drop happens at the far end.

Software interfaces inherit and adjust MTU in ways that are easy to overlook. A bridge interface typically takes the minimum of its member ports, so adding a port with a smaller MTU silently reduces the bridge’s capability. A bond interface has one MTU for the aggregate and applies it to its member ports, so the value is configured on the bond rather than on a member. A virtual interface created by a hypervisor may default to 1500 regardless of the physical link, so jumbo frames configured on the host do not reach the guest. The receive path implications of a large frame, including how it is split across buffers, are described in the network interface receive path.

The reason to bother with jumbo frames at all is the header-to-payload ratio. At 1500 bytes the 18-byte header and trailer are 1.2 percent of the frame; at 9000 they are 0.2 percent. That difference is real but modest, and it is smaller than the cost of a single misconfigured device on the path. Deployments that adopt jumbo frames usually do so to reduce per-packet processing in the stack, where the saving is larger than the wire saving, or to reduce fragmentation of tunnelled traffic. The wire-level frame format that the larger payload still has to fit is described in Ethernet frame layout, and the way several physical links are combined without changing the MTU is covered in link aggregation and frame distribution.