UDP Source Port Entropy and Underlay ECMP

A leaf-spine fabric carries several equal-cost paths between any two hosts and picks one per packet by hashing a fixed set of header fields. The hash needs variety in its input: two packets that differ in no field the hash reads produce the same result and take the same path. Encapsulation removes that variety, because the fields the underlay can see describe the tunnel, not the flows inside it.

The collapse

Between two endpoints, every encapsulated packet has the same outer source address, the same outer destination address, the same IP protocol number, and the same UDP destination port. The outer IP header may carry a DSCP value, which is the same for all traffic in a class. If the hash reads those fields, the result is one value, and every flow between the two hosts takes one link.

The effect is that the underlay’s multipathing stops working at exactly the point where the traffic is heaviest. A fabric sized on the assumption that a thousand flows spread across eight spine links gets one link carrying everything between a pair of hosts. The paths are still available, and the fabric still believes it is load balancing, because its hash is doing what it was configured to do. The symptom is a small number of very hot links and a large number of cold ones, with the hot links dropping while the cold ones are idle.

The problem is not specific to overlays. The same collapse happens with any encapsulation that hides the inner headers: IPsec tunnels, GRE tunnels, and plain IP-in-IP have it. What makes it tractable in a virtual network is that the encapsulating endpoint controls the outer header, and one field in it is free to vary.

The source port as an entropy carrier

The UDP source port in an encapsulated packet has no semantic role. Nothing replies to it; the destination port identifies the tunnel, and the source port is not used to demultiplex return traffic, because the return traffic is a separate encapsulation in the other direction. The specification for VXLAN therefore leaves it available, and the usual convention is to fill it with a hash of the inner headers.

The inner headers that are hashed are the ones that identify a flow: the inner source and destination addresses, the inner protocol, and the transport ports when the inner packet has them. A five-tuple hash over those values is computed once per packet at the endpoint, and the low bits of the result become the outer source port. The underlay’s hash then reads the outer five-tuple, which now varies with the inner five-tuple, and the distribution across equal-cost paths is restored.

Two constraints on the port value are worth stating. RFC 7348 recommends the dynamic range 49152 to 65535, which is the portion of the port space that IANA sets aside for private use, so a port in that range is unlikely to collide with a service the underlay cares about. And the value should not be the tunnel’s own destination port, 4789, so that a middlebox that classifies on the port number alone cannot read the datagram as a reply in the opposite direction. Deriving the port from a hash and then adding a fixed offset avoids both problems.

#include <stddef.h>
#include <stdint.h>

/* A simple 32-bit flow hash. Real implementations use a stronger
   function; the structure is what matters here. */
static uint32_t flow_hash(const void *key, size_t len)
{
    const uint8_t *p = key;
    uint32_t h = 2166136261u;
    for (size_t i = 0; i < len; i++) {
        h ^= p[i];
        h *= 16777619u;
    }
    return h;
}

/* Map an inner flow to an outer UDP source port in [49152, 65535]. */
static uint16_t outer_src_port(uint32_t inner_hash)
{
    return (uint16_t)(49152u + (inner_hash % 16384u));
}

The range in that function is 16,384 values, which is fourteen bits of port entropy. The underlay’s hash typically selects among a small number of paths, so fourteen bits is far more than enough to spread flows; the constraint is not the width but the quality of the inner hash. A hash that produces the same value for many distinct flows gives the same port and the same path, and the collapse returns in a subtler form.

Where the entropy is not enough

The scheme fails in four situations, and each has a different remedy.

A single flow is a single flow. If one TCP connection carries the bulk of the traffic between two hosts, hashing its five-tuple yields one value, and the connection takes one path no matter how many are available. No encapsulation trick fixes this, because the underlay must preserve per-flow ordering and spreading one flow across paths reorders it. The remedies are at the transport layer, not the tunnel: more connections, or a transport that tolerates reordering.

The inner packet may have no transport ports. An ICMP echo, or a non-initial fragment of a larger datagram, has no ports to hash, so the endpoint falls back to the addresses and the protocol number. Fragments of the same datagram must all hash the same or they arrive out of order and reassembly stalls, which is why the fallback uses the identification field where one exists.

The endpoint may not hash at all. An implementation that sets the source port to a constant, or to a value derived from the endpoint’s own identity, produces a packet that the underlay hashes to a constant. The failure is quiet, because the tunnel works correctly and the fabric looks healthy while one link saturates.

The path may be selected by something other than the five-tuple. Some fabrics hash on the outer destination only, or use a routing scheme where the path is determined by the destination address and not by the flow. In that case no amount of source-port variation changes the path, and the fabric’s design, not the encapsulation, is the constraint.

Interaction with path MTU and with flow affinity

The source port is also the only field that lets an endpoint correlate an ICMP error from the underlay back to an inner flow. When the underlay reports that a packet was too large, it quotes the outer headers, including the source port, and the arithmetic behind that report is covered in tunnel encapsulation overhead and path MTU. If the port encodes a hash of the inner flow, the endpoint can recompute the hash for candidate flows and find a match, or it can keep a small table of recent hash values. Without the encoding, the error is unattributable and the endpoint can only react globally.

Flow affinity is the other side. Many fabrics deliberately keep a flow on one path for its lifetime, because reordering within a flow is expensive at the transport layer. The source-port scheme is compatible with affinity as long as the hash is stable for the life of the flow, which it is when the inner five-tuple does not change. A middlebox that rewrites the inner ports mid-flow breaks the affinity and can cause the flow to move to a different path, which is a reordering event the transport layer will treat as loss.

There is a subtle version of this with NAT. If the endpoint performs address translation on the inner packet before hashing, the hash uses the translated tuple, and the same inner flow can map to different outer ports at different times if the translation changes. Hashing before translation is the safer order, because it makes the outer port a function of what the sender chose rather than of what the endpoint rewrote.

What to measure, and what the numbers mean

The observable is the distribution of traffic across the underlay’s equal-cost paths. A fabric with eight paths between two endpoints should see roughly one eighth of the flow count on each, and the byte counts should be proportional to the flows’ sizes. A distribution where one path carries most of the bytes while the others carry most of the flows indicates that the hash is working for small flows and that one large flow is dominating, which is the single-flow case rather than a hashing failure.

The other observable is the number of distinct outer source ports in use between a pair of endpoints. If it is one, the endpoint is not hashing. If it is a small constant, the endpoint is hashing a field that does not vary. If it is large but the path distribution is still skewed, the underlay’s hash is reading a field the endpoint is not varying, or the fabric has fewer usable paths than its configuration suggests. The overlay and underlay split explains why the underlay is entitled to see only the outer headers, and therefore why the endpoint, not the fabric, must supply the variety the fabric needs.