Link Aggregation and the Hash That Chooses a Member

Several physical links between two devices can carry more traffic than one, if the traffic is spread across them. The spreading is done per frame, at the transmitter, with no per-flow state and no negotiation about which link carries what. That constraint, no state, is what makes the mechanism cheap enough to run in hardware at wire rate, and it is also why the distribution can be uneven: the hash decides, and the hash does not know how much data each flow will send.

The bundle as one logical link

An aggregation presents a single logical interface with the combined capacity of its members. The individual links are still separate at the physical layer, each with its own transceivers and its own autonegotiation, and each still carries ordinary Ethernet frames with unmodified headers. Nothing in the frame indicates which member it travelled on, so the receiving device must be able to accept a frame on any member and treat it as having arrived on the aggregate.

The reason for the bundle is not only capacity. A bundle of two links survives the loss of one, and the aggregate’s capacity drops to the surviving member’s rather than to zero. The convergence is not instantaneous: the protocol must notice that a member has stopped responding, which takes a few times the protocol’s timer interval, and during that interval frames hashed to the failed member are lost. The protocol’s timer and the resulting detection time are the practical limit on how quickly the aggregate heals.

The control protocol and what it negotiates

Members can be bundled by static configuration, in which both ends are told which links belong together, or by a control protocol that verifies the far end agrees. The protocol is LACP, defined alongside the aggregation standard. Each end sends frames to a reserved group address 01:80:c2:00:00:02, carrying a system identifier, an operational key, the sender’s port identifier and state, and the sender’s view of the neighbour’s corresponding values.

The key is the important field. Ports with the same key and the same system identifier, as seen by the far end, may be aggregated. A mismatch in key, in system identifier, or in the port’s operational state causes the port to be left out of the bundle rather than aggregated, which prevents two links that are configured differently from being combined into something neither end expects.

LACP’s timers are short and long: a fast interval of one second and a slow interval of thirty seconds, negotiated by a flag in the protocol frame. The short interval detects a failed member faster at the cost of more protocol traffic; the long interval is the default on many implementations. There is also a distinction between active and passive mode: an active port initiates the protocol, a passive one only responds. Two passive ports never form a bundle, because neither speaks first. The failure is silent in the sense that the links come up and are simply not aggregated, with no error reported.

Why distribution must be a hash

The distributor chooses a member for each frame. It could choose round-robin, which would spread frames evenly by count, but round-robin reorders frames within a conversation: two frames sent back to back can take different links, and if the links have different latencies, they arrive out of order. Ethernet has no reordering mechanism and TCP treats reordering as loss, so round-robin across members would degrade every connection on the bundle.

The standard therefore requires that frames belonging to one conversation stay on one member. The conversation is identified by a hash over fields in the frame, and the member is selected by the hash value. Because the hash is computed from the same fields for every frame of a conversation, the choice is stable without any state.

The fields hashed are implementation-defined, and the choice determines both how evenly traffic spreads and whether the spread is stable. Common selections are:

• Source and destination MAC addresses only, the minimum required.
• Source and destination MAC plus the EtherType.
• The above plus the IP source and destination addresses.
• The above plus the transport source and destination ports.

Each additional field improves the spread across conversations but also changes the hash when the field changes, which can move a conversation to another member mid-stream. Including the transport ports means a single host pair with many connections spreads across members, which is what a server with a large number of clients wants. Excluding them means all traffic between two hosts stays on one link, which is what a router carrying many host pairs wants, because the host pairs already provide the diversity.

The hash function itself is usually a CRC or an XOR-fold of the selected fields, truncated to the number of bits needed to index the member table. The truncation matters: a two-member bundle needs one bit, so half the hash space goes to each member, and the distribution is as even as the hash is. A three-member bundle needs a modulo-three step or a table lookup, and a poor hash function can produce a systematic bias where one member carries noticeably more than another.

/* Member selection over the five-tuple: hash the flow, fold, then index. */
static uint32_t crc32_le(uint32_t crc, const uint8_t *p, size_t n)
{
    while (n--) {
        crc ^= *p++;
        for (int i = 0; i < 8; i++)
            crc = (crc & 1u) ? (crc >> 1) ^ 0xEDB88320u : (crc >> 1);
    }
    return crc;
}

static unsigned member_for(const struct flow_key *k, unsigned n)
{
    uint32_t h = 0xFFFFFFFFu;
    h = crc32_le(h, k->src_ip, 4);
    h = crc32_le(h, k->dst_ip, 4);
    h = crc32_le(h, (const uint8_t *)&k->src_port, 2);
    h = crc32_le(h, (const uint8_t *)&k->dst_port, 2);
    h = crc32_le(h, &k->proto, 1);
    h ^= h >> 16;              /* fold so that a small n sees every bit */
    return h % n;
}

The fold before the modulo is not cosmetic. Without it, a hash whose variation is concentrated in the high bits produces a nearly constant index when the member count is small, and the bundle behaves as if it had one member.

The single-flow ceiling

Because one conversation maps to one member, a single TCP connection between two hosts cannot exceed the capacity of one physical link. A bundle of eight 10 Gbit/s links has 80 Gbit/s of aggregate capacity, and a single transfer across it runs at 10 Gbit/s. This is not a bug and no configuration fixes it; it is the ordering guarantee’s price.

The practical consequence is that an aggregate’s benefit depends on having many conversations. A server with thousands of clients spreads well. A backup appliance replicating to a single peer over one connection does not, and adding members to the bundle will not speed it up. The measurement that reveals this is per-member counters: a bundle whose members show very unequal byte counts is either hashing poorly or carrying a small number of large flows.

The reverse case is a bundle with many small flows that hash unevenly because the hash has few bits of entropy in the fields selected. Consider traffic between a host and its default router: every frame on that link carries the same MAC pair, whatever the host’s destinations and however many connections it opens. A hash over MAC addresses alone therefore places all of that host’s off-subnet traffic on one member. Including the IP addresses separates the conversations and spreads them; a hash restricted to MACs has nothing else to work with.

Failure detection and the residual risk

A member is removed from the bundle when the protocol stops hearing from it, or when the physical layer reports loss of signal. The second is faster and more reliable when it is available, because it is a hardware indication rather than a timeout. A link that is physically up but not passing frames, which happens with a failing optic or a misbehaving intermediate device, is only detected by the protocol timeout.

Frames hashed to a member that has failed but not yet been removed are lost. They are not retransmitted by the link layer, so the loss surfaces at the transport. For TCP this is ordinary loss and is recovered, at the cost of a congestion response that reduces the sending rate even though the path’s capacity has only dropped by one member’s worth. For UDP it is simple loss with no recovery.

There is a subtler failure: a member that is passing frames in one direction only. The protocol’s frames may travel in the working direction and be lost in the broken one, so the far end’s view and the near end’s view disagree. The standard’s response is that a port must be able to receive as well as transmit to be aggregable, and an implementation that checks only that it can transmit will keep a half-broken link in the bundle indefinitely.

Interaction with the rest of the link layer

The bundle is a single link from the perspective of everything above it, so the bridge treats it as one port. Learning records the aggregate’s port, not the member, which means a station’s address maps to the bundle and frames for it may leave on any member. That is consistent with the hash: the bridge’s decision is per destination, and the distributor’s is per conversation, and the two are independent.

The aggregate carries one MTU, and implementations apply it to the members, so the value is set on the bundle rather than on an individual link. A member that rejects the aggregate’s frame size drops the frames the hash sends it, and the distributor, which selects on the hash alone, has no way to know. The configuration requirement is described in link MTU and jumbo frames.

A bundle between two bridges is also a single edge in the spanning tree graph, so the tree does not block individual members and the aggregate’s full capacity is available to the tree. This is the main reason aggregation is preferred over adding parallel links that the tree would block. The tree’s treatment of ports is covered in spanning tree port states, and the per-port queues that a priority-tagged frame is placed in are described in 802.1Q VLAN tagging.