Link Aggregation Member Selection and the Mismatches That Stall It

A link aggregation group presents one logical link built from several physical ones, and the two properties operators expect from it, more bandwidth and protection against a member failing, hold only within limits that the standard does not hide but that the configuration often does. Bandwidth adds across members only for traffic that the selection function spreads; resilience holds only for failures the control protocol can detect. Both properties are determined by decisions made per frame and per member, and neither is a property of the bundle as a whole.

The distributor and the selection function

The standard describes aggregation as a distributor that takes frames from the logical link and hands each to one member, plus a collector that takes frames from members and presents them as one link. The distributor’s choice is made by a selection function, and the standard deliberately leaves that function unspecified: any function that maps a frame to a member is conformant, provided the mapping is stable enough that frames of one conversation arrive in order.

Stability is the binding constraint, not load balance. Ethernet has no reordering mechanism, and higher layers treat reordering as loss. If two frames of one transport connection were sent on different members with different delays, the receiver would see them out of order and the sender would retransmit. The selection function therefore hashes on fields that identify a conversation, so that all frames of that conversation land on one member.

The fields hashed are typically the source and destination link-layer addresses, the source and destination network addresses, and for some implementations the transport ports. The set is configurable, and the configuration is per device, which means two ends of a bundle can hash differently without any protocol error. That asymmetry is legal and produces a bundle that works, forwards every frame, and distributes unevenly.

Why one flow cannot exceed one member

The consequence of hashing a conversation onto one member is that a single conversation’s throughput is capped at the capacity of one member, not the bundle. A bundle of four links carries up to four times the traffic of one link only if the traffic consists of many conversations whose hashes spread across all four.

The number of distinct conversations needed for good spread depends on the hash width and the field set. With a small field set, such as link-layer addresses only, a host pair that exchanges many transport connections still produces one hash value, because all the connections share the same address pair. A gateway serving many clients behind one router produces many hashes if the client addresses are included, and few if the hash covers only the router’s address and the gateway’s.

The practical result is a bundle that is congested on one member while the others are lightly loaded, with no error anywhere. The counters show it: one member’s utilization is high, the others low, and the sum is below the bundle’s capacity. Raising the bundle’s capacity does not help, because the bottleneck is the selection function’s granularity rather than the total.

Widening the hash field set is the usual remedy, and it has a cost. Including transport ports increases the number of distinct hashes, but it also means that a conversation is identified more finely, and a middlebox that rewrites ports, such as a NAT device, changes the hash mid-conversation and can reorder frames. The trade is between distribution quality and stability under header rewriting, and the right side of the trade depends on whether the path contains such a device.

Negotiation: what LACP establishes and what it does not

The control protocol for aggregation, defined in the same standard, exchanges information between the two ends so that both agree on which physical links belong to which bundle. Each end sends frames carrying a system identifier, a key that identifies the aggregation, and per-port state. A link becomes a member of a bundle when the two ends agree that their keys match and both are willing to aggregate.

The negotiation is what distinguishes a bundle that is deliberately configured from one that is not. A static bundle, configured identically on both ends with no control protocol, works as long as the configuration matches, and fails in a way that is hard to see when it does not: the two ends disagree about which links are bundled, and frames are distributed according to each end’s own idea, so a frame sent on member two at one end may arrive on member three at the other, which does not break anything by itself but does break the assumption that a member failure affects both directions equally.

The protocol’s convergence is slow by design. Frames are sent periodically, and a member is only considered usable after the exchange has completed, which means a bundle that is brought up takes on the order of seconds before it forwards. A member that is added to a working bundle also passes through the exchange before carrying traffic, and during that interval it is present in the interface list but not selected by the distributor.

Mismatches that leave partial forwarding

The failure modes that cost the most time are the ones where some traffic passes. Three are common.

A key mismatch produces a bundle where each end aggregates a different subset. If one end’s configuration gives all four ports the same key and the other gives two ports one key and two another, the two ends form bundles of different membership. Frames from the larger bundle may arrive on a port the other end does not consider a member of the corresponding bundle, and the receiving end’s collector delivers them, because the collector accepts frames on any member of the bundle it has formed. Traffic flows and the mismatch is invisible except in the per-member counters and in the protocol’s own state.

A speed or duplex mismatch among members is usually rejected by the negotiation, but a static bundle does not negotiate and accepts it. Mixing a faster and a slower member in a static bundle produces a hash that sends some conversations to a link that cannot keep up, and the result is loss for those conversations only. The signature is loss correlated with a specific flow rather than with load.

A one-sided bundle, where one end is configured for aggregation and the other is not, produces the most confusing behaviour. The aggregating end distributes frames across members; the non-aggregating end treats each port as an independent link, possibly with its own address or its own forwarding state. Frames arrive on ports the other end did not expect them on, and whether they are accepted depends on the other end’s configuration. Spanning tree protocols treat the members as separate links in this case and may block some of them, which reduces the bundle to a subset and produces loss that looks like a spanning tree problem.

Diagnosing without trusting the aggregate

The bundle’s aggregate counters hide the distribution, so the useful view is per member. Utilization per member, error counts per member, and the protocol’s per-port state together distinguish the three cases above: uneven utilization with clean errors is a hash granularity issue; errors on one member only is a member fault; and a member that is up but not selected is a negotiation or configuration mismatch. The per-member error counters are the ones whose meaning is least obvious, and interface drop counters sets out which discard point each of them records.

# per-member counters, and which members the kernel is actually using
ip -s link show bond0
cat /proc/net/bonding/bond0    # member state, and the active aggregator

The member state reported there distinguishes a link that is physically up from one that is usable by the bundle, and the distinction is exactly the one that matters. A member that is not part of the active aggregator carries nothing while the rest of the bundle is healthy, which is correct behaviour and also means its counters stay at zero.

The hash configuration should be read on both ends and compared, because the standard permits them to differ and nothing reports the difference. Where the field sets differ, the bundle is functioning and the distribution is not what either end’s configuration alone would suggest.

Where the mechanism stops

A bundle protects against member failure and against a cable or transceiver fault on a member. It does not protect against the failure of the device the members connect to, since all members terminate there, and it does not protect against a failure that affects all members simultaneously, such as a power loss or a software fault. Those require a redundant pair of devices, and the interaction between aggregation and the protocols that select between redundant devices is where the two mechanisms can defeat each other.

A bundle also multiplies the effect of a configuration error. A change applied to one member’s configuration, such as a change to the allowed set of tagged frames, affects only that member and produces loss for the conversations hashed onto it. The loss is a fraction of the traffic, intermittent by conversation, and the natural first hypothesis is a fault in the member rather than a configuration difference. Comparing the members’ configuration rather than their counters resolves it faster, and the general problem of changes that take effect on part of a bundle is the same one that confirmed commit exists to make reversible.