Load Balancer Session Affinity and Its Failure Modes

A pool of identical application servers is the normal answer to load, and a pool of identical servers is only identical if none of them holds anything the others do not. Sessions break that assumption. When the record for a visitor lives in one process’s heap, the request must reach that process, and the load balancer acquires a second job besides spreading work: remembering which server belongs to whom.

Affinity is a mechanism for that memory. It is worth separating from session storage, because affinity and storage solve the same problem at different layers, and the combination of both is what most deployments actually run.

Two places to put the decision

At layer 4, the balancer sees a TCP connection with a source address, a source port, a destination address, and a destination port. With no visibility into HTTP, the only stable key is the source address, so the mapping is a hash of the client IP — ip_hash in nginx, source hashing in most hardware balancers. It has the virtue of requiring no protocol parsing and the vice of being wrong in both directions.

Source-address hashing collides for anyone behind a shared egress. A corporate NAT, a mobile carrier’s CGNAT, or a school proxy presents thousands of clients as one address, so all of them land on one backend while the rest idle. It also breaks for clients whose address changes: mobile devices hand off between networks, and a new address means a new backend and a lost session. Source ports change per connection and cannot be used, since they are allocated per socket.

At layer 7, the balancer parses the request and can key on a cookie. Two variants exist. In passive mode the balancer reads a cookie the application already sets and hashes its value, which keeps the application unaware of the balancer. In insert mode the balancer appends its own cookie naming the selected backend, removing it from the request before forwarding. Insert mode requires no application changes but adds a second cookie that must be scoped and protected; a client that replays a stale balancer cookie to a pool that has since changed shape routes to a node that may no longer exist.

What the hash does when the pool changes

Modulo arithmetic is the obvious construction and the wrong one. Mapping a key $k$ to backend $h(k) \bmod N$ balances perfectly and reshuffles almost everything when $N$ changes: going from $N$ to $N+1$ moves roughly $N/(N+1)$ of all keys, which for a pool of ten is about 91 percent. Every one of those moves is a session that must be re-established somewhere it has never been.

Consistent hashing, described by Karger and colleagues in 1997, places both keys and nodes on a circular hash space and assigns each key to the first node clockwise. Adding or removing a node only moves the keys in the arc that node owns, so a pool of $N$ nodes loses about $1/N$ of its assignments when one node disappears — 10 percent at ten nodes, 2 percent at fifty. The plain formulation produces uneven arcs, so each physical node is represented by many virtual points on the ring, typically a few hundred; more points means a more even distribution and more memory in the balancer’s table.

Rendezvous hashing reaches a similar property differently: for each key, every node computes a hash of the key and its own identity, and the highest score wins. A node’s departure removes only its own candidacy, so the affected keys redistribute among the survivors with the same $1/N$ order. It needs no ring structure and handles arbitrary weights naturally, at the cost of $O(N)$ work per lookup instead of $O(\log N)$.

Maglev-style hashing, as used in large front-end fleets, precomputes a fixed-size lookup table from the backend set, so that every balancer derives the same table and makes the same decision for the same key without coordinating per request. The property that matters operationally is agreement: two balancers behind one anycast address must not disagree about which backend owns a key, or a client’s requests alternate between two session contexts.

The failure affinity cannot fix

Affinity reduces the probability that a request lands on the wrong node. It does not make the state durable, and a node loss is exactly the case where the state matters. With 10 nodes and one failure, roughly a tenth of sessions lose their records at once; on a busy site that is a visible wave of logouts, and it arrives precisely when the system is already degraded.

Replication inside the cluster trades that for coherence. The session record is written to two or three nodes and read from the owner, with the replicas promoted on failure. Correctness depends on the write having reached the replicas before the request completes, so the write path becomes a quorum write — more latency on every request that changes session state, in exchange for surviving a node loss. Hinted handoff, as described in the Dynamo paper, covers the case where a replica is unreachable at write time: the write is parked on a healthy node and delivered later, which means a read may briefly see the old value.

The simpler answer is to remove the local state entirely and put the record in a shared store, at which point affinity stops being necessary for correctness and remains only as a cache-locality optimization. That is the trade explored in Key Design and Eviction in External Session Stores.

Rolling deploys and the churn budget

Every deployment that replaces instances changes the pool shape twice: old instances leave and new ones arrive. With consistent hashing the newly added nodes claim their arcs immediately, so a fraction of sessions move to instances that have never seen them. If the application stores sessions in process memory, the move is fatal; if it stores them in a shared cache keyed by identifier, the move is free and the rebalancing is invisible.

Connection draining changes the shape of the event but not its existence. Removing a node from the ring before terminating it lets its keys migrate to the successor, which is correct for a shared store and useless for in-memory state. Draining does prevent a burst of failed requests mid-flight, and a drain period shorter than the client’s longest request leaves requests severed; the usual figure is a small multiple of the observed p99 request duration.

Weights interact badly with stickiness. Weighting a node to give it more traffic means its arc is larger, which is consistent, but a weight change is a pool-shape change for hashing purposes and remaps keys in proportion. Scaling by adding nodes is a smaller perturbation than scaling by changing weights on existing ones, even when the capacity change is the same.

Long-lived connections concentrate load

Affinity operates at connection setup, and HTTP keep-alive means a client reuses one connection for many requests. A browser that opens six connections pins those six to the same backend for as long as they live. With a client-side timeout of a few minutes, a burst of clients that connected during a load spike keeps sending to the backends chosen then, so the balancer cannot correct the imbalance until connections turn over. Shortening idle keep-alive timeouts increases rebalancing frequency at the cost of more TCP and TLS handshakes; a handshake with a full certificate exchange costs several round trips, while a resumed TLS 1.3 handshake costs one.

WebSocket and server-sent-event connections make this permanent by design. The connection is the session’s transport, so the backend is fixed for the connection’s lifetime, and a node restart terminates every conversation it hosted. Affinity is not a policy choice in that case; it is a property of the protocol. Recovery requires the client to reconnect and re-subscribe, which means the application needs a way to reattach to durable state on a different node.

Health checks and the retry that follows

A balancer that removes a backend on a failed health check will happily retry the request against another backend. For an idempotent GET that is correct. For a POST that already mutated state on the first backend before the connection broke, retrying executes the operation twice. Concurrent Mutations and Versioned Session Records covers the identifier-based side of that problem; the retry side belongs to idempotency keys and is a separate mechanism.

Health checks also interact with affinity when the check is shallower than the dependency. A node that passes its liveness probe while its session store connection is broken will keep receiving sticky traffic and fail every request that needs the record. Readiness checks that include the store round trip catch that state, at the cost of removing nodes during a store blip and concentrating their load on the survivors.

Choosing between affinity and shared state

Affinity is cheap: one hash per request, no additional network hop, no new failure domain. It is also a bet that nodes fail rarely and that losing a fraction of sessions is acceptable. Shared state is expensive: a lookup on every request, a new component with its own latency and availability, and a dependency that must be reasoned about during outages. It is the only option that makes a node’s failure unobservable to the visitor.

A middle position routes on a stable partition rather than a node: the token hashes to one of, say, 1024 partitions, and a membership service maps partitions to live owners. Scaling reassigns whole partitions, so a node addition moves a bounded set of them, and a node loss moves the partitions it owned to survivors. The indirection adds a lookup — cached, since membership changes rarely — and decouples the client-visible key from the physical topology, which means a resharding event does not have to be a mass logout.