Load Balancer Listeners and Target Selection

A load balancer presents one address to clients and a set of addresses to itself. The interesting parts of its behaviour are all in the join between those two views: which client requests are accepted, which target receives each one, and what happens to a connection when the target behind it stops responding. None of these is a property of packet forwarding. They are properties of a proxy that holds two connections and translates between them.

A listener is a match on protocol and port

A listener is the front door. It is defined by a protocol, a port, and for encrypted protocols a certificate. A balancer with a listener on TCP port 443 accepts TCP connections there; one with a listener on HTTP port 80 accepts HTTP requests there and nothing else. There is no implicit wildcard: a balancer with no listener on a port refuses connections to that port, and the refusal is a TCP reset or a dropped SYN depending on the implementation rather than an HTTP error.

The distinction between a TCP listener and an HTTP listener is not cosmetic, and it determines everything downstream. A TCP listener forwards a byte stream. It does not parse the protocol inside, so it cannot route on a path, cannot add or remove headers, and cannot reuse a connection to the target for a different client. A TLS listener terminates the handshake at the balancer, which means the balancer holds the private key and the target sees plaintext or a second, separate TLS session depending on how the target is configured. An HTTP listener parses requests, which is what makes path-based routing and header rewriting possible, and which also means the balancer must enforce limits on request size and header count because it is buffering what it parses.

The ordering of listeners matters where protocols overlap. A balancer that terminates TLS on port 443 and also has a TCP listener on 443 is ambiguous, and the platform resolves it by refusing to create the second listener rather than by picking one. Where the ambiguity is not refused, the resolution is documented per platform and is not portable between them.

Choosing a target

For each new connection or request, the balancer selects one target from the registered set. The targets are private addresses inside the VPC, so the balancer reaches them the same way any interface would, through the routing described in route table precedence and the local route. The algorithm is one of a small number, and the choice has consequences that are larger than they appear.

Round robin walks the set in order. It assumes every target can handle the same load, which is true when targets are identical and false the moment one is slower, because a slower target accumulates connections and its share of the work grows rather than shrinks. Least outstanding requests sends to the target with the fewest connections currently open, which adapts to slow targets at the cost of tracking a counter per target and of being biased when request durations are heavy-tailed. A hash of the client address or of a header gives affinity, at the cost of an uneven distribution when the hash space is smaller than the set, and at the cost of moving a client to a different target whenever the set changes.

The algorithm operates on targets that are considered healthy, and that set is not the same as the registered set. A target that fails its health check is removed from selection without being deregistered, so the set of candidates changes continuously and independently of configuration. This is the mechanism that makes a rolling replacement work: a new target is registered, passes its check, receives traffic, and an old one is deregistered, fails its check, and stops receiving it, with the balancer never sending a request to a target that is not ready.

Health checks are themselves connections. A check that opens a TCP connection every few seconds is a small but nonzero load on every target, and a check that performs an HTTP request is a larger one. The check’s definition of healthy is also its definition of the target: a target that returns 200 on the check path but fails every real request is considered healthy, because the balancer has no other information. This is the single most common source of a balancer sending traffic to a broken target, and it is not a bug in the balancer.

Two connections, not one

The balancer does not forward packets. It terminates the client connection, and it opens a separate connection to the target. Between them it holds two state machines and copies bytes.

This is the decision that defines the whole mechanism, and it is worth stating what it buys and what it costs. Terminating means the balancer can inspect and rewrite, which is what enables path routing, header insertion, and protocol upgrades. It also means the client’s connection is with the balancer, so the client’s view of the target’s identity is whatever the balancer chooses to present. The target sees the balancer as its peer, not the client, and the client address survives only if the balancer inserts it into a header or if the protocol carries it separately.

The cost is that the balancer is in the data path for every byte, and it must buffer. An HTTP listener parses a request before choosing a target, which means the request headers are held in memory at the balancer; the size limit on those headers is therefore a property of the balancer, not of the application, and a request that the application would accept can be rejected at the balancer with an HTTP 431 or an equivalent. The limit is documented per balancer and is measured in tens of kilobytes, which is generous for ordinary requests and insufficient for designs that carry authentication assertions or routing metadata in headers.

client ──TCP──▶ [ balancer: listener 443, TLS terminated ]
                        │
                        ├─ selects target T2 (healthy, fewest outstanding)
                        │
                        └──TCP──▶ [ T2: 10.20.4.17:8080, plaintext ]

The second connection is a full TCP connection with its own handshake, its own retransmissions, and its own state. When the client disconnects, the balancer may keep the target connection open for reuse or close it, depending on the listener type. A TCP listener cannot reuse anything: the connection is a byte stream with no message boundaries, so its lifetime is the client’s lifetime. An HTTP listener can keep the target connection alive across requests, which reduces handshake cost but introduces the possibility that a target closes a reused connection at the exact moment the balancer sends a request on it, producing a failure that the client sees as a reset on an otherwise healthy request.

Client address and the proxy protocol

Because the target’s peer is the balancer, the source address in the packet is the balancer’s. Applications that log client addresses or make decisions based on them need the original address conveyed some other way.

For HTTP, the conventional mechanism is a forwarding header. The balancer appends the client address to a header that the request carries, and the application reads the first or last entry depending on how many proxies are in the chain. The ordering is the security-sensitive part: a client can send its own forwarding header, and an application that trusts the first entry trusts a value the client chose. The correct reading is from the trusted end, and the number of trusted hops must be known. RFC 7239 defines a header for this purpose with an explicit syntax for multiple hops; the older convention, which carries the addresses as a comma-separated list without structure, is what most applications parse, and it is the one that is easiest to spoof.

For TCP, there is no header to append, so the balancer prepends a small header to the connection itself before the application’s bytes. The target must be configured to expect it, and the two sides must agree on whether it is present; a target that expects it and receives ordinary bytes misparses the first line of the request, and a target that does not expect it and receives it sees garbage at the start of the stream. This is a per-target configuration, and a mixed set of targets where some expect the header and some do not produces failures that depend on which target was selected.

Some balancers support a transparent mode in which the client address is preserved in the packet and the target’s reply is routed back through the balancer. That requires the balancer to be on the return path, which in a virtual network means the target’s default route points at the balancer or the target’s address space is manipulated. The mechanism is more expensive to operate and is used when the application cannot be changed. It is also distinct from address translation: a balancer that rewrites the source address is performing a translation of its own, and the return path then depends on that translation’s state rather than on routing, the same dependency internet gateway, NAT, and source address translation describes for a NAT gateway.

What happens when a target fails

The failure behaviour is where the two-connection model becomes visible.

If a target stops responding, the balancer’s connection to it times out. The client’s connection is still open, and the balancer must decide whether to close it or to retry the request on another target. For an idempotent request that has not been sent, retrying is safe and is usually automatic. For a request that was sent and partially processed, retrying can duplicate a side effect, which is why retry behaviour is a per-listener setting rather than a universal default.

If a target fails its health check, it is removed from selection, but existing connections are not redirected. A client whose connection was established before the removal continues to be served by that target until the connection ends or the target stops responding. For a long-lived connection, this means a failed target can keep receiving traffic on its existing connections for as long as they last, which is why a rolling replacement must account for connection draining rather than relying on the health check alone.

If the balancer itself fails, the consequence depends on how many nodes it has. A balancer is a managed resource with a domain name and one or more addresses, and the addresses are not the same across zones. A client that resolved the name to an address in a zone that fails continues to send to an address that no longer answers, and DNS caching extends the failure beyond the zone’s own duration. Reducing the record’s time to live limits that window but does not eliminate it, because resolvers are free to cache longer than the record says. This is the reason a balancer’s availability is a property of its addressing as much as of its capacity, and it is why the address family and the zone placement of the balancer are architectural decisions rather than defaults.

The limit that is not about bandwidth

Balancers are commonly described in terms of requests or connections per second, and those limits are real, but the one that surprises designs is the connection count. Because the balancer holds both sides of every connection, its state is roughly double the client-side count, and the per-node limit on concurrent connections is reached before the bandwidth limit in workloads that hold many long-lived connections open.

A workload that opens a connection per client and keeps it open for the duration of a session, such as a websocket service, consumes balancer state proportional to concurrent users rather than to request rate. The remedy is either a balancer sized for connection count or a design that does not require the balancer to hold the connection for the session’s lifetime, and the second is usually a larger change than the first.