BGP Sessions and Route Propagation over Hybrid Links

A dedicated link between a customer’s network and a provider’s region carries traffic, but the harder problem is deciding which traffic should use it. The link has an address range on each side and a set of prefixes that should be reachable across it, and those prefixes change: a new subnet appears, a prefix is withdrawn when a site is decommissioned, a backup path becomes preferred when the primary fails. Carrying that information by hand means editing route tables at both ends for every change. Running BGP over the link means each side tells the other what it can reach, and the change propagates.

The session and what it exchanges

A BGP session is a TCP connection on port 179 between two routers, each identified by an autonomous system number, which is two bytes in the original encoding and four bytes in the extended one. The session is established by an OPEN message that carries the local AS number, a hold time, and a BGP identifier; each side sends one, and the session reaches Established when both have been accepted.

After establishment, the session carries UPDATE messages. An UPDATE advertises a set of prefixes that share a set of path attributes, and withdraws prefixes that are no longer reachable. Nothing is exchanged periodically: the session is incremental, and the only regular traffic is a KEEPALIVE at roughly a third of the negotiated hold time. A hold time of 90 seconds with a keepalive of 30 is a common pairing, and the choice of hold time is a direct trade between detection speed and sensitivity to transient loss, since a session that misses keepalives for the hold time is torn down and every route learned from it is withdrawn.

The routes learned over the session are candidates, not decisions. Each is entered into the routing table with its attributes, and the selection process compares candidates for the same prefix. The result is one best route per prefix, which is what gets installed and what gets re-advertised to other sessions.

The attributes that decide

BGP’s selection is a fixed sequence of comparisons, applied in order, and the sequence is standardised. For routes to the same prefix, the first comparison is local preference, a value local to the receiving router and not carried between autonomous systems in the ordinary case. Higher local preference wins. Because it is local, it is the mechanism by which an operator says “prefer the dedicated link over the internet path” without affecting what anyone else does.

Next is the AS path length: the route whose path attribute lists fewer autonomous systems wins. The path is a list of AS numbers, each router prepends its own on advertisement, and it serves two purposes at once. It is a loop detector, since a router that sees its own AS number in the path rejects the route, and it is also a crude distance metric. It is crude because it counts hops between autonomous systems, not hops between routers, so a path through two large transit providers looks shorter than a path through four small ones regardless of the actual latency. Prepending is the standard way to make a path look longer: an operator advertises the same prefix with its own AS number repeated, which biases remote selection away from that path without withdrawing it.

Then come origin type, multi-exit discriminator, and finally the peer address as a deterministic tie-break. The multi-exit discriminator is the only attribute designed to be compared between two sessions to the same neighbour AS, and it is the correct tool for preferring one of two links to the same provider. Using AS path prepending for that purpose works but leaks the preference to every other network that sees the advertisement.

One attribute carries no ranking at all and is still the most useful for steering: the community. A community is a tag attached to a route, carried between autonomous systems, and meaningless to BGP itself. Its value comes entirely from what the receiving network does with it, which is why communities are how a customer tells a provider’s routing policy what to do without either side agreeing on anything beyond the tag’s meaning. RFC 1997 defines the attribute and the well-known values; the private ranges are what an operator uses to encode its own intent.

# Session state progression, as reported by a router.
Idle -> Connect -> Active -> OpenSent -> OpenConfirm -> Established
# Established is the only state in which routes are exchanged.
# Active means the router is retrying the TCP connection, not that it works.

The last line of that sequence is the one operators misread. A session in Active is not functioning; it is the state of a router that has failed to connect and is waiting to try again. A monitoring system that alerts on “not Established” is correct, and one that alerts on “Active” alone is reporting a symptom of a connection failure rather than the failure itself.

What the provider side does with the routes

The routes learned over a hybrid attachment are not installed into a single VPC’s table. They are installed into the transit gateway’s table, and from there they are propagated to the attachments that the table is associated with. This is the mechanism by which a customer’s on-premises prefixes become reachable from every VPC attached to that table, without any per-VPC configuration. The table’s role as the single place where policy is held is the same one described in why VPC peering is not transitive and what a transit gateway adds, extended from VPC-to-VPC routing to routing learned from outside the provider.

Propagation is not automatic in both directions. The gateway advertises the VPC prefixes to the customer over the session, but which prefixes it advertises depends on the table’s configuration, and a VPC that is not associated with the table is not advertised. The asymmetry is deliberate: the customer’s routes arrive as candidates and are filtered by the table’s rules, while the provider’s routes leave according to what the table contains. A design that expects a new VPC to be reachable from on-premises as soon as it is created will find that it is not, because the VPC’s attachment has to be associated with a table that the session is attached to.

RFC 8212 states the requirement that route propagation be governed by explicit policy rather than by implicit defaults. The practical form of that requirement is that a session should have an inbound filter and an outbound filter, and that neither should be “permit everything” by omission. An unfiltered inbound session accepts whatever the peer sends, including a default route or a prefix the peer does not own, and an unfiltered outbound session advertises the provider’s entire table to the customer.

The failure that propagates

A BGP session that flaps, meaning it repeatedly establishes and fails, does more damage than a session that is simply down. Each transition from Established to Idle causes the routes learned from that session to be withdrawn, and each return causes them to be advertised again. Every withdrawal and re-advertisement is a change that propagates to every table the routes reached, and a topology with several thousand prefixes and a flapping session generates a continuous stream of updates.

The standard defence is route dampening, which penalises a prefix each time it is withdrawn and suppresses it once the penalty exceeds a threshold, with the penalty decaying over time. The mechanism reduces the update churn at the cost of keeping a prefix suppressed after the underlying link has recovered, which is a real availability cost: the link is up and traffic is not using it. Dampening is therefore not universally enabled, and where it is disabled the churn reaches every router that receives the routes.

The second failure is subtler and more damaging. A session that stays Established but stops receiving updates leaves the router believing routes that the peer has withdrawn. Because BGP is incremental, there is no periodic refresh that would correct this; the hold timer only detects a dead peer, not a peer that is alive and silent. A router that has learned a route and never receives the withdrawal will keep advertising it, and traffic will be sent to a destination that is no longer reachable, with the failure appearing as a black hole somewhere past the provider’s edge.

Where the session meets the rest of the model

Two interactions are worth stating because they are not obvious from the BGP specification.

The first is with the local route. A prefix learned over a session and a prefix that is part of a VPC’s own address space compete in the gateway’s table, and the provider’s local knowledge of its own VPCs takes precedence over anything learned. A customer that advertises a prefix overlapping a VPC’s prefix will find that the advertisement is not honoured for the overlapping portion, and the traffic to those addresses goes to the VPC. Address planning therefore has to avoid overlap with the provider’s allocations, which is the same constraint peering imposes, arriving through a different mechanism.

The second is with the endpoint model. A private endpoint exists precisely so that a consumer does not need a route to the provider’s prefix, and a hybrid session exists to create exactly such routes. The two are complementary rather than competing: the session carries bulk connectivity to a customer network, while endpoints carry per-service access without route exchange. A topology that uses sessions for everything ends up advertising prefixes that overlap, and a topology that uses endpoints for everything cannot reach the customer’s own hosts. Private endpoint services and the reverse-direction connection describes the case where the address spaces are the reason the session cannot be used.

Reading the table

The final point is that a hybrid route table is a record of policy, not of topology. Two paths to the same prefix may both exist, and the installed one is the winner of a comparison that involved local preference, path length, and attributes that were set deliberately at one end or the other. When traffic takes an unexpected path, the question is not which link is up but which attribute made the difference, and the answer is usually a local preference that was set once and forgotten, or a prepend that was added to steer one prefix and affects every prefix advertised with it.

That is why the configuration of a session is best read as a small program: the filters determine the input, the attribute settings determine the ranking, and the selection algorithm is fixed. Everything the network does with the routes follows from those three, and nothing about the physical links changes the outcome unless the session state changes first.