Draining a Router for Maintenance Without Dropping In-Flight Traffic

Taking a router out of service is a change to the control plane that must complete before the data plane changes. If the device is powered off while it is still the best path for some destinations, the traffic it was carrying is dropped at the moment of removal, and every conversation through it fails. If it is drained first, the control plane advertises a worse path, the network converges onto that path, and the device becomes idle before it is touched. The interval between those two events is the whole technique, and getting it right requires estimating convergence rather than guessing at it.

Two mechanisms: advertise worse, or withdraw

The control plane offers two ways to make a router stop being selected. The first is withdrawal: the device stops advertising the destinations it originated or the routes it learned, and its neighbours remove it from their tables. The second is cost inflation: the device continues advertising the same destinations with a metric so high that no other path loses to it, so neighbours keep the routes but select alternatives.

Withdrawal is simpler to reason about and more disruptive when it goes wrong. A withdrawal propagates as an update, and the receiving routers must recompute their best paths. During that recomputation, some routers may briefly have no path at all, which is a transient blackhole even though the final state is correct. Whether that happens depends on whether the withdrawn route was the only one known, and on whether the protocol’s diffusing computation holds the route until alternatives are confirmed.

Inflation avoids the blackhole in the common case because the route remains present throughout. A router that receives an inflated route still has an entry for the destination; it simply prefers a different one. There is no moment at which the destination is unreachable because a route disappeared. The cost is that the inflated route stays in the tables, so the device remains a candidate path for anything that loses its preferred path during the maintenance window, which is a hazard if the device is about to be powered off.

Most operational practice combines the two: inflate first to shift traffic away without creating a gap, wait for the shift to complete, then withdraw or shut down. The order matters because it separates the two failure modes: the shift is the risky step and it happens while the device is still forwarding, and the removal happens after the device carries nothing.

Metric inflation in a link-state interior protocol

In a link-state protocol, the metric is carried in the router’s own advertisement of its links. Raising the metric on all of them makes the router a poor transit node while leaving it reachable, which is what a maintenance drain needs. The standard mechanism for this is an advertisement of the router’s links with the maximum representable metric, which the protocol specification defines specifically so that a router can be taken out of transit without being removed from the topology.

The important property is that the router remains in the topology and remains reachable as a destination. Its own interface addresses stay advertised, so management access through the network continues to work, and the router can still originate and receive traffic for its own addresses. Only transit through it is discouraged.

The mechanism’s effect is not instantaneous. The advertisement must be generated, flooded to every router in the area, and processed. Flooding is bounded by the topology: a change reaches every router within a number of hops, and each hop adds processing and propagation delay. On a large area the interval is on the order of seconds, and it is longer when the change is large, because a router that must recompute its shortest-path tree does more work for a change that affects many entries.

A second effect extends the interval further. Routers that were using the drained device recompute, and their recomputation may change their own advertisements, which propagates as a second wave. The waves settle, but the settling takes longer than the first flood, and the drain interval must cover the settling rather than only the first propagation.

What happens to traffic already in flight

A packet that is in transit through the drained router when the change takes effect is not affected by the change. It was forwarded according to the state the router held when it arrived, and it continues on its way. The drain moves future traffic; it does not move traffic already queued or already on the wire.

This is the reason the drain interval has a lower bound that has nothing to do with convergence. A router with deep queues holds packets for the time it takes to drain them, and a device under load can hold packets for a noticeable interval. The interval is bounded by the queue depth divided by the departure rate, which for a congested interface can be substantial. The drain must allow those queues to empty before the device is disturbed, and the queue depth is observable in the interface counters described in interface drop counters.

Transport connections add a second in-flight consideration. A connection that was using the drained path and is moved to a new path experiences a change in round-trip time and possibly a change in the maximum transmission unit. A path with a smaller MTU produces blackholing for packets above the new size unless discovery runs, and discovery takes at least a round trip. The connection survives, at the cost of a retransmission and a reduction in its congestion window. This is unavoidable and is the reason draining is preferable to an abrupt removal: the cost is a window reduction rather than a connection reset.

The exterior protocol case and the shutdown community

For routes exchanged with external peers, the mechanism is different because the metric is not a link cost but a path attribute sequence. Raising the local preference or prepending the autonomous system number makes the path less attractive to the peer’s decision process, and prepending is the conventional way to shift traffic without withdrawing.

A more precise mechanism exists for the specific case of taking a session down. A well-known community is defined for graceful shutdown, and a router that attaches it to its advertisements signals to its peers that it is going out of service. Peers that understand the community apply their configured policy for that signal, which is typically to deprefer the path or to treat the routes as withdrawn, and peers that do not understand it treat the routes normally. The community is a signal rather than an instruction: its effect depends entirely on the receiving peer’s policy, so a drain that relies on it must be verified at the peers rather than assumed from the local configuration.

The standard also defines a corresponding behaviour for the session itself. Rather than tearing down the transport connection abruptly, which forces the peer to withdraw every route learned from it at once and reconverge, a graceful shutdown keeps the paths present while the peer deprefers them and converges, and closes the session only afterwards. The distinction matters because an abrupt close produces a burst of withdrawals that the peer processes as a single event, while the graceful sequence delivers a policy-driven change that propagates like any other attribute update.

Graceful restart, specified separately for BGP, covers the narrower case of a control-plane restart rather than a removal: the forwarding state is preserved and the peer keeps using the routes it already holds while the session is re-established.

Estimating the interval instead of guessing

The drain interval should be derived from three components, each measurable.

The first is the time for the control-plane change to reach every router that must act on it. This is bounded by the flooding and processing time of the interior protocol, which can be observed directly: the change is made, and the local and remote routing tables are checked for the absence of the device from the transit paths. The interval from the change to the last router showing the new state is the propagation time, and it is measurable without disturbing anything.

The second is the time for the traffic to move. Traffic moves when the routers’ forwarding tables change, not when their routing tables change, and the two are separated by whatever delay the implementation applies before installing a new best path. Observing the drained device’s own interface counters is the direct measurement: the counters fall as traffic moves, and the interval from the control-plane change to the counters reaching a low value is the shift time.

The third is the time for the queues to drain after the traffic stops arriving, which is bounded by the queue depth at the moment of the last arrival.

# Watch the drained device's egress rate fall as the change propagates.
# Sample the byte counters on the transit interfaces once per second.
prev=0
while :; do
  cur=$(cat /sys/class/net/eth0/statistics/tx_bytes)
  echo $((cur - prev)); prev=$cur
  sleep 1
done

The loop is deliberately crude: the point is that the drain is confirmed by the device’s own counters reaching a steady low value, not by a timer expiring. A fixed interval chosen in advance is either longer than necessary, which extends the maintenance window, or shorter, which drops traffic.

The failure that the technique cannot prevent

Draining protects against the device being removed while it is a selected path. It does not protect against the device failing while it is drained, which is the same as any other failure. Nor does it protect against a network where the alternative path is worse than the drain assumed: if the only alternative is a link that is already near capacity, shifting traffic onto it produces congestion, and the drain has converted a maintenance event into an outage on a different device.

That risk is estimated before the drain, not after. The alternative path’s headroom must exceed the traffic being moved, and the estimate of that traffic comes from the drained device’s counters rather than from a flow-based figure, because the counter is exact and the sampled figure is not. The difference between the two measurements, and the reason the counter is the right one for a capacity decision, is covered in flow sampling error.

The final step is the one most often skipped: verifying after the maintenance that the device is again a selected path and that the traffic has returned. A device that comes back with a configuration change, or with an interface that did not come up, is a device that is present in the topology and not carrying what it carried before. The same counters that confirmed the drain confirm the return. If the maintenance included a configuration change, applying it under a confirmation window that reverts on failure bounds the damage in a way that a manual rollback does not, which is what confirmed commit and automatic rollback provides.