Moving Workloads with Overlapping Private Address Space

A migration that moves part of a system into a provider’s network usually encounters the same obstacle: the source and the destination both chose their addresses from the same private ranges. RFC 1918 reserves 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 for private use, and every organisation that adopted them early carved subnets out of the first of those. Two networks that both contain 10.20.0.0/16 cannot be connected by adding a route, because a router cannot distinguish the two copies of an address. Every packet to 10.20.1.5 is ambiguous, and the forwarding table has no way to express “the one over there”.

Why a route cannot disambiguate

A route maps a destination prefix to a next hop. If both networks contain the prefix, the more specific route wins, and specificity is the only tiebreaker available. There is no field in an IPv4 header that identifies which network the destination belongs to; the destination address is the whole of the information available to the forwarding decision.

This is a property of the addressing architecture rather than a limitation of any particular router. The address is an identifier of an interface, and routing assumes that identifiers are unique within the scope where routing happens. Two networks that use the same identifiers must be kept in separate scopes, or the identifiers must be rewritten at the boundary.

There are three ways out. Renumber one side, which is a project in itself and often impossible for a provider-managed network whose address plan is fixed. Translate at the boundary, which introduces state. Or avoid the direct connection entirely and reach the far side through an application-level proxy, which is translation with a different failure surface. In practice, translation is what gets used, and it is worth understanding precisely what it costs.

Translation is a stateful mapping, not a route

A translator that joins two overlapping networks rewrites addresses in the IP header and maintains a table of the mappings it has created. The table entry is what makes the return path work: a packet arriving from the far side with a rewritten destination is matched against the table, and its source is rewritten back. Without the entry, the return packet is unroutable.

Two mapping designs are used, and the choice determines what breaks.

A one-to-one static mapping assigns each address on one side a unique address on the other, out of a separate pool. It is stateless in the sense that the mapping does not depend on the connection, so a translator restart does not break established connections, and any number of translators can share the mapping if it is distributed. It requires a pool as large as the number of addresses that must be reachable, which is why it is used for the subset of hosts that need to cross, not for the whole network.

A dynamic mapping allocates an outside address and port when the first packet of a connection is translated, and frees it when the connection ends. It scales to far more inside hosts than there are outside addresses, at the cost of being stateful per connection. A translator that fails loses every mapping it held, and every connection through it breaks. This is the standard NAT behaviour that RFC 3022 describes, and the behavioural requirements in RFC 4787 exist because the details of when mappings are created, refreshed, and expired differ between implementations and matter to applications.

The port dimension is what makes dynamic translation possible at all. Two inside hosts can share one outside address if their connections are distinguished by the source port the translator assigns. This works for TCP and UDP because both have ports. It does not work for ICMP, which has no ports, and RFC 5508 addresses how ICMP queries are translated by embedding the identifier in the mapping. It does not work for protocols that carry addresses inside the payload, which is the subject of the next section.

The payload problem

NAT rewrites the IP header and, for TCP and UDP, the transport header. It does not parse the application payload. Any protocol that embeds an address or a port inside its payload therefore carries information that the translator cannot fix, and the far end receives an address that is not reachable from where it sits.

The classic cases are well known. FTP’s active mode sends the client’s address and port in a PORT command, so a client behind a translator announces an inside address that the server cannot reach. SIP carries addresses in its headers and in the session description, so a call set up through a translator describes endpoints that do not exist on the other side. Protocols that return a list of addresses in a response, such as a directory service or a cluster membership API, hand out inside addresses that the far side cannot use.

The general fix is an application-level gateway: the translator parses the payload, recognises the embedded address, and rewrites it. This works for a fixed set of protocols and fails for anything the translator does not understand, including any protocol whose payload is encrypted or whose encoding is not the one the gateway expects. The alternative fix is to avoid translation for that protocol by using a one-to-one mapping so that the announced address is valid on both sides, which is the reason static mappings are often reserved for the services with the most awkward protocols.

A migration plan should therefore enumerate, per protocol, whether the payload carries addressing. That enumeration is more useful than a list of ports, because the port tells the translator where to listen and the payload tells it whether listening is enough.

Asymmetric returns and the requirement that both directions traverse the translator

Translation assumes that both directions of a connection pass through the same translator, because the return path needs the state that the forward path created. When the two directions take different paths, the return packet arrives at a device with no mapping and is dropped or, worse, translated into something incorrect.

Asymmetric routing appears in migrations for predictable reasons. A destination network advertises a route for the translated prefix into the source network, and the source network’s own default route sends return traffic to a different exit. A workload that has one interface on the translated segment and one on the native network chooses its source address per connection, and the choice determines which path the return takes. A tunnel that carries only one direction of the traffic, which is a common misconfiguration when the tunnel is established from one side, produces exactly this.

The structural fix is to make the translator the only path between the two address spaces, so that no other route can carry traffic that would need translation. That means filtering: the source network must not have a route to the far side’s native addresses, and the far side must not have a route to the source’s native addresses. Where a route exists, some traffic will use it, and the failure will be intermittent and dependent on which host initiated.

A second, subtler requirement is that the translated prefix must not collide with anything else in either network’s routing table. A translator that maps 10.20.0.0/16 to 172.31.20.0/24 is only safe if 172.31.20.0/24 is unused on both sides. Choosing the mapping range is an address-planning exercise with the same constraints as the original plan, and it deserves the same care.

Path MTU, tunnels, and the fragments that disappear

Where the connection between the two networks is a tunnel, the effective MTU is reduced by the encapsulation overhead. An IP-in-IP tunnel removes 20 bytes; a GRE tunnel removes at least 24; an IPsec tunnel in transport mode removes more, and in tunnel mode more still. A 1500-byte path becomes a 1440-byte path or smaller.

Two behaviours follow. Packets larger than the reduced MTU must be fragmented or dropped with an ICMP “fragmentation needed” message, and the message must reach the sender for path MTU discovery to work. A translator or firewall that drops the ICMP message, which is common because the message is often filtered, leaves the sender retrying a packet that can never fit. The connection appears to hang on large transfers and work on small ones, which is a signature worth recognising.

The other behaviour is that fragmentation interacts badly with translation. A fragmented packet whose first fragment carries the transport header can be translated, but subsequent fragments carry no port information and cannot be matched to a mapping unless the translator reassembles or tracks fragments by identifier. Implementations differ, and RFC 4787’s requirements for fragment handling exist precisely because the differences cause failures that are hard to attribute.

The practical mitigation is to set the MTU on the translated path explicitly, or to clamp the maximum segment size for TCP connections that traverse it, so that no packet needs fragmentation in the first place. Clamping is a workaround with a cost: it reduces throughput per connection by the ratio of the clamped size to the path MTU. It is nevertheless preferable to a connection that stalls on large responses.

What the translation layer means for the migration plan

A translation layer is a component in the dependency graph whether or not it appears in anyone’s architecture diagram. It has its own availability, its own capacity limits in the form of mapping table size and translation throughput, and its own failure mode, which is that a restart breaks every connection in flight. It must therefore be treated as part of the cutover boundary, not as an implementation detail of the network.

Its lifetime is also a decision. Translation is a bridge that allows two address plans to coexist; it is not a permanent architecture, because every packet that crosses it pays a lookup and every protocol that embeds addresses pays a gateway. The migration plan should state when the translation is removed, which usually means when one side is renumbered or when the last workload that needs to cross has moved. A bridge with no removal date becomes a permanent dependency that nobody owns, and it is the component most likely to be discovered during decommissioning as the reason the old network cannot be shut down.