Network Namespaces and veth Pairs as Virtual Routers

A router is a box with several interfaces, a forwarding table, and the ability to move a packet from one interface to another. Nothing in that description requires the interfaces to be physical or the box to be a separate machine. A Linux network namespace supplies the box: its own interfaces, its own addresses, its own routing table, its own ARP cache, its own firewall rules, and its own set of listening sockets. A veth pair supplies the wire between two boxes. Together they are enough to build an arbitrary topology out of nothing but processes and kernel objects.

What a namespace actually owns

A network namespace is a set of network stack state. Creating one gives a fresh copy of the parts of the stack that are per-namespace rather than per-system: the list of interfaces, the routing tables, the neighbour tables, the netfilter rule sets, the socket hash tables, and the ephemeral port allocator. A process that enters the namespace sees only that state. A process in the initial namespace sees none of it, and a socket created in one namespace cannot be used to send on an interface that exists only in another.

The separation is thorough enough that the same address can exist in two namespaces without conflict, and the same port number can be bound twice, once in each. That is what makes a namespace usable as a tenant container: the tenant’s view of the network is complete and self-consistent, and the host’s view is unaffected.

Interfaces are the exception to the isolation, because an interface has to exist in exactly one namespace at a time. An interface can be moved between namespaces, and moving it takes it out of the source and puts it into the destination in one operation. That move is the primitive on which every virtual topology is built.

Two namespaces can share nothing else. They cannot share a routing table, and there is no way to make a route visible in two namespaces at once. When two namespaces must exchange traffic, the exchange happens through an interface that exists in one of them and is connected to an interface that exists in the other.

The veth pair as a wire

A veth pair is two interfaces created together and connected: a frame transmitted on one is received on the other. The pair behaves like a length of cable with a plug at each end, and the two ends can be placed in different namespaces. Placing one end in namespace A and the other in namespace B creates a link between A and B, with no other effect.

The link is Ethernet, not a point-to-point IP tunnel. Each end has its own MAC address, generated when the pair is created, and each end maintains its own MTU, its own statistics counters, and its own flags. Because it is Ethernet, the ends can be put into a bridge, can carry VLAN tags, can participate in ARP, and can be enslaved to a bond. Because it is a pair rather than a shared medium, a frame sent by one end is always received by the other and never by a third party.

The kernel implements the pair as a direct handoff: the transmit path on one end calls the receive path on the other within the same system call, without a queue that another process can observe and without a copy through a socket. This is why a veth link is cheap compared with a userspace tunnel, and also why it cannot cross a machine boundary. It is a local construct.

A topology is then built by repetition. Two namespaces joined by a veth pair form a link. A namespace with two veth pairs, each going to a different neighbour, is a router if forwarding is enabled and routes are configured. A bridge inside a namespace joins several veth ends into a shared segment. A host namespace with a veth end going to each guest namespace, all enslaved to one bridge, is a switch. The same construction, with the bridge replaced by encapsulation, is the host-side half of the overlay and underlay split.

# two namespaces, one link, addresses on both ends
ip netns add left
ip netns add right
ip link add veth-left type veth peer name veth-right
ip link set veth-left netns left
ip link set veth-right netns right

ip -n left  addr add 10.0.1.1/24 dev veth-left
ip -n right addr add 10.0.1.2/24 dev veth-right
ip -n left  link set veth-left up
ip -n right link set veth-right up
ip -n left  link set lo up
ip -n right link set lo up

ip netns exec left ping -c 1 10.0.1.2

That sequence produces a working link in three commands that create it and three address or state commands per end. The lo interface is down by default in a new namespace, so a service bound to 127.0.0.1 inside the namespace accepts no connections until that interface is brought up.

Routing between two namespaces

To make one namespace a router, three things are needed: forwarding enabled in it, an address on each of its links, and a route telling it where the other side’s networks are. Forwarding is a per-namespace sysctl, so enabling it in one namespace does not affect the host.

The packet path is then: the source namespace emits a frame on its veth end, the router namespace receives it, the IP layer consults the router’s forwarding table, selects the outgoing interface, and transmits a new frame on the other veth end. The original Ethernet header is discarded and a new one is built, because the link is a different link. The IP header’s TTL is decremented, and if it reaches zero an ICMP time-exceeded is generated from the router’s address on the incoming interface.

Each of those steps is a normal part of the IP stack, executed in the namespace’s context. The consequence is that a virtual router built this way has the full behaviour of a real one, including ICMP errors, fragmentation handling, and the netfilter hooks. It is not a simplified forwarding path; it is the same code that forwards packets on a physical router, with the interface list replaced.

What the traversal costs

The cost of a namespace hop is the cost of the receive and transmit paths, not a copy through userspace. A frame that enters a namespace is processed by the same functions that process a frame from a physical NIC, and a frame that leaves is processed by the same transmit path. The saving compared with a physical link is the wire, the PHY, and the driver; the cost that remains is the protocol processing, the per-packet metadata allocation, and the table lookups.

Two costs are specific to the virtual construction. The first is that each namespace boundary is a scheduler-visible point in some configurations: a packet that must be queued because the receiving namespace’s backlog is full will be processed later, possibly on a different CPU, and the ordering guarantees that a single queue provides are weaker than a physical link’s. The second is that the number of namespaces multiplies the amount of state the kernel holds. Each namespace has its own conntrack table, its own neighbour and fragment state, and its own netfilter rules, and those structures are allocated whether or not the namespace carries traffic.

The scaling limit is therefore not the per-packet cost, which is modest, but the aggregate state and the number of hops. A topology that routes a packet through six namespaces performs six full forwarding passes, each with its own lookups and TTL decrement, and the total is roughly six times a single pass. Physical routers have the same property, which is why topologies are designed with few hops rather than many.

Where this model stops

A namespace is a complete network stack, so it can be a router, a host, a firewall, or a load balancer. What it cannot do is share a forwarding decision with another namespace. Two namespaces cannot consult one table, and a route installed in one is invisible to the other. Any topology that needs a shared control plane must therefore run that control plane as a process that configures each namespace through the kernel’s interface, rather than as state the kernel holds once.

The second limit is that the data plane is the kernel’s. A namespace’s forwarding path is the kernel’s IP stack, with its generality and its per-packet costs. Where the requirement is a large match-action table with many fields, or a program that rewrites headers in ways the standard stack does not express, the namespace is the wrong layer; the work moves instead to the programmable switches covered in match-action pipelines and flow caching.

The third limit is that the namespace boundary is a security boundary only if the process inside it is confined. A process with the ability to create namespaces and to move interfaces between them can reach any namespace on the host, which is why the privilege to do so is restricted. The isolation that a namespace provides is isolation of network state, not isolation of authority.