Node Distance and the Real Cost of a Remote Memory Access

A load that misses the last-level cache either reaches a DRAM controller on the same socket or one on the other socket. Both complete, and on a machine with one socket the distinction does not exist. On a two-socket server it is the difference between a request that stays inside a package and a request that crosses a link, occupies it in both directions, and is answered by a memory controller belonging to a different set of cores. Nothing in the instruction set signals which case a program is in. The address alone decides.

That is the whole premise of non-uniform memory access: the physical address space is partitioned among nodes, each node owns the DRAM attached to it, and any core may read any address. Uniformity of access is preserved; uniformity of cost is not.

The transaction a remote load becomes

Each node owns a distinct range of the physical address space, described by firmware and reflected in the node’s memory zones. A given physical address therefore has exactly one home node for the lifetime of the machine’s configuration, and every core in the system can name that node from the address alone. Within a node, the address is further hashed across the node’s own memory controllers and cache slices, so consecutive physical pages inside one node are spread over that node’s channels; that hashing never crosses a node boundary, because crossing it is precisely what makes an access remote.

A local miss travels from the core, through the private caches, to the portion of the uncore that owns the address, and out to a DRAM channel. A remote miss travels the same path until it reaches the socket boundary, where the request crosses the link to the socket that owns the address. On Intel servers that link is Ultra Path Interconnect; on AMD servers it is Infinity Fabric. Both are cache-coherent, both carry a request in one direction and a 64-byte data line in the other, and both have finite bandwidth shared by all traffic between the two endpoints.

Coherence adds a second, less obvious cost. The home node keeps a directory recording which caches hold each line it owns. When a core on node 1 writes a line homed on node 0, the request must reach node 0, node 0 must invalidate any other sharers, and the acknowledgment must come back. If the line is cached by a core on node 2, the invalidation is forwarded there and its response crosses the fabric again. A line that is shared between sockets therefore costs interconnect messages on every write, even though no DRAM access ever occurs. This is why remote sharing of a hot cache line can be worse than remote reading of cold memory: the traffic is proportional to the number of writes, not to the number of lines.

Distance is a rank, not a duration

Firmware publishes the topology in two ACPI tables, and the one that matters for cost is the System Locality Distance Information Table. It is a square matrix of bytes, one row and column per proximity domain, where the value 10 means local and larger values mean further away. The specification treats 10 as local, reserves values below it, and uses the byte 255 to mean that no relative distance is defined between the two domains.

The number 21 is common for a single hop between two sockets on x86 servers, and larger values appear on machines with more hops or with sub-socket domains, but the value is deliberately relative. It is not a latency in nanoseconds and not a bandwidth. It exists so that software can rank candidate nodes without knowing the machine’s clock. The kernel uses it to order the fallback list consulted when the local node cannot satisfy an allocation, and to score scheduling decisions when a task could run on more than one node. Reading it as a timing is a category error; reading it as an ordering is exactly right.

The distance matrix is also where topology bugs show up. A firmware table that reports a symmetric matrix where the hardware is asymmetric, or that reports a small distance between two nodes that share a saturated link, produces a kernel that makes systematically wrong placement choices. A distance of zero between distinct nodes is the extreme case: it tells the scheduler that moving a task costs nothing.

Which resource a remote access actually consumes

Latency and bandwidth behave differently under the same asymmetry, and the difference determines which policy is appropriate.

For latency, the local DRAM round trip on current server hardware is roughly 80 to 100 nanoseconds, and a remote access adds the traversal of the link in both directions plus the remote node’s own memory-controller latency, typically landing between one and a half and two and a half times the local figure, with the exact multiple depending on the machine’s fabric. A pointer-chasing workload, such as a tree walk or a hash probe sequence, is dominated by these serialized round trips, and a program whose working set sits on the wrong node loses that multiple on every dereference.

For bandwidth, the aggregate scales with the number of nodes because each node has its own DRAM controllers, but a single node’s share is bounded by the links leaving it. A remote access consumes bandwidth on the link in the direction of the request and again in the direction of the response, so it is not free even when the fabric is idle. When one node streams data from another’s memory, it consumes link capacity that the other node would otherwise have for its own local traffic. There is no isolation mechanism for this; the fabric is a shared resource with no per-tenant accounting.

This asymmetry in the asymmetry matters. A workload that is latency-bound should keep its data on the node where it computes. A workload that is bandwidth-bound and reads mostly its own data should be spread so that each node’s DRAM channels are used, which is a placement decision in the opposite direction. The two goals conflict, and no single policy satisfies both.

Topology is not the same as socket count

It is tempting to assume one node per socket, and on many machines that is what numactl --hardware reports. It is not a rule.

AMD’s EPYC processors are built from core complexes, each with its own path to memory, and firmware can expose them as one, two, or four proximity domains per socket. Intel server processors can similarly subdivide a socket into clustered domains. A two-socket machine configured this way presents four or eight nodes, with distances that distinguish same-socket-but-different-domain from cross-socket. Code that assumes node == socket will place two threads on “the same node” when they are in fact a hop apart, and will compute the wrong number of partitions for a per-node data structure.

The reverse also occurs: nodes with memory but no CPUs, which is what a memory-expansion device or a persistent-memory region can look like, and nodes with CPUs but no memory, which appear when a memory range is not described in firmware. A node with no CPUs cannot fault in a page on its own behalf, so it is only reachable through an explicit policy. A node with no memory is a placement target that always falls back.

Finding where a page actually is

The kernel will answer the question directly for any mapped address. The get_mempolicy call with the MPOL_F_NODE and MPOL_F_ADDR flags returns the node of the page backing the address. An address that has never been touched has no page and therefore no node, which is why a placement report is only meaningful for memory already in use.

#include <numaif.h>
#include <stdio.h>

/* Returns the NUMA node backing *p, or -1 if the page is not resident. */
int node_of(const void *p)
{
    int node = -1;
    if (get_mempolicy(&node, NULL, 0, (void *)p,
                      MPOL_F_NODE | MPOL_F_ADDR) != 0)
        return -1;
    return node;
}

The distinction between the node a page is on and the node a thread is running on is the subject of the rest of this cluster. The hardware that a thread executes on is chosen separately from where its memory lives, through the mask mechanism described in CPU Affinity Masks and What Pinning Actually Costs. The topology that firmware describes, and the way the kernel turns it into an ordering, is covered in Reading NUMA Topology from ACPI Tables. Where a page lands in the first place is decided at the fault that creates it, described in First-Touch Placement and the Faulting Thread. When the goal is aggregate bandwidth rather than minimum latency, the alternative is to spread pages deliberately, as Interleaving Pages Across Nodes.

What the distance model leaves out

The model is static. It describes nominal reachability, not current load, and it says nothing about how many other tasks are streaming through the same link at the same moment. Two tasks can each be perfectly local and still contend for the same DRAM channels if they were placed on the same node; two tasks can each be remote and suffer less than the distance suggests if the fabric is quiet. Every placement decision the kernel makes is a bet on a number that was true at boot.

The model is also coarse. A single value per node pair has to stand for read latency, write latency, and bandwidth, which on real hardware differ and can differ in direction. Firmware that reports one number is summarizing a link whose behavior depends on message size, on whether the access is a read or a write, and on whether the line is already cached elsewhere. The kernel’s placement heuristics are correspondingly approximate, which is the reason explicit policy exists at all.