NUMA Placement and the Cost of Touching Another Socket's Memory

On a machine with two sockets, a core on one socket reading a line from the other socket’s memory pays a latency several times the local figure and consumes bandwidth on the interconnect that is shared with every other cross-socket access. The interesting part is that nothing in a program’s source code determines whether a given access is local. The placement was decided by the kernel when the page was first written, by whichever core happened to fault it in.

That is the core of the NUMA problem: the memory a program allocates is not necessarily the memory it should use, and the mechanism that decides placement is a side effect of the allocation path rather than an expression of intent. On a single-socket machine the distinction does not exist and the problem is invisible; on a two-socket machine it can cost a large fraction of throughput, and on a four-socket machine it can make a correctly parallelized program slower than its single-threaded version.

First touch, not allocation

The default policy on Linux is local allocation. When a page is faulted in, that is: when a thread first writes to an address that has no physical page behind it, the kernel allocates a page from the node of the CPU that took the fault. The allocating thread would typically be the one that called malloc, but malloc does not fault the page. It reserves virtual address space and returns; the physical page arrives at the first store.

The gap between reservation and fault is where placement goes wrong. A program that allocates a large array on the main thread, then hands regions of it to worker threads, has all of the array on the main thread’s node. Every worker on the other socket then reads remotely for the entire computation. The faulting thread is not the consuming thread, and the policy cannot know that.

The same mechanism explains the behaviour of calloc and of freshly mmap-ed anonymous memory. What matters is not which call produced the address but which thread first wrote to each page. This is why the term used is first touch, and why the technique of having each worker initialize its own region before use is the simplest correct fix.

/* Each worker first-touches its own slice, so pages land on its node. */
void init_slice(double *base, size_t per_worker, int workers) {
    #pragma omp parallel num_threads(workers)
    {
        int id = omp_get_thread_num();
        double *slice = base + (size_t)id * per_worker;
        for (size_t i = 0; i < per_worker; i++)
            slice[i] = 0.0;          /* fault in locally, before any use */
    }
}

The initialization loop is not wasted work. It performs the faults that would otherwise have happened on the wrong node later, and it does so while the pages are written by exactly the thread that will later read them. The exception is memory that is initialized once and read many times, where the fault cost is negligible against the access pattern and the placement still matters for all reads that follow.

What the distance costs

The magnitude is worth being concrete about, in orders rather than in measurements. A local memory access that misses every cache level costs on the order of a hundred nanoseconds on a current server CPU. An access to memory on another socket costs several times that, and the multiple grows with the number of hops in the interconnect topology: a two-socket machine has one hop between any pair of remote nodes, a four-socket ring can require two.

Bandwidth has a different shape and is often the more important constraint. Each socket has its own memory controllers, and local accesses drawn from threads on that socket are served at the socket’s full bandwidth. Cross-socket traffic shares the link between the sockets, which is a fraction of the aggregate local bandwidth. A workload whose threads all read from one node’s memory obtains roughly the bandwidth of one socket plus the interconnect, not the sum of the machine’s controllers.

Access pattern	Latency order	Bandwidth available
Thread reads its own node’s page	local, ~100 ns	full local bandwidth
Thread reads the other node’s page	remote, several × local	shared interconnect
Thread reads a page written by a peer thread	remote or local, depending on placement	as placed
Read-only page replicated on both nodes	local	full local bandwidth, at the cost of copies

The third row is the one that surprises: pages do not move to follow the readers. A page written on socket 0 and read on both sockets is served locally to socket 0’s threads and remotely to socket 1’s, permanently, unless something migrates it. The kernel’s page migration for automatic balancing exists and copies pages toward the node that accesses them most often, but it is a sampling-based heuristic with a real cost per migration, and it does not help a stable split of readers.

Pinning threads without placing memory

Setting CPU affinity is the common first step, and it is necessary and insufficient. Pinning a worker to a core on socket 1 does not move any page to socket 1. What it does is stabilize the relationship: once a thread runs predictably on one socket, the placement decision becomes a property of memory rather than of scheduling noise.

Without pinning, the scheduler moves threads between sockets, and every move inverts the locality for whatever pages that thread uses. A thread that was local becomes remote; a page that was remote becomes local. The average cost is the same, and the variance rises. Pinning removes the variance and makes the first-touch decision meaningful, because a thread that never moves has a well-defined node. It also stabilises the place where a thread sleeps and wakes, which is the same property that makes the wakeup side of a contended mutex predictable rather than uniformly bad.

The combination that works is: allocate memory per node on the node that will use it, fault it there, and pin the users to that node. Each element of that sequence is required for the others to matter. A program that pins but does not place memory has the same access pattern as before, with less scheduling flexibility. A program that places memory but does not pin has pages correctly placed relative to the faulting thread’s node and a scheduler free to move the readers elsewhere.

/* Bind the current thread to a single CPU; node follows from the CPU. */
#include <sched.h>

int pin_to_cpu(int cpu) {
    cpu_set_t set;
    CPU_ZERO(&set);
    CPU_SET(cpu, &set);
    return sched_setaffinity(0, sizeof set, &set);
}

The node for a given CPU is discoverable through the kernel’s topology files, and the node identifier is what memory policy is expressed in. The mapping is not the identity in general: on a machine with SMT enabled, sibling hyperthreads share a core, and on a machine with a non-uniform node-to-core layout the CPU numbering may interleave sockets. The topology files are the authority, not the CPU number modulo the number of sockets.

Splitting work along nodes rather than across them

The placement problem becomes much easier when the data partitioning matches the hardware topology. If a computation is split into two halves that each read and write a distinct region, and each half runs entirely on one socket, every access is local and the interconnect carries nothing.

The difficulty is the reduction. Combining two halves requires one half’s data on the other’s node, which is a remote access. The cost is proportional to the boundary size rather than to the working set, and the usual design minimizes the boundary: exchange only the partial results, not the inputs. A stencil computation on a grid split along a plane exchanges one plane of values; the interior of each half never crosses the interconnect.

Coordination is the other boundary. A barrier involving all threads on both sockets requires cache-line transfers across the link, and the latency of those transfers is what a barrier’s critical path consists of. A barrier per node with one cross-node handshake narrows the boundary, which is the same tree argument as Amdahl’s Law and the Serial Fraction That Caps Speedup, applied to the topology: the serial fraction of a barrier is proportional to the number of nodes involved, not the number of threads.

Where the model is not worth implementing

NUMA-aware placement is a substantial amount of machinery: node discovery, thread affinity, per-node allocators, and a data layout that permits per-node partitioning. It is justified when a workload is compute-bound and its working set exceeds the last-level cache, so that memory latency is on the critical path; it is also justified when the workload is bandwidth-bound and the interconnect is the constraint.

It is not justified when the working set fits in cache, because then the pages are touched rarely, and it is not justified for a workload that is dominated by I/O waits, because the cores are idle whenever the memory would matter. A service processing a thousand requests per second touches its heap a few times per request and spends most of its time waiting on sockets; the placement of the heap is not the bottleneck.

There is also a diagnostic problem: a program with bad placement looks like a program with bad parallelism. The throughput curve rises with threads and then flattens, exactly as it does for a serial fraction, and the flattening is caused by bandwidth on the interconnect rather than by a lock. Hardware counters distinguish them, remote line fetches grow with thread count when the problem is placement, and the shape of the curve alone does not.

The measurement that settles the question is a comparison of the same code with and without placement. On a single socket the two are identical by construction, so the difference is a property of the machine and its topology, and it is available only by running on the machine in question rather than by reading its specification.