NUMA First-Touch and the Memory Bandwidth Ceiling per Node

A node with two sockets and a hundred gigabytes per second of memory bandwidth printed on its specification sheet does not deliver a hundred gigabytes per second to a program that treats its memory as one flat pool. Each socket owns a subset of the memory controllers, and a core reaches the other socket’s memory through a link between the sockets that has its own, smaller capacity. The nominal figure is the sum of the local controllers; the achievable figure depends on where the pages happen to live relative to the cores reading them.

On a cluster, this is the last layer of the performance hierarchy before the network, and it is the one most often left to chance. A rank that reads mostly remote memory runs at a fraction of the node’s bandwidth while showing no communication at all, and the resulting scaling curve looks like a network problem.

Pages live on a node, not in the machine

Non-uniform memory access means exactly what it says: the cost of a load depends on which memory controller serves the cache line. A local access goes through the core’s own controller. A remote access goes through the inter-socket interconnect to the other controller, which adds latency and consumes capacity on a link shared by every core on the socket.

The interconnect is the constraint. Its bandwidth is typically comparable to one socket’s local memory bandwidth, not to the sum of both, so a workload that sends a large fraction of its traffic across the link saturates it while the local controllers are half idle. The latency penalty is smaller in relative terms than the bandwidth penalty but still large enough to matter for pointer-chasing code: a remote miss costs more than a local one by a factor that depends on the generation of the interconnect, often tens of percent.

Virtual memory hides the structure but does not remove it. A virtual page is mapped to a physical frame on a specific node, and the translation is what the hardware uses to decide where to send the request. The operating system chooses the node at allocation time, and the policy it uses is the subject of the next section.

First touch decides placement

The default policy on Linux for anonymous memory is first touch: the page is allocated on the node of the CPU that first writes to it. Reads of an unallocated page fault and map a shared zero page, so a read does not establish placement; a write does.

This produces a failure mode that is easy to create and hard to see. A program allocates a large array, initializes it in a serial loop on one thread, and then enters a parallel region in which threads on both sockets read and write the array. Every page was placed by the initializing thread, so every page lives on one socket, and half the threads run remotely for the entire computation. The initialization loop is a tiny fraction of the runtime and determines the placement for the whole run.

The repair is to make the initialization parallel and to use the same thread-to-data mapping as the computation. If thread t will own the range [t*n/T, (t+1)*n/T) during the solve, thread t should write that range first. The two loops must use the same partition, or the pages end up on the wrong nodes even though the initialization was parallel.

#include <stdlib.h>
#include <omp.h>

/* Parallel first touch: the thread that will compute on a range
   also writes it first, so its pages land on that thread's node. */
double *alloc_local(long n)
{
    double *a = malloc((size_t)n * sizeof *a);
    if (!a) return NULL;

    #pragma omp parallel for schedule(static)
    for (long i = 0; i < n; i++)
        a[i] = 0.0;

    return a;
}

The schedule(static) clause matters. With the default schedule the iteration-to-thread mapping is unspecified, and the compute loop may use a different mapping from the initialization loop, which scatters pages across nodes and defeats the purpose.

Two complications are worth knowing. Transparent huge pages change the granularity: a huge page is placed by the first touch anywhere within it, so one thread touching one byte can pull two megabytes onto its node. That is usually helpful, because it reduces translation overhead, but it makes placement coarser and can concentrate a large region on one node if the initialization is not partitioned carefully. Second, memory that is allocated by a library before the parallel region, such as an MPI internal buffer, is placed by whichever thread initialized the library, which is typically the main thread before any binding has taken effect.

The roofline at the socket boundary

The roofline model compares attained floating-point rate against two ceilings: the processor’s peak and the memory system’s delivered bandwidth multiplied by the arithmetic intensity of the kernel. Attained performance is the minimum of the two.

Arithmetic intensity is flops per byte moved from memory. A seven-point stencil that streams two arrays reads eight bytes and writes eight bytes per point for seven flops, an intensity of about 0.44 flop per byte. A socket that delivers 100 GB/s therefore supports about 44 GFLOP/s for that kernel, well below the vector peak of the cores. The kernel is memory-bound, and the only ways to improve it are to raise the intensity through cache blocking or to lower the bytes moved.

The NUMA structure changes the effective bandwidth in the roofline. If half the traffic is remote and the inter-socket link delivers less than the local controllers, the delivered bandwidth in the denominator falls, and the roofline drops with it. The cores are not slower; the memory system is.

This is also where the network competes. Incoming messages are written into memory by the adapter, through the same controllers that serve the cores. A rank receiving a large halo at a high rate consumes memory bandwidth that the roofline assumed was available to arithmetic. On a node with several ranks, the aggregate receive rate can be a substantial fraction of the local bandwidth, and the halo exchange cost then appears twice: once as network time and once as memory contention.

What this does to a rank

A rank is a process, and the scheduler may move it between cores unless it is bound. Migration is not free under first-touch: the pages stay where they were allocated, so a rank that starts on socket 0, allocates its arrays, and is later scheduled on socket 1 reads all of its data remotely. Binding the rank to a set of cores on one socket prevents the migration and keeps the placement valid.

The choice of how many ranks per socket follows from the bandwidth arithmetic. If one rank can saturate a socket’s bandwidth with a memory-bound kernel, a second rank on the same socket adds contention rather than throughput. If the kernel is compute-bound, more ranks per socket may be better because they keep more execution units busy. The right number is a property of the kernel’s intensity, not a universal constant, which is why hybrid configurations with one rank per socket and several threads inside are common for memory-bound codes.

A rank also allocates memory on behalf of the library. Receive buffers, unexpected-message queues, and collective scratch space are allocated during initialization, often before the application has set any policy. If initialization happens on one socket, those buffers are remote for ranks on the other, and every message pays the inter-socket hop in addition to the network hop. Some implementations expose memory-binding options for exactly this reason; the portable fallback is to start the ranks with the desired binding already in place, so that first touch during initialization lands correctly.

Detecting and avoiding remote traffic

The first diagnostic is the operating system’s own NUMA statistics, which report per-node allocation and hit counts. A large imbalance between nodes for a process that should be spread evenly is the signature of serial first touch. Hardware performance counters can report local versus remote DRAM accesses on processors that expose the distinction, and a memory-bandwidth benchmark run with the same binding as the application establishes the achievable ceiling for that configuration.

The second diagnostic is a bandwidth measurement of the kernel itself. A kernel that attains far less than the roofline predicts, with low cache miss rates and no communication, is usually limited by something other than the arithmetic. Running the same kernel with all threads on one socket, and then with the array interleaved across sockets, separates the placement effect from the kernel’s own limits.

Interleaving is the blunt remedy. Spreading pages round-robin across nodes gives every core a mixture of local and remote accesses and avoids the case where one socket holds everything. It is worse than correct placement for a bandwidth-bound kernel, because half the accesses are remote by construction, but it is far better than the pathological case and it requires no knowledge of the access pattern. Binding plus parallel first touch is the precise remedy and needs the code to know which thread owns which data, which is usually true for a decomposed solver and rarely true for a general-purpose library.

The final consideration is that the node is the allocation unit. A job that requests a full node and uses half of it has paid for bandwidth it does not use; a job that requests half a node and gets cores on both sockets without the corresponding memory may be unable to use even that. The scheduler’s resource model and the memory system’s topology are not the same model, and the scaling study is where the mismatch becomes visible.