Measuring Remote Access with Hardware Counters

Placement decisions need evidence, and the evidence comes in two flavors that answer different questions. System-wide per-node counters say how much traffic crossed the fabric and in which direction. Per-process page accounting says where a program’s pages currently sit. Neither alone identifies a misplaced data structure; together they usually do.

Per-node traffic is an aggregate, not an attribution

The kernel maintains per-node statistics that count allocation and free events by node, including the events that indicate a page was allocated on one node in response to a request from another. numastat reads them.

The counters that matter are the ones whose names describe cross-node behavior. An allocation that a task on node 1 requested but that was satisfied from node 0 appears in a cross-node counter on one side and a local counter on the other. A page that was migrated between nodes appears in a migration counter. A remote free, meaning a page freed by a task on a different node from where it was allocated, is counted separately, and a high remote-free count is the signature of a data structure being dismantled by the wrong node.

The limitation is that these counters describe the system, not any process. On a machine running one workload they are attributable by inspection. On a machine running twenty, a rise in cross-node allocation says that something is allocating remotely, and nothing about what. The counters are also cumulative since boot, so a program has to sample them twice and subtract to see a rate, and the subtraction is only meaningful over an interval long enough for the counters to move.

The counters are printed per node rather than as a system total, and the direction of a misplacement shows up in which counter moves. A node whose foreign counter climbs is a node whose tasks are taking memory from somewhere else, and the node that served those allocations records the corresponding miss. Nothing in the output names the task that caused an event and nothing in it is scoped to an address range, so the counters point at a node and stop there.

Page residency is per process and tells the other half

/proc/<pid>/numa_maps gives, per virtual memory area, the number of pages currently resident on each node, alongside the policy in force. Summed across areas, it yields a residency profile: how a process’s memory is distributed.

The profile answers “where is the data” and not “where is the data being used”. A process whose pages are all on node 0 and whose threads all run on node 1 has a perfect residency profile and a completely wrong placement. The profile is only informative next to the process’s CPU placement, which /proc/<pid>/stat field 39 reports as the last CPU the task ran on, and which ps or taskset -p will show directly.

Combining the two produces the first useful signal: for each thread, the node it runs on and the node distribution of the memory it touches. A thread on node 1 whose relevant area is entirely on node 0 is a candidate for either pinning the thread to node 0 or migrating the area to node 1, and the choice between those depends on whether other threads also use the area.

Hardware events that separate local from remote

The processor’s performance monitoring unit can count memory events that distinguish where a request was served. On Intel server processors the uncore exposes events for memory traffic at the integrated memory controller and at the interconnect link, and the core exposes events for cache misses with a response-type or source field. On AMD processors the data fabric exposes similar events, and the instruction-based sampling facility records, per sampled instruction, the source of the data that was fetched.

The exact event names are model-specific and change between generations, which is why naming them in general prose is a mistake. What is stable is the shape: an event that counts last-level cache misses whose data came from local DRAM, and a companion event that counts misses whose data came from a remote node’s DRAM. The ratio of the second to their sum is the remote access ratio, and it is the number that placement policy is trying to minimize.

The counting has to be done per node, because the interesting quantity is not the machine’s total remote ratio but each node’s. A machine where node 0 serves most of its requests locally and node 1 serves most of its remotely has an asymmetric problem: typically one node running the hot workload and the other holding its pages.

Sampling, rather than counting, is what makes the result attributable to code. A sampling facility that records the instruction pointer and the data source for each sampled miss yields a profile where remote misses cluster by function and by data address. That profile is what turns a ratio into a diagnosis: which structure, in which code, is being read across the fabric.

Why the counters disagree

Three systematic reasons explain most discrepancies between per-node traffic and per-process residency.

Shared pages are counted once in residency and many times in traffic. A page mapped by ten processes appears in ten residency profiles and contributes to every access any of them makes. A shared buffer therefore looks well-placed from every process’s point of view while generating remote traffic for most of them.

Page-cache pages are attributed to the file, not the reader. A memory-mapped data file is resident once, and every process that reads it contributes traffic to the same page. The residency profile of a process that maps a large file may show almost no anonymous memory and a great deal of file-backed memory, and the placement of the latter was decided by whichever process first read the file.

Speculative and prefetched accesses are counted as traffic but not as demand. A hardware prefetcher that streams a remote array generates fabric traffic without any load instruction having missed, so the traffic counters rise while a per-instruction sample shows nothing. This is not a measurement error; it is real bandwidth consumption, and it is one of the reasons a bandwidth-bound workload can be hurt by remote placement even when its latency profile looks acceptable.

#include <stdio.h>
#include <string.h>

/* Residency profile of one area, from /proc/self/numa_maps.
 * A line looks like:
 *   7f2c00000000 default anon=4096 dirty=4096 N0=3072 N1=1024
 * Parse the N<node>=<pages> fields to build the distribution. */
static void residency(const char *line)
{
    const char *p = line;
    while ((p = strstr(p, " N")) != NULL) {
        int node, pages;
        if (sscanf(p, " N%d=%d", &node, &pages) == 2)
            printf("node %d: %d pages\n", node, pages);
        p += 2;
    }
}

The parsing is trivial and the interpretation is not: a distribution is only meaningful against the set of nodes the area’s users run on, and that set is not in the file.

A measurement sequence that converges

The order of measurement matters, because each step narrows the next.

Start with the per-node cross-allocation counters over a known interval to establish whether the machine has a placement problem at all. If cross-node allocation is negligible and the workload’s throughput is as expected, there is nothing to fix and no further measurement is warranted.

If cross-node traffic is significant, move to per-process residency and CPU placement to find which process and which area. A process whose threads and pages disagree is the target.

Then use sampled hardware events on that process to find the hot data. The samples convert “this process’s pages are on the wrong node” into “this structure is read across the fabric on every iteration”, which is a statement that can be acted on.

Finally, verify after the change with the same counters. A migration that does not move the remote ratio has not fixed anything, and the counters are the only way to know. The relationship between a migration and the ratio it is supposed to improve is covered in Page Migration: What a Move Between Nodes Costs, and the mechanism that performs migrations without being asked is described in Automatic NUMA Balancing and Its Failure Modes.

What cannot be measured this way

The counters describe accesses that reach the memory system. They say nothing about accesses that hit in a cache, and those are the majority. A structure that is read repeatedly and stays resident in a private cache costs nothing regardless of which node its page is on, and no memory-side counter will show it.

This is the reason a remote access ratio is not a proxy for performance. A workload can have a high remote ratio and be entirely cache-resident, in which case the ratio is measuring a cost that is not being paid. The ratio becomes meaningful only for accesses that actually reach DRAM, which is why the useful derived quantity is remote misses per unit of work rather than remote accesses per total access.

The second blind spot is the fabric itself. A link that is saturated shows up as latency in every access that crosses it, and no per-node counter reports link utilization as such. The traffic counters approximate it, and the approximation is only valid if the link’s capacity is known, which it generally is not from inside the operating system.