Automatic NUMA Balancing and Its Failure Modes

First-touch placement is a bet made once, at fault time, and never revisited. A process that starts on one node and later grows to use all of them keeps its early pages on the node that happened to fault them, and every thread that ends up elsewhere pays for the original decision forever. Automatic NUMA balancing is the kernel’s attempt to correct that bet at runtime, using the page fault machinery itself as the observation channel.

The design is elegant and the failure modes are specific, which makes it a good case study in the limits of heuristic placement.

Faulting on purpose

The mechanism is easiest to understand as an inversion of normal fault handling. Ordinarily a fault means a page is missing and the kernel must supply it. Under automatic balancing, the kernel periodically removes present entries from page tables so that the faults that follow will tell it something.

The kernel marks a range of a task’s pages as not present, but keeps the underlying page in memory. The page table entry is replaced with a special encoding that the fault handler recognizes. When the task touches the address again, it faults, and the handler knows three things it did not know before: which task, which address, and which CPU, and therefore which node, the access came from. The page itself is intact, so the fault is resolved immediately by reinstalling the entry. The cost is one fault per sampled page per sampling period.

That cost is the first thing to understand about the mechanism. It is not free. Every sampled page costs a fault, and a fault on a present page costs on the order of a microsecond including the return to userspace, against a few hundred nanoseconds for a cache hit on a local page. On a workload with a small, hot working set, the sampling faults can cost more than the remote accesses they are meant to fix.

From samples to decisions

The samples accumulate per task, in a per-task structure that records the node of each observed fault. When the count for a node exceeds a threshold, the task is a candidate for migration to that node: the scheduler may move the thread so that it runs where most of its faults are.

Separately, the samples accumulate per page. The kernel records, in each page’s metadata, the node of the last task that faulted on it. When a task on node 1 faults on a page homed at node 0, the page’s recorded node becomes 1, and after enough such observations the kernel attempts to migrate the page to node 1. Migration is attempted only when the imbalance is repeated, because a single remote access is not evidence of a stable pattern.

Two mechanisms therefore run in parallel: tasks move toward their pages, and pages move toward their tasks. They are not coordinated beyond sharing the same sample stream, and in principle they can act on the same sample in opposite directions. The kernel’s protection against oscillation is a rate limit per task and a per-page check that refuses to migrate a page that was moved recently.

/* Illustrative pseudocode for the decision, not kernel source. */
for each sampled fault(task, page, faulting_node):
    task->sample_count[faulting_node] += 1
    if task->sample_count[faulting_node] > task_threshold:
        try_migrate_task_to(task, faulting_node)

    if page->last_fault_node != faulting_node:
        page->last_fault_node = faulting_node
        page->remote_fault_count += 1
        if page->remote_fault_count > page_threshold:
            try_migrate_page_to(page, faulting_node)

The thresholds are the tuning surface. A low threshold reacts quickly and migrates on noise; a high threshold reacts slowly and misses short-lived phases. Neither is right for every workload, which is why the mechanism is a heuristic rather than a policy.

Where it goes wrong

The first failure mode is a workload whose phases are shorter than the reaction time. A program that processes dataset A, then dataset B, then A again, with each phase lasting a few milliseconds, will have samples from phase A being acted on during phase B. The migrations arrive after the phase that justified them has ended, and the result is a steady stream of migration work that never converges. The cost is pure overhead, and the placement is no better than first-touch would have been.

The second is a workload with many threads sharing pages. Every thread that faults on a shared page contributes a sample, and the page’s recorded node flips according to whichever thread faulted most recently. The page is then migrated toward a node that only some of its users are on, and the migration itself invalidates the TLB entries of every thread mapping it. A page shared by threads on every node has no correct location, and the balancing mechanism will keep trying to find one.

The third is a workload whose threads are themselves migratory. If the scheduler moves a thread to follow its pages, and the pages then follow the thread, the pair can travel together across the machine, incurring a migration on every step. The rate limit bounds the damage but does not eliminate it.

The fourth is interaction with explicit policy. A task whose memory policy is MPOL_BIND is not balanced by default; the policy can be installed with the mode flag that requests balancing for that task anyway, which is the documented way to keep both a constrained node set and adaptive placement. A task left under MPOL_DEFAULT at both the task and the range level is a candidate. A program that sets policies on some ranges and not others therefore gets a mixture of static and adaptive placement, and the boundary between them is not visible in any single report.

Turning it off and on

The mechanism is controlled by a sysctl, kernel.numa_balancing, which is a system-wide switch. Setting it to zero stops the sampling and the migrations entirely. The per-task route runs through the memory policy interface: a task that installs an MPOL_BIND policy is excluded from balancing unless it asks for it with the mode flag that re-enables balancing for that task, so a job scheduler that manages placement explicitly can keep the mechanism off for the processes it controls without changing the machine-wide setting.

There are three legitimate reasons to disable it for a workload.

The workload has already set explicit policies that are correct, in which case balancing adds faults and can only move pages that the policy left unconstrained.

The workload’s access pattern is stable and its placement is already right, in which case the sampling faults are pure cost.

The workload is latency-sensitive at the microsecond scale, where a sampling fault is a visible stall. A trading or control-loop thread that must not fault during a critical section cannot tolerate the mechanism’s interference.

The counter-argument is that disabling it freezes whatever placement first-touch produced, which for a long-running service that grew from a small initial footprint is usually wrong. The correct comparison is not “balancing versus perfect placement” but “balancing versus first-touch”, and first-touch is often much worse.

What the counters show

The kernel exposes the sampling and migration activity per node and per task, and the per-page outcome in /proc/<pid>/numa_maps as the per-node page counts. A workload under balancing shows its pages drifting over time in that file; a workload with balancing disabled shows them fixed.

A task that is being migrated repeatedly shows a high migration count in its per-task scheduler statistics with no improvement in its remote access ratio. That pattern of migration activity without placement convergence is the signature of a phase-shorter-than-reaction-time workload, and the remedy is an explicit policy rather than more balancing.

The measurement side of this is the subject of Measuring Remote Access with Hardware Counters, which describes the counters that distinguish a page that is remote but rarely touched from one that is remote and hammered. The explicit alternative to balancing is described in Page Migration: What a Move Between Nodes Costs, and the reason the fault is the only observation channel available is covered in First-Touch Placement and the Faulting Thread.

Why the design is constrained this way

An alternative design would sample accesses without faulting, using hardware performance counters that record the address of each cache miss. That is what a profiling tool does, and it is far cheaper per sample. The kernel does not do it because the counters are a scarce, per-CPU resource with no ownership model that would let the scheduler use them continuously across all tasks, and because sampling at the required rate would perturb every workload on the machine rather than only the one being balanced.

A second alternative would be for the kernel to maintain explicit access history per page. The storage cost is prohibitive: a few bytes per page on a machine with hundreds of gigabytes is hundreds of megabytes of metadata, and the update would have to happen on every access rather than on a sampled subset.

Fault-based sampling is the compromise that costs nothing when the mechanism is off, costs a bounded amount when it is on, and requires no hardware support beyond what every architecture already provides. Its accuracy is limited by the sampling rate, and its reaction time is limited by the fault cost, and both of those limits are the reason the failure modes above exist.