CPU Affinity Masks and What Pinning Actually Costs

A memory policy decides where pages live. An affinity mask decides which CPUs may run the thread that reads them. The two are independent knobs, and setting one without understanding the other produces the common failure where a thread is pinned to a node and its data is not, or its data is local and the thread is not.

What the mask constrains

sched_setaffinity takes a thread identifier and a bitmask of CPUs, stored as an array of unsigned longs whose width depends on the kernel’s word size. The kernel copies the mask in, intersects it with the set of online CPUs, and refuses the call if the intersection is empty. A mask naming only offline CPUs is an error, not a request that takes effect when those CPUs come back.

The mask is per thread, not per process. Threads created by clone inherit the mask of the creating thread, so a process that pins its main thread and then spawns workers pins them all unless it resets them. This is often not what was intended, because the useful pattern is usually one mask per worker, chosen to match that worker’s data.

The mask is also a constraint on migration, not a reservation. A CPU listed in the mask may run the thread; it does not run it exclusively. If the mask names four CPUs, the scheduler may move the thread among those four at any time, and each move changes the node the thread is on if the CPUs belong to different nodes. A mask that spans nodes defeats the purpose of pinning for locality even though it satisfies the letter of the request.

The narrower question, whether the thread is actually placed near its memory, is not something the mask can express. It says only where the thread may execute. If a mask names CPUs on node 1 and the thread’s pages are on node 0, the pinning is precise and the placement is wrong.

The three reasons to pin, and their costs

Pinning appears in three distinct roles, and each has a different cost.

The first is latency stability. A thread pinned to one CPU never migrates, so it never pays the cost of moving its working set from one cache hierarchy to another, and its remote access ratio is constant. The cost is that the pinned CPU is not available to absorb other work: when the thread is descheduled, that CPU sits idle while other CPUs in the system queue. This is the classic trade of tail latency against throughput, and it is the right trade only when the thread’s latency matters more than the machine’s utilization.

The second is avoiding cross-node migration. A thread that migrates from node 0 to node 1 leaves its pages behind and begins accessing them remotely, while its old cache lines on node 0 become remote sharers. The scheduler’s migration heuristics try to avoid this by preferring CPUs on the thread’s current node, but under imbalance they will migrate anyway. A mask restricted to one node makes the cost explicit: migration can only happen within the node.

The third is isolating a workload from interference. A mask that excludes certain CPUs, combined with an affinity mask on the other workload, gives two workloads disjoint CPU sets. This is what cpuset cgroups do at a larger scale, and it is the only mechanism that guarantees a CPU is not shared. Its cost is the same as the first: capacity stranded on one side cannot serve the other.

Why the load balancer behaves differently under pinning

The scheduler’s periodic load balancer works by moving tasks from busier to idler run queues. A task whose mask excludes the idle CPU is not a candidate for that move, so the balancer simply skips it. The consequence is a system that looks unbalanced in aggregate utilization while every individual run queue is behaving correctly under its own constraints.

The balancer’s unit of decision is the scheduling group, which reflects the topology: siblings within a core, cores within a node, nodes within the machine. When a mask makes a task ineligible to move across a group boundary, the balancer cannot rebalance that group against its neighbor. The group’s capacity is effectively partitioned, and any imbalance inside it can only be fixed by moving work within the group.

This produces a failure mode that is easy to misread: a machine with 64 CPUs and 8 pinned threads shows 56 CPUs busy and 8 idle, with no runnable task ever queued on the 8. Nothing is wrong. The pinning has declared that those 8 CPUs are reserved, and reserving them means they can be idle while the rest is saturated.

The other side of the same behavior is that unpinned threads tend to follow their data rather than the other way around. The scheduler’s NUMA balancing, described in Automatic NUMA Balancing and Its Failure Modes, migrates tasks toward the node where their pages are, so a workload that is not pinned will converge on a reasonable placement on its own. Pinning overrides that convergence with a static decision, which is better only when the static decision is known to be right.

Setting a mask correctly

The practical details of the call are where most mistakes occur. The cpu_set_t type is fixed at 1024 CPUs; machines with more than that require the dynamically sized CPU_ALLOC family, and code that uses the fixed type on a large machine silently truncates the mask.

#define _GNU_SOURCE
#include <sched.h>
#include <pthread.h>
#include <stdio.h>

/* Pin the calling thread to exactly one CPU. */
int pin_to_cpu(int cpu)
{
    cpu_set_t set;
    CPU_ZERO(&set);
    CPU_SET(cpu, &set);          /* undefined if cpu >= CPU_SETSIZE */
    if (sched_setaffinity(0, sizeof set, &set) != 0) {
        perror("sched_setaffinity");
        return -1;
    }
    return 0;
}

CPU_SET on an index at or beyond CPU_SETSIZE is undefined behavior rather than an error return, which is why the dynamic allocation path exists. The kernel interface takes a byte count and a pointer, so the userspace type is only a convention; the syscall itself accepts any mask size the caller declares, up to the kernel’s limit.

Reading back the mask is worth doing after setting it. sched_getaffinity returns the intersection that the kernel actually accepted, and comparing it against the requested set catches the case where half the CPUs named were offline.

Affinity, memory policy, and the order of operations

The two knobs interact through the fault path. Setting an affinity mask before a thread touches its memory means the pages are faulted on the pinned CPU’s node, which is what makes pinning useful for locality. Setting the mask afterward changes where the thread runs but not where its pages are, and the thread then pays remote access for the rest of its life unless something migrates the pages.

The reverse order is also common and also wrong: a program that allocates and initializes its data in a single-threaded prologue, then pins workers to different nodes, has already placed everything on the prologue’s node. The workers are now correctly pinned to nodes whose memory they do not own.

The reliable pattern is to set the mask first, then allocate and touch on each thread, or to set a memory policy that matches the mask so that the faulting node and the target node agree. The distinction between where a page is and where a thread runs, and the reason the fault decides the former, is covered in First-Touch Placement and the Faulting Thread.

The case against pinning

For a workload whose threads are interchangeable and whose data is shared, pinning removes the scheduler’s ability to respond to load and gains nothing, because the data is on every node anyway. For a workload with more threads than CPUs, pinning produces queueing on the pinned CPUs while others idle, and the scheduler cannot help. For a workload that runs alongside others on the same machine, pinning without isolation merely relocates contention.

The signal that pinning is appropriate is a measurable asymmetry: a thread whose data is provably on one node and whose remote access ratio is high. The signal that it is not appropriate is a desire for predictability in general, which the scheduler already provides by preferring the current node and by penalizing migrations that cross nodes.

Pinning is a statement about a specific placement that the program knows better than the kernel does. When the placement itself is under the program’s control, through an explicit allocation on a named node and a matching policy, the mask and the allocation are set together and the statement is verifiable; Partitioning a Workload So Every Thread Reads Local Memory describes that arrangement. When the program does not know, the statement is a guess that the scheduler is no longer allowed to correct.