Reading NUMA Topology from the SRAT and SLIT

Before any placement decision can be made, something has to answer a question the CPU cannot: which memory ranges belong to which group of processors, and how expensive is it to reach from one group to another. The processor knows its own APIC identifier and nothing about locality. The answer comes from firmware, in two tables the operating system reads once at boot and never re-reads. Everything the kernel does later, from the fallback order for allocations to the scoring of scheduler placement and the zones that appear under /sys/devices/system/node/, is derived from those two tables.

Two tables, two different questions

The Static Resource Affinity Table answers “what is in this domain”. The System Locality Distance Information Table answers “how far apart are the domains”. The split is deliberate: membership changes with hardware configuration, while distance is a property of the interconnect, and firmware can describe either without the other. A machine with a SLIT but no SRAT is legal, as is a SRAT with no SLIT.

Both tables are arrays of variable-length entries, introduced by a standard ACPI system description table header. Every entry begins with a one-byte type and a one-byte length, so a parser must advance by the length field rather than by a fixed stride. That single rule is where most hand-written parsers break, because the entries are not all the same size and later revisions added types that older parsers do not know.

The unit both tables share is the proximity domain, a 32-bit integer. It is an arbitrary label chosen by firmware, not an index into anything. Values may be sparse and need not start at zero, though in practice they do. The kernel renumbers them into a dense array of node identifiers, which is why the number a program sees in /sys/devices/system/node/node3 is not necessarily the domain number that firmware wrote in the table.

SRAT entries that matter

The SRAT carries several entry types. Two of them define the topology.

A processor affinity entry names a proximity domain and an APIC identifier, either an x2APIC ID in the modern 64-bit form or the older 8-bit APIC ID, and declares that the processor with that identifier belongs to that domain. There is one entry per logical processor, so a machine with 256 hardware threads has 256 such entries, each 16 bytes for the 8-bit APIC form or 24 for the x2APIC form, which widens the identifier field to 32 bits. A flag bit in each entry marks whether the processor is enabled; a disabled entry means the firmware has described a socket that is physically present but not populated, or a core that has been fused off.

A memory affinity entry names a proximity domain and a physical address range: a 64-bit base address, a 64-bit length, and a flags field marking whether the range is enabled. The entry is 40 bytes, and the flags distinguish an enabled range from one that is present but offline, and mark a range that is hot-pluggable or backed by non-volatile memory. The ranges are not required to be contiguous, and on real machines they are not: firmware describes many small ranges because some address space is reserved for MMIO or for interleaved regions. A memory range with a base of zero and a length covering the low megabyte is common and is not an error.

Two further entry types matter for correctness. A generic initiator affinity entry attaches a device, identified by its ACPI or PCI path, to a proximity domain; this is how a NIC or an accelerator is declared to be close to a particular node, and it is the basis for the kernel’s notion of device locality. A generic port affinity entry does the same for a port rather than a device.

The parser’s job is to build, for each domain, the set of logical processors and the set of physical address ranges. The kernel then creates one node per domain that has memory, attaches the processors, and computes the zone boundaries. A domain that appears in processor entries but never in memory entries becomes a CPU-only node; a domain with memory but no processors becomes a memory-only node. Both are representable, and both cause trouble for code that assumes every node can run a task.

Why the SLIT is a matrix of bytes

The SLIT is structurally simple: a 64-bit header carrying the number of proximity domains, followed by an N×N array of unsigned bytes. Entry (i, j) is the relative distance from domain i to domain j. The specification defines 10 as local, reserves values 0 through 9, and defines 255 as meaning the two domains have no relative distance: a hard statement that a path does not exist, not merely a large number.

Three properties of this design are easy to miss.

The matrix is nominally symmetric but need not be in practice. Nothing in the format forces entry (i, j) to equal entry (j, i), and a machine where reads are cheaper in one direction than the other could in principle express that. Software that assumes symmetry, for example by using only the upper triangle, will be correct on most hardware and silently wrong on hardware that does not.

The diagonal is required to be 10, and the values are not comparable across machines. A distance of 21 on one server and 32 on another do not mean the second machine is 1.5 times slower; they mean the firmware author chose different numbers for links that may be identical. This is why the kernel treats the values as weights for ordering, never as a cost model to be converted into time.

The matrix size is bounded by the number of domains, and the header field is a 64-bit count. A malformed table claiming an enormous count must be rejected before the array is allocated, which is a check that has been the subject of firmware bugs.

From tables to the kernel’s view

Linux consumes both tables early in boot and constructs a pg_data_t per memory-bearing node. Three user-visible consequences follow.

/sys/devices/system/node/nodeN/distance exposes row N of the SLIT verbatim, as space-separated decimal numbers. That file is the most direct way to see what firmware claimed, and comparing it against the node membership in cpulist and meminfo is how topology misdescriptions are diagnosed.

The kernel also derives a fallback list per node by sorting the other nodes by distance. When an allocation cannot be satisfied from the preferred node, because it is out of memory or because the allocation is constrained by a cpuset, the allocator walks that list in order. A SLIT that reports all-zero distances flattens the list into an arbitrary order, and allocations then fall back essentially at random.

When the SLIT is absent, the kernel synthesizes distances from the SRAT alone: same domain is 10, different domains get a fixed default. The behavior is defined, not a failure, but it discards any information about which cross-node links are cheaper, and on a machine with a nonuniform fabric that information is exactly what placement policy needs.

/* Shape of the SLIT as it appears in memory, after the ACPI header. */
struct slit {
    u64 locality_count;        /* number of proximity domains, 64-bit in the table */
    u8  distance[];            /* locality_count squared bytes, row-major */
};

/* A defensive reader: reject the table rather than trust the count. */
static int slit_distance(const struct slit *s, u64 from, u64 to)
{
    if (from >= s->locality_count || to >= s->locality_count)
        return -1;
    return s->distance[from * (size_t)s->locality_count + to];
}

The multiplication is the part worth checking. A count of 65536 produces a 4 GiB array, and a count that firmware wrote by accident produces an allocation the kernel cannot satisfy. The index arithmetic also has to be done in a width that cannot overflow at the declared count, which on a 32-bit build means widening before multiplying.

Where the description stops being true

Firmware tables are a snapshot of intent, and several things can make them wrong.

Virtualization is the largest source. A hypervisor presents a synthetic SRAT and SLIT describing virtual NUMA topology, which may or may not correspond to the physical placement of the guest’s memory. When a guest is backed by a single host NUMA node, its synthetic two-node topology is a fiction that costs the guest a migration it cannot benefit from. When the guest’s memory is spread across host nodes, the synthetic topology understates the real cost. The guest cannot distinguish these cases from inside.

Hot-plug is the second. Memory added after boot appears in the SRAT only if firmware updates it, and firmware support for that is inconsistent. The kernel can accept memory from a device that reports its own locality, but a range whose domain is unknown cannot be placed correctly.

Finally, the tables describe hardware, not the operating system’s use of it. They say nothing about which node currently has free pages, how loaded each link is, or whether a node has been taken offline by an administrator. Those are runtime facts, and the static description is the baseline against which they are measured. The placement decisions built on that baseline are covered in First-Touch Placement and the Faulting Thread, and the cost model the distances feed into is described in Node Distance and the Real Cost of a Remote Memory Access.