Control Groups v2: How Memory and CPU Limits Are Enforced

Namespaces partition what a process can see. They do not partition how much of the machine it can consume, and a process that can see nothing outside its namespace can still allocate every free page on the host. Control groups supply the second half: a hierarchy of accounting domains, with limits attached at nodes, and enforcement implemented at the points where the kernel allocates the resource. The two mechanisms are independent, which is why a container can have perfect namespace isolation and no resource isolation, and why the memory limit is a distinct object from the mount table.

The hierarchy and the accounting rule

Cgroup v2 exposes a single unified tree, mounted at /sys/fs/cgroup, with a cgroup.controllers file at each node listing the controllers available to its children. A controller is enabled for a subtree by writing +memory to cgroup.subtree_control in the parent, and only then do the controller’s interface files appear in the children. A process belongs to exactly one cgroup, and its resource consumption is charged there and aggregated upward through every ancestor, so a limit set at an interior node bounds the sum of everything below it regardless of how the leaves are divided.

The no internal process constraint distinguishes v2 from v1. A cgroup that has children and also has processes of its own may not have controllers enabled for distribution to those children, so the tree is either a set of leaves holding processes or a set of interior nodes holding limits. Distributing controllers to children requires the parent’s process list to be empty, which forces a shape where the limits are set on interior nodes and the processes live at the leaves. The constraint exists because in v1 a process could be in one cgroup for memory and another for CPU, and no consistent aggregate could be computed; v2’s single membership makes the aggregate well defined and the constraint is the price.

Memory: charge at allocation, reclaim at the limit

Memory accounting is per-page and happens at fault time. When a process faults a page in, the kernel charges the page to the cgroup of the faulting process, and the charge is recorded in the page’s memcg_data field so that reclaim and migration can find the owner later. Page cache is charged to the cgroup that first touched the page, which means a file read by one container and then read by another is charged entirely to the first: the accounting follows the page, not the reader.

The limit is memory.max. When a charge would push the cgroup’s usage above the limit, the charging path first attempts reclaim within the cgroup: it shrinks the LRU lists of that cgroup only, evicting clean page cache and swapping anonymous pages if a swap device is configured for the cgroup. Reclaim is synchronous with the fault, so the faulting process waits. If reclaim cannot free enough, the kernel invokes the cgroup’s OOM killer, which selects a victim among the cgroup’s processes by oom_score and kills it. The kill is per-cgroup, so the host’s OOM killer is not involved and processes outside the cgroup are unaffected.

That sequence explains the most common misreading of a memory limit: it is not a cap that makes allocation fail. malloc succeeds, the page is faulted, the fault triggers reclaim, and the process either continues after a pause or dies. A program that checks the return value of malloc sees nothing wrong until the OOM kill, and the kernel log records the kill against the cgroup rather than the process.

memory.high provides a softer control. Exceeding it does not trigger the OOM killer; it throttles the allocating process and pushes it into direct reclaim, so a cgroup above memory.high runs slowly rather than dying. memory.low and memory.min are the inverse: they mark a subtree as protected from reclaim pressure originating outside it, so a page charged below a memory.min node is skipped by reclaim unless the node itself is over its limit. The four knobs form a range over one resource: memory.min and memory.low are protections, memory.high is a throttle, and memory.max is the hard cap. The ordering between them matters. The hard protection of memory.min takes precedence over the best-effort protection of memory.low on the same subtree, and going over memory.high never invokes the OOM killer while reaching memory.max with nothing left to reclaim does.

A page charged to a cgroup cannot be reclaimed if it is pinned by a kernel object that does not participate in reclaim, and the cgroup’s usage then reflects memory that no amount of pressure will free. Page tables, socket buffers, and some slab objects are charged this way. memory.current includes them; memory.stat breaks the total into anon, file, slab, kernel_stack and other lines, and the slab_unreclaimable figure is the one that explains a cgroup stuck at its limit with an empty page cache.

CPU: bandwidth as a quota over a period

CPU control has two independent axes. Weight, set by cpu.weight with a default of 100 and a range of 1 to 10000, distributes time among runnable cgroups when the CPU is contended and has no effect when it is idle. Bandwidth, set by cpu.max in the form quota period, imposes an absolute ceiling: the cgroup may run for quota microseconds in every period microseconds, with the period defaulting to 100000 (100 ms) and the quota able to be max for no limit.

Enforcement happens in the scheduler’s periodic accounting. Each cgroup has a bandwidth timer that expires once per period; when a cgroup’s runtime exceeds its quota within a period, its tasks are removed from the run queue and the cgroup is marked throttled until the next period. The tasks are not blocked in a syscall and are not signalled — they are runnable but not scheduled, and the delay is visible to them only as elapsed time. A quota of 50000 with a period of 100000 therefore gives at most half a CPU per period, spread across however many threads the cgroup has.

The distinction between the two axes is the distinction between fairness and rate limiting. Weight answers “given contention, what share does this group get” and produces a proportional share that varies with the number of runnable groups. Quota answers “what is the maximum this group may consume” and is independent of contention. A cgroup with a small weight and no quota can still consume an entire machine when nothing else is running; a cgroup with a large weight and a small quota is throttled regardless.

# At most half a CPU, measured over a 10 ms period.
echo "5000 10000" > /sys/fs/cgroup/box/cpu.max
# Relative share when contended, no ceiling.
echo 200 > /sys/fs/cgroup/box/cpu.weight

The period is the parameter that determines burstiness. A short period smooths the limit and increases the frequency of throttling events; a long period allows larger bursts within a period and longer stalls at the boundary. The default of 100 ms is a compromise: short enough that a throttled task is not delayed for a perceptible interval, long enough that the accounting overhead is negligible. The period is bounded between 1 ms and 1 s, and the two ends trade the same way: near the 1 ms floor the timer and quota-transfer overhead becomes measurable, while near the 1 s ceiling a cgroup that exhausts its quota early in the period stays idle for the remainder of it.

Where the controllers meet the namespaces

Cgroup membership is orthogonal to namespace membership. A process in a container’s namespaces can be in any cgroup the runtime chooses, and the cgroup is the object that holds the limit. When a container is created, the runtime makes a cgroup, moves the init process into it, and the children inherit membership at fork. The namespace’s PID 1 and the cgroup’s root are different objects, and the runtime must keep both consistent as processes are added.

The delegation rule matters for unprivileged containers. A cgroup subtree can be delegated to a less privileged process by granting it write access to the subtree’s directory and to its cgroup.procs, cgroup.threads and cgroup.subtree_control files, while withholding access to the resource interface files in that directory, which answer to the parent. The delegate can then build a sub-hierarchy and divide the resources it was given, and the migration rules keep the delegation contained: a non-root writer cannot move a process across the delegated boundary. Without delegation, a container that wants to manage its own sub-groups needs CAP_SYS_ADMIN in the cgroup namespace, and the cgroup namespace itself — CLONE_NEWCGROUP — only changes the view of the hierarchy, presenting the delegate’s subtree as /. It does not grant authority.

Several resources are outside the memory and CPU controllers and still matter. pids.max caps the number of processes in a cgroup, and it is the direct defence against a fork bomb that a PID namespace cannot stop, since the namespace bounds names, not counts. io.max caps bytes and operations per second per device. cgroup.freeze stops every task in a subtree without signalling them. Each is enforced at its own allocation point, and none of them interacts with the others: a cgroup can be within its memory limit, within its CPU quota, and still be unable to fork.

Failure modes cluster around the accounting boundaries rather than the limits. A page charged to a cgroup that is then migrated to another cgroup is re-charged on migration, which costs time proportional to the resident set. A process that changes cgroup membership takes its already-charged pages with it only on explicit migration; the charge does not follow a process that merely forks. Kernel memory charged to a cgroup is not reclaimable in the way user pages are, so a limit set below the kernel’s own overhead for the workload produces OOM kills that no application tuning can prevent. The remaining pieces of a container — the syscall filter and the capability set that decide what the process may do with the resources it has — are covered in seccomp-BPF and the Filtering of System Calls and Capabilities and the Decomposition of Root Privilege.