Memory and CPU Limits Under the cgroup v2 Unified Hierarchy

Resource limits in cgroups are not quotas in the sense of a counter that refuses work when exhausted. They are policies applied at allocation time, and the difference shows up in what happens when a limit is reached: memory allocation triggers reclaim, then the OOM killer; CPU allocation triggers throttling until the next period. Understanding which of those paths a workload will take requires knowing the interface files and the order in which the kernel consults them.

One hierarchy, one file per knob

cgroup v2 replaced the per-controller hierarchies of v1 with a single tree, where each cgroup has interface files for every controller enabled on it. A container’s cgroup is a directory in that tree, and its limits are the values in memory.max, cpu.max, pids.max, and their siblings. Controllers are enabled by writing to cgroup.subtree_control in the parent, and a controller is only active on a cgroup if the parent enabled it: a limit written to a file whose controller is not enabled is rejected rather than silently ignored, which is one of the improvements over v1’s proliferation of separate mounts.

The “no internal processes” rule is the structural constraint. A cgroup with child cgroups enabled in subtree_control cannot have processes of its own; processes live only in leaves. This forces a clean tree where accounting rolls up from leaves to root, and it is why a runtime places the workload in a leaf cgroup and the runtime’s own supervisor in a sibling.

Memory: three thresholds, one direction

Memory in cgroup v2 is accounted per-page to the cgroup that first faults the page in, and it stays charged there until reclaimed or until the page is unmapped everywhere and freed. Shared pages, such as those from a shared library mapped into many containers, are charged to the first toucher. That accounting choice is why a page cache-heavy workload’s memory.current can exceed the sum of its anonymous memory: page cache is charged too, and it is reclaimable.

Three files matter.

memory.max is the hard limit. When a charge would exceed it, the kernel first attempts reclaim within the cgroup: it writes back dirty pages, drops clean page cache, and swaps out anonymous pages if swap accounting is enabled. If reclaim cannot free enough, the allocation fails and the OOM killer selects a victim within the cgroup. The victim is chosen by a heuristic over the cgroup’s processes, weighted by their memory usage and by oom_score_adj; it is not necessarily the process that made the allocation.

memory.high is a throttle, not a wall. Charges above it cause the allocating task to be throttled and reclaim to be run in its context, so the cgroup is pushed back under the threshold rather than being killed. It is the mechanism for keeping a container under pressure without terminating it, and its cost is latency: the task sleeps in reclaim.

memory.low and memory.min are protections in the other direction. memory.low makes the cgroup’s memory preferentially un-reclaimed when an ancestor is under pressure; memory.min makes it completely unreclaimable. They matter in a tree where a parent’s limit is shared among children: without protection, a sibling’s allocations can reclaim a working set that another sibling needs.

memory.peak records the high-water mark since the file was last reset, which is the number to read when deciding whether a limit is sized correctly. memory.events counts the times each threshold was hit, including oom and oom_kill, so a container that never died can still be shown to have been under memory pressure.

The subtlety is that reclaimable page cache makes memory.current a poor proxy for what a workload needs. A container that reads a large file once fills its cgroup with clean page cache, which counts against memory.max until reclaim drops it. A workload sized by its anonymous memory alone can therefore hit memory.max while its actual live data is far smaller, and the observed behaviour is a slowdown from reclaim rather than a kill.

CPU: bandwidth and weight are different controls

The CPU controller exposes two independent knobs that are frequently confused.

cpu.max is bandwidth control: two numbers, a quota and a period, both in microseconds. A value of 50000 100000 means 50 ms of CPU time per 100 ms period, which is half of one CPU. When a cgroup exhausts its quota within a period, all its tasks are throttled until the period boundary. Throttling is a hard stop: the tasks are runnable but not scheduled, and the cgroup’s cpu.stat records nr_throttled and throttled_usec.

cpu.weight is proportional share: a number from 1 to 10000, default 100, applied only when the cgroup is competing with siblings for CPU. An idle system gives a low-weight cgroup as much CPU as it asks for. Weight is hierarchical: a child’s share is computed against its siblings under the same parent, then that parent’s share against its own siblings, so a weight change deep in the tree does not affect competition at a higher level except through the parent’s allocation.

The interaction is where surprises live. A cgroup with a quota but a large weight still gets throttled at its quota; weight only decides who runs while quota remains. A cgroup with no quota and a small weight runs without limit when the machine is idle, and gets little when it is not. Setting both is normal: the quota bounds the worst case for other tenants, the weight shapes fairness during contention.

Throttling also has a burst-amplification effect. A workload that runs in short bursts can consume its entire period’s quota in a few milliseconds and then sleep for the rest of the period, which is fine if it has nothing to do. If it has work queued, it waits, and latency spikes to nearly a full period. The period is 100 ms by default, so the worst-case added delay is bounded by that, which is why latency-sensitive workloads sometimes use a shorter period and a proportionally smaller quota rather than a longer one.

Multi-threaded workloads meet the quota collectively. A cgroup with a quota of one CPU and eight runnable threads does not run one thread fast and seven idle; it runs whichever threads are runnable until the quota is gone, so the aggregate progress is one CPU’s worth, and individual thread latency depends on scheduling.

Pressure, not utilisation

Utilisation tells how much resource was consumed; it does not tell whether anything was waiting. The PSI files at /proc/pressure/memory, /proc/pressure/cpu, and /proc/pressure/io report stall time: the proportion of wall-clock time during which at least one task was stalled on the resource. Each line has some (at least one task stalled) and full (all non-idle tasks stalled), with two averages and a total in microseconds.

For memory, full is the interesting figure: it is the time tasks spent waiting in direct reclaim, which is time the workload made no progress. A container at 90% of its memory.max with no pressure is fine; one at 60% with rising full is thrashing on reclaim and will get slower as the limit is approached.

For CPU, pressure measures runnable-but-not-scheduled time, which is what throttling produces. A cgroup throttled by cpu.max shows cpu.pressure rising in step with nr_throttled, and the two together distinguish “the quota is too small” from “the host is oversubscribed”.

PSI is the signal to build autoscaling and admission decisions on, because it measures the symptom that users experience rather than the cause. It is also cheap to read: the kernel maintains the counters, and polling them is a file read rather than an aggregation over processes.

What the limits do not bound

The memory controller accounts for memory charged to the cgroup, and some kernel memory is not charged or is charged to a different cgroup. Slab objects allocated on behalf of a container are charged where the kernel decides, and socket buffers may be charged to the cgroup that created the socket or to the one that owns the receiving socket depending on the protocol and the kernel version. A container can therefore consume memory outside its memory.max accounting, which is one of the shared-state leaks described in The Shared Kernel Boundary and What Containers Cannot Isolate.

The PID limit (pids.max) counts tasks, including threads. A container with a limit of 100 and a runtime configured for 200 threads will fail to start the excess threads with EAGAIN, and the error appears as a thread creation failure deep in the application, not as a limit violation.

CPU quota bounds CPU time, not wall time. A process blocked on I/O, waiting on a lock, or sleeping consumes no quota. A container that appears to have spare CPU by cpu.stat may still be slow because of memory reclaim or I/O contention, and the cgroup files will not show it; that is what PSI and the io controller are for.

The hierarchy also bounds what a child can do, not what a parent can do to it. Limits are enforced from the parent downward: a child’s memory.max is capped by what its ancestors allow, but the parent can lower a child’s limit at any time, and the child cannot prevent it. Resource control is a property of the tree, and a container’s own view of its limits, filtered through the cgroup namespace described in How Seven Namespaces Compose Into a Container, is a view of a policy it does not own.