Throttling I/O with cgroup Controllers and What the Limits Share

A control group’s I/O limits are expressed as a maximum rate in bytes per second, a maximum rate in operations per second, or a weight that governs how much of a busy device the group gets. The numbers look like resource reservations, and they are not. They are policy applied at a point in the block layer where the identity of the issuing process is already resolved to a group, where the request may be much larger than the operation the process performed, and where page-cache writeback happens in a context that is not the writer’s own. Each of those facts changes what a limit actually constrains.

The accounting point

Requests reaching the block layer carry a control group association. The association is set when the page or request is created, so the group that is charged is the one the issuing task belonged to at that moment, not the one it belongs to when the request completes. For a synchronous read, both are the same. For a buffered write, the bytes are charged when the dirty page is created, which is during the write call, while the physical I/O happens later on a flusher thread with a different identity. The controller keeps the association with the request so that the writeback is charged to the original group.

This is why limits bite at two different moments. A bandwidth limit is enforced when a request is issued to the queue, so it constrains physical device traffic. The dirty-page thresholds are global, not per-group: a group that writes at a high rate can push the whole system toward the dirty limit and cause throttling for everyone in the write path. Per-group dirty limits exist but are not universally configured, so isolation of write bandwidth does not automatically isolate the latency cost of writeback pressure.

Reads are simpler. A read that misses the page cache issues a request charged to the reader’s group, and a bandwidth limit delays it directly. A workload that reads mostly from cache is invisible to the controller because no device operation occurs to charge.

Rate versus weight

A rate limit is a ceiling on a group, enforced by a token bucket that refills over time. It applies regardless of whether the device is busy, so a group with a 100-megabyte-per-second cap gets at most that even on an idle device. The mechanism is delay: requests above the current allowance wait in the controller’s queue. Latency under the limit therefore degrades as a function of the request size and of how far ahead the group has spent its budget, and a group that writes in large requests can overshoot the nominal rate for a moment and then wait for a full refill interval.

A weight, by contrast, is a share of the device under contention. Groups are served in proportion to their weights when the device is saturated, and no group is limited when the device is idle. The classic fair-queueing question returns here: shares are defined per queue, so on a multi-queue device the arbitration is not a single global ordering. A group’s share is a property of the queue the requests land on, which depends on the CPU that issued them, and a workload spread across many CPUs spreads its requests across many queues.

This asymmetry matters for the intended use. Weights give isolation without wasting idle capacity. Rates give a hard guarantee of an upper bound, which is what a multi-tenant system usually needs to prevent one tenant from consuming an entire device, but they leave the remaining capacity unallocated unless something else is running.

Burst, and why the window matters

Token-bucket enforcement has two parameters, the rate and the burst size, and the burst parameter rarely gets attention until it causes a surprise. A refill interval is defined per controller — historically on the order of a hundred milliseconds. A group that has been idle accumulates one interval’s worth of tokens, and a group that is throttled spends them as a batch. An application that writes in intermittent bursts therefore sees its burst pass at full speed and then a stall for the remainder of the interval.

The stall duration is bounded by the interval, not the rate. Two groups with the same byte rate but different burst allowances show completely different latency distributions: the one with the larger burst completes its write faster and waits longer afterwards. For a latency-sensitive workload, the choice of burst size is the decision that determines whether the limit produces smooth pacing or a sawtooth pattern.

What the controller cannot see

Three classes of traffic sit outside the accounting. Direct I/O operations are charged, so they are not among them. The genuinely unaccounted classes are:

Memory-mapped writes, which dirty pages through the fault path. The accounting still attaches to the page, so the writeback is charged, but the moment at which the group can be throttled moves from the write call to the fault, and a program that maps a file and stores to it never sees a write request to be delayed. The same applies to a program that relies on msync for durability.

File system metadata operations, whose I/O may be grouped into a journal commit owned by a commit thread. The bytes go to the device on behalf of whichever transaction triggered the commit, and the charge attaches to the commit rather than to each individual operation. A group that creates and deletes many small files can generate substantial device traffic through transactions that do not look like the group’s own requests.

Swap and reclaim traffic, which is not attributable to the requesting group at all in the general case.

A fourth limit is physical: the queue depth of the device is shared. If one group is allowed a very large number of outstanding requests by its own concurrency, it can fill the device’s tags and the other groups’ requests wait even though their rates are within their own limits. The controller’s per-group delay counts queue slots as a resource, but a group that keeps many requests in flight is consuming them.

Fairness on the write path

The most misleading case for the model is a set of groups each writing buffered data. Each write is cheap because it lands in the page cache, and all of them are charged at the point of dirtying. The flusher then writes pages for all groups, and if the flusher’s throughput is below the sum of the groups’ nominal rates, the backlog grows until the global dirty limit is hit and every writer, in every group, is throttled by the same wall-clock computation. The limit the groups see is then not their own but a shared one, applied in proportion to their dirtying rates rather than to their configured maxima.

Fair, in other words, degrades to a proportional share of a shared bottleneck exactly when the bottleneck is memory rather than the device. This is the same phenomenon as the writeback burst described in What write() Actually Does Before Data Leaves the Process, now with a second layer of policy on top.

Configuring with the right target

The measurable quantity a limit should be chosen against is the device’s sustainable rate for the workload’s access pattern, not its peak. A flash device’s sustained write rate under garbage collection is below its burst rate, and the difference is exactly the copy volume described in Write Amplification and Garbage Collection in Flash Translation Layers. A limit set above the sustainable rate leaves the device doing reclaim on behalf of one group while another group waits, which is worse than a limit that accounts for the device’s real capacity.

Verifying that a limit is doing what it claims requires measuring at two levels: the device’s own completion statistics, which show what reached the medium, and the group’s own accounting counters, which show what was charged. The gap between them is writeback that has not landed yet or traffic that was charged elsewhere. Because the block layer merges and reorders requests, as described in Writeback Scheduling and Merging at the Block Layer, the number of operations the controller counts is not the number of device commands either. The counters are the truth about what was charged; the device statistics are the truth about what happened.