Quota Arithmetic: Why a Shared Pool Cannot Promise Every Tenant Its Share

Every platform team eventually does this arithmetic: a node pool has some number of cores and some amount of memory, ten tenants each want a quota, and someone has to decide what number goes in each quota. The decision looks like division. It is not, because the quota bounds a declaration while the scheduler bounds placement, and the two are computed from different quantities.

Requests, limits, and what each one controls

A container declares a request and a limit for each resource. They are consumed by different subsystems and they mean different things.

The request is what the scheduler uses. When placing a pod, the scheduler sums the requests of all pods already assigned to each node and picks a node where the new pod’s request fits within allocatable capacity. Requests are therefore a reservation against the scheduler’s bookkeeping, not against the hardware. A node whose pods request 32 cores has no room for another pod requesting a core, even if the processes inside those pods are idle and the cores are physically free.

The limit is what the cgroup enforces. For CPU, the limit is a bandwidth quota: a cgroup that consumes more than its quota within a period is throttled until the next period, and the period is 100 ms by default. For memory, the limit is an out-of-memory threshold: a cgroup that touches more than its limit triggers the kernel’s OOM killer within that cgroup, not across the node. Memory limits are hard in a way CPU limits are not, because throttling delays work while OOM kills it.

The gap between the sum of requests and the sum of limits on a node is the overcommit ratio. A node with 32 cores where pods request 16 and are limited to 32 is overcommitted two to one on CPU. That is normal and intentional: most workloads do not use their limit continuously, and without overcommit a cluster would need capacity for the peak of every workload simultaneously, which is close to the sum of the peaks rather than the peak of the sum.

The arithmetic of fitting quotas to a pool

Given a pool of $N$ nodes with $c$ allocatable cores and $m$ allocatable bytes each, the scheduler’s capacity for reservations is $N \cdot c$ and $N \cdot m$, minus the daemonsets and system reservations that consume capacity on every node before any tenant pod is placed. Those platform overheads are not a rounding error: a node running a container runtime, a network daemon, a log shipper, and a monitoring daemon may have a meaningful fraction of its allocatable capacity already committed, and that fraction has to come out before dividing.

The naive quota is the reservation capacity divided by tenant count. For ten tenants on a pool with 300 allocatable cores after overhead, that is 30 cores each. If every tenant’s declared requests sum to exactly 30, the pool is exactly full, and the first tenant to need an eleventh replica is blocked. There is no headroom for the pod that is rescheduled after a node failure, no room for a rolling update that briefly runs old and new pods together, and no room for a tenant’s legitimate growth.

Headroom is therefore a term in the arithmetic, not an afterthought. A rolling update of a deployment with $r$ replicas and a surge allowance of one extra pod needs capacity for $r+1$ during the update. A node failure removes $c$ cores from the pool until the node is replaced, so the pool must tolerate losing one node without any tenant failing to reschedule. Both are derivable from the pool shape rather than guessed, and both reduce the number that goes into a quota.

The formula that results is not a division but a budget:

$$\text{quota}_i \le \frac{N \cdot c - \text{overhead} - \text{failure reserve} - \text{surge reserve}}{k}$$

where $k$ is the number of tenants expected to be at their quota simultaneously. That last factor is the one teams get wrong. If ten tenants each have a 30-core quota but only three are ever busy, the pool can be much smaller than 300 cores, because the quotas are admission bounds rather than reservations. Nothing prevents the platform from setting quotas that sum to more than the pool, and nothing prevents the scheduler from refusing to place the pods when they all try at once.

What happens when the sum of quotas exceeds the pool

This is the deliberate design point of most shared platforms: quotas bound a single tenant’s blast radius, and the pool is sized for the expected concurrent demand rather than the sum of the bounds. The trade is explicit. A pool sized for the sum of quotas never rejects a tenant’s legitimate request and is idle most of the time. A pool sized for concurrent demand uses capacity efficiently and produces a class of failure where several tenants scale at once and the later ones cannot be scheduled.

The failure is legible when it is reported properly: a pod stays pending with a scheduling condition naming insufficient capacity. It is illegible when the platform reports only “pending”, because the tenant cannot tell whether the problem is their request, their quota, or the cluster.

Two mechanisms make the trade safer. Priority classes let the platform define which workloads are displaced first when the pool is full, so a batch job yields to a tenant’s production service rather than competing with it. And preemption, where enabled, lets a higher-priority pending pod evict a lower-priority running one, which converts “cannot schedule” into “scheduled after evicting something less important”. Preemption has its own cost: the evicted workload is disrupted, and if its controller immediately recreates it, the two can trade places repeatedly unless the priority difference is large enough to make the outcome stable.

The memory asymmetry

CPU and memory need separate treatment because their failure modes differ. CPU overcommit degrades latency: a throttled cgroup runs slower, and the slowdown is proportional to the overcommit. Memory overcommit risks termination: a cgroup that exceeds its limit loses a process, and the process chosen is the one the kernel’s OOM scoring prefers, which may not be the one that caused the pressure.

For CPU, a defensible policy is to set requests at a realistic steady-state usage and limits well above it, accepting throttling at peak. For memory, the equivalent policy is dangerous, because the peak does not throttle, it kills. Memory limits should therefore be set above the observed high-water mark with a margin, and the platform should not assume that a memory limit will be respected gracefully.

The consequence for quota arithmetic is that memory quotas are closer to hard reservations than CPU quotas are. A tenant whose memory quota sums to more than the pool’s memory is not overcommitting a compressible resource; it is promising an amount of memory that does not exist, and the promise is broken by an OOM kill rather than by a slowdown.

Measuring whether the arithmetic held

The arithmetic above is a model, and models need checking against the fleet. Three signals show whether the model is right.

The ratio of allocated requests to allocatable capacity per pool shows reservation pressure. A pool at 90 percent allocated has little room for surge or failure; a pool at 40 percent is paying for idle capacity.

The ratio of actual usage to requests shows whether requests are honest. A fleet where usage is consistently a small fraction of requests is a fleet whose reservations are inflated, which wastes capacity in exactly the way the model was meant to avoid. A fleet where usage exceeds requests regularly is a fleet whose scheduler is placing pods on the assumption of a demand that does not exist, and whose nodes are relying on overcommit to survive.

The count of pending pods by reason shows whether the model’s failure mode is occurring. Pending for insufficient capacity means the pool is full; pending for exceeded quota means the tenant’s own bound is binding. The two have completely different remedies, and a platform that does not distinguish them will apply the wrong one.

The remaining piece is what happens to a tenant whose quota is genuinely too small: the escalation path, and the reason the quota exists at all, connect to tenant isolation in a shared platform. The way a preview environment is sized from the same arithmetic is covered in ephemeral preview environments.