Quota Accounting Lags Admission, and What That Permits

A quota is a promise that a namespace cannot consume more than some amount of a resource. The promise is enforced by rejecting writes, and rejecting writes requires knowing current usage at the moment of the write. That knowledge is not free, and it is not instantaneous: usage is derived from the objects that exist, and the objects that exist are changing while the derivation runs. Every quota system therefore has a window in which its accounting is behind reality, and the behaviour inside that window is what determines whether the quota is a hard limit or a strong suggestion.

The number being checked is a derived figure

A resource quota object has two halves. The declared limits are configuration: a maximum count of pods, a maximum sum of CPU requests, a maximum number of load balancer services. The observed usage is status: a running total, maintained by a controller that lists the objects in the namespace, filters them by the quota’s scopes, sums their declared requests, and writes the result back.

Admission consults that status. It does not sum the namespace itself on every request, because doing so would mean listing every object in the namespace for every pod creation, which is quadratic in the size of the namespace and would dominate the write path. The design is the standard one for a hot path: maintain a derived value out of band and read it cheaply.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-budget
  namespace: checkout
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    count/pods: "200"
    count/services.loadbalancers: "2"

The declared values are absolute, so the quota expresses a ceiling rather than a rate. There is no notion of a burst allowance, and no notion of time: a namespace at its limit stays at its limit until something is deleted, regardless of how long it has been idle.

The check-then-act window

Two creations arriving at the same moment both read the same usage figure. If the figure says there is room for one, both are admitted, and the namespace ends up over its limit by one. This is not a bug in any particular implementation; it is the shape of the problem. The admission path can serialise checks against a single quota object, which removes the simultaneous-read case for objects that go through the same path, but serialisation does not remove the window between admitting an object and the accounting controller observing it.

Implementations narrow the window in two ways. The first is to make the check and the accounting share a lock or a queue, so an admission decision for a namespace waits until the accounting for the previous decision has been applied. The second is to delay admission when the accounting is known to be behind — the request is held briefly rather than decided against a stale figure. Both reduce the overrun; neither eliminates it, because the accounting is ultimately recomputed from a snapshot of live objects and the snapshot has a boundary.

The practical consequence is that a quota should be set with a margin. A team with a quota of forty CPU cores can briefly hold forty-one, and a platform that sizes its aggregate capacity at exactly the sum of the quotas has sized it below what the quotas permit. The overrun is bounded by the number of concurrent admissions the window can contain, not by the quota value, so the margin does not need to be proportional — but it does need to be non-zero.

Defaults and scopes decide what counts

A pod that declares no CPU request consumes nothing from a requests.cpu quota, which makes the quota trivially bypassable. Two mechanisms prevent that, and they interact.

A limit range in the namespace supplies defaults for containers that omit requests and limits. Defaulting runs before quota admission in the standard plugin order, so the pod arrives at the quota check already carrying the defaulted values and is charged for them. A namespace with a quota on compute resources and no limit range will reject pods that omit the fields outright, which is a clearer failure than silently charging nothing but a worse experience for the tenant.

Scopes change the set of objects a quota applies to. A quota scoped to non-terminating pods stops counting a pod the moment it enters a terminal phase, which means a namespace full of completed jobs does not appear to be at its pod limit. The same pod, under an unscoped quota, counts until it is deleted. Neither behaviour is wrong, and choosing between them is a statement about whether the namespace’s limit is about concurrency or about object count.

Priority classes add a third dimension. A quota can be scoped to a priority class, so that high-priority workloads have a separate budget from batch work. This is the mechanism that keeps a batch job from consuming the headroom reserved for latency-sensitive services, and it depends on the priority class being assigned by the platform rather than chosen by the tenant — a tenant who can set an arbitrary priority class can spend any budget.

Objects that count twice, and objects that do not count at all

The accounting is per namespace and per object kind, which produces several asymmetries that tenants notice.

A deployment does not consume quota. The pods it creates do, and they are created by the deployment controller under a different identity, passing through the same admission path. A tenant who creates a deployment with a replica count that exceeds the remaining budget gets a deployment that exists, reports a desired replica count, and produces pods that are rejected. The deployment’s status shows the failure as a condition, and the tenant’s original request succeeded. The platform’s interface has to make that visible, because the API did what it was asked and the outcome is still a failure.

A pod that is terminating still occupies its share until it is gone. Deletion is not instantaneous: the pod passes through a grace period, its containers stop, and the object is removed. During that interval the replacement pod and the terminating pod both count, so a rolling update of a namespace at its quota limit stalls — the new pod cannot be admitted until the old one is deleted, and the old one is not deleted until the new one is ready. The standard remedies are a quota with headroom for one extra replica, or a maxSurge of zero on the workloads, and both are policy decisions the platform should make rather than leaving to each team.

Cluster-scoped objects are outside the namespace quota entirely. Persistent volumes, node ports, and cluster roles have no per-namespace ceiling, which is why platforms that offer storage or ingress as a service need a separate accounting for them. The quota object simply has no view of them.

Why a quota is a guardrail and not a scheduler

Admission-time quota answers one question: may this object exist. It says nothing about whether the cluster can run it. A namespace within its CPU quota can hold pods that never schedule because the nodes are full, and a namespace at its pod-count limit can be running workloads that use a tenth of the node capacity. The quota bounds the declared demand, not the delivered capacity, and the two diverge whenever requests are inaccurate.

The divergence has a direction. Over-declared requests make the quota bind before the cluster is full, which wastes capacity. Under-declared requests let the namespace consume more than its share of the nodes while staying inside its quota, which degrades its neighbours. Neither is detectable from the quota object alone; both require comparing declared requests against actual usage, which is a different measurement from the one quota maintains.

That is the argument for two layers with different failure modes. Quota admission is a fast, local check that bounds the blast radius of a runaway workload and makes cost attribution possible. Capacity management is a slower, global process that watches utilisation and adjusts the aggregate. A platform that relies only on quota will discover that its limits are either loose enough to be meaningless or tight enough to block legitimate work, because a single static number is being asked to serve both purposes. The declared request that the quota charges for is also what the scheduler uses to place the pod, so an inaccurate request distorts both layers at once; keeping the two measurements separate is the only way to tell which one is wrong.

Designing a guardrail that degrades predictably

Three properties make a quota system survivable.

The limit is visible before the request. A tenant who discovers the quota by having a deployment fail has been failed by the interface. Exposing current usage against the limit, in the same place the tenant requests resources, converts a rejection into a planning input.

The rejection is attributable. A quota rejection names the quota and the resource. A rejection caused by a policy that also rejects the same object is indistinguishable at the API level, and the tenant has to guess which rule fired. Separating the messages is worth the effort, because the remedies are different.

The margin is deliberate. Every quota has a window, and every window has a width. Setting the quota to the intended ceiling and then adding the platform’s aggregate capacity at exactly the sum of the quotas is the configuration that turns a bounded overrun into an unbounded one. The margin belongs in the capacity plan, stated as a fraction, and it should be revisited when the number of namespaces or the admission concurrency changes — which is also why quota changes deserve the same review as any other change to the platform’s guarantees, as discussed in versioning the platform interface.