Admission Control as the Enforcement Point for a Golden Path

A golden path that is not enforced is a suggestion. Enforcement has to sit somewhere the tenant cannot route around, and in a container platform that place is admission: the stage between an API request being authenticated and the object being persisted. Every object that enters the store passes through it, including objects created by other controllers, which is what makes it the only place where a platform-wide invariant can actually hold.

Two hooks with different powers

Admission runs as an ordered chain of plugins, some built in and some external. The external ones are the interesting ones: a webhook is an HTTPS endpoint the API server calls with the incoming object and a request context, and it replies with an admission decision.

Mutating admission runs first. A mutating webhook may return a patch to the object, and the API server applies it before continuing down the chain. This is how defaults are injected: the sidecar, the resource limits, the labels that tie a workload to a tenant, the security context that forbids privilege escalation. Because the patch is applied to the object before validation, every later validator sees the mutated object.

Validating admission runs after mutation. A validating webhook may only allow or deny. It cannot change the object. This restriction is deliberate: validation is the place for invariants, and an invariant that can be repaired by the validator is not an invariant but a default.

The consequence is that a rule can be written in either place, and the choice changes its meaning. A mutating webhook that “adds a label if missing” produces objects that satisfy the rule. A validating webhook that “rejects objects without the label” produces tenants who must satisfy it. The first is invisible to the tenant and the second is a contract clause; writing the rule in the wrong place yields either silent repair of a tenant error that should have been surfaced, or a rejection for something the platform could have supplied.

Why mutation order is undefined

Multiple mutating webhooks may be registered for the same resource. Their relative order is not guaranteed across the fleet, and each sees the output of whichever ones happened to run before it. Two webhooks that both write the same field produce a last-writer-wins outcome with an unpredictable winner.

This is not a scheduling bug to be worked around; it is the reason mutating webhooks must be written to be independent. The discipline is that each mutating webhook owns a disjoint set of fields, and where a webhook needs a value another webhook might supply, it must either supply it itself or tolerate its absence. A webhook that reads a field written by a sibling and branches on it will behave differently depending on registration order, and the difference will be rare enough to survive testing.

Reinvocation is the mechanism that partially addresses this. After all mutating webhooks have run, the API server may re-invoke the ones that declared they need it, so a webhook can observe fields added by later ones. It is a partial fix: reinvocation happens at most a bounded number of times, and a webhook that needs three rounds of cooperation with its siblings has a design problem, not an ordering problem.

Denial is a message, not a boolean

The most consequential design decision in an admission layer is what a denial says. A denial carries a message, a reason code, and optionally details. The message is read by a tenant at the moment their deployment fails, and it is the platform’s primary interface in the failure case.

A denial that says “admission webhook denied the request” is a support ticket. A denial that says which rule fired, which field violated it, what the observed value was, and what the acceptable values are is a self-service fix. The cost of the good message is that the policy has to carry the explanation as data rather than as an error string assembled in code, because the explanation must be consistent across every rule and every resource.

The same logic applies to the timing. Admission denies the request, so nothing is persisted, and the tenant sees the error at the API call. This is the cheapest possible failure point: no partial state, no rollback, no half-created workload. Contrast with a policy enforced by a controller after the fact, which must detect the violation, then delete or quarantine an object that has already been scheduled and possibly started. The trade is that admission is a synchronous dependency on the write path: a webhook that is slow makes every write slow, and a webhook that is down makes writes fail depending on its failure policy.

Failure policy and the availability of the write path

Each webhook declares a failure policy: Fail or Ignore. Fail means the request is rejected if the webhook cannot be reached or returns an error. Ignore means the request proceeds as if the webhook had allowed it.

The choice is a direct trade between safety and availability. A validating webhook for a security invariant must be Fail, because Ignore converts an outage of the webhook into an outage of the invariant: during the window, any object can be created. A mutating webhook that adds a convenience label can reasonably be Ignore, because the object remains valid without the label.

The operational consequence of Fail is that the webhook becomes a component whose availability bounds the platform’s write availability. It needs the same treatment as the API server: multiple replicas, a disruption budget, a short timeout, and a scope narrow enough that it does not see traffic it does not need. Timeouts are typically in the low seconds; a webhook timeout of thirty seconds means every write can be delayed by thirty seconds during a partial outage, which turns a degraded webhook into a stalled platform.

Matching rules bound the blast radius. A webhook scoped to one resource type and one namespace selector sees only those requests. A webhook scoped to everything sees every write in the cluster, including the writes made by the platform’s own controllers, which is how a platform ends up in a deadlock where its controllers cannot create the objects their own webhook requires.

Policy as data versus policy as code

A validating webhook can embed its rules in a compiled handler, or delegate to a general policy engine that evaluates a declarative rule set against the request. The difference is who can change a rule and how a change is reviewed.

Embedded rules change with a deployment of the webhook. A rule change is a code change, reviewed by the platform team, released on the platform’s schedule. This is appropriate for invariants the platform owns and no tenant should influence.

A policy engine reads rules from a store, often as objects in the cluster itself. Rule changes then become data changes, which can be reviewed by a different group and applied without a deployment. This is appropriate for rules that tenants or a security function need to adjust, and it introduces a distinct risk: the policy store becomes part of the enforcement path, and a malformed rule can deny everything. A policy engine whose rules are themselves admitted by a validating rule is the usual mitigation, so a syntactically valid but semantically catastrophic rule is caught before it takes effect.

What admission cannot do

Admission sees a request. It does not see the cluster, and it does not see the future. Three limits follow.

It cannot enforce a global constraint that depends on the current aggregate state unless it queries that state, and a query per admission makes the webhook a bottleneck and a cache-coherency problem. A rule like “no more than fifty replicas of this class cluster-wide” is enforceable only approximately under admission, because two concurrent requests can each observe forty-nine.

It cannot validate a value that only exists after the object is scheduled. A rule about which node a workload lands on is a scheduling constraint, not an admission constraint.

It cannot repair a tenant’s intent. A mutating webhook that silently changes a requested value produces an object that differs from what the tenant asked for, and the tenant will debug against their manifest rather than against the stored object. Defaults are fine; overrides are a trap. Where the platform must override, the object should carry a record of the original and the override, so the difference is discoverable.

The relationship between what admission enforces and what the path declares is the subject of what a golden path guarantees: admission is where a declaration becomes binding. The lifecycle problem of changing an admission rule once many objects already exist is handled in propagating template changes across an instantiated fleet.