What an Admission Webhook Can and Cannot Decide

A policy that rejects a bad object is only useful if it runs before the object exists. That requirement fixes its position in the write path: after authentication has established who is asking, after authorization has established whether they may write at all, after defaulting has filled in fields, and before the object is persisted or its effects are visible to anyone else. Admission control is that position, and its placement explains both what it is good at and the class of rules it structurally cannot enforce.

The important consequence is easy to miss. Admission sees one request, at one instant, with no ability to look at the cluster atomically and no ability to roll back another request it has already approved. Rules that need a consistent view of several objects, or that depend on state outside the API server, are not admission rules no matter how they are written.

The request object is the whole world

A validating webhook receives an AdmissionReview containing the object being written, the operation (CREATE, UPDATE, DELETE, CONNECT), the user and group information of the requester, and — on updates — the previous version of the object. It also receives an options field whose contents depend on the operation, and a dryRun flag.

The oldObject field is what makes update-specific policy possible at all: “labels may not be removed”, “the storage class may not change”, “replicas may not decrease below the current value” are all comparisons between the new and old object. Without it, every update would have to be evaluated as if the object were new, and a policy that forbids a field from being set would also forbid it from remaining set.

Several things are conspicuously absent. There is no transaction handle, so a webhook cannot read two objects and be sure they will still hold those values when the write commits. There is no cluster-wide snapshot. There is no ordering guarantee between webhooks at the same stage beyond the order the API server happens to invoke them, so a webhook cannot assume another has already run. And there is no retry semantics tied to the object’s identity: a rejected request is simply rejected, and the client decides whether to try again.

Defaulting happens before admission, which matters more than it sounds. A policy that requires resources.limits.cpu to be present may never see an object missing it, because a limit-range defaulting plugin filled it in first; conversely, a policy that rejects objects where the field is absent is untestable through the normal path. Policies must be written against the object as it arrives, not against the YAML the tenant believes they submitted.

Mutation first, validation second, and the cost of that order

Mutating webhooks run before validating ones. A mutating webhook can inject sidecars, add labels, rewrite image references, or attach default tolerations, and the validating webhooks that follow see the mutated object. This ordering is what allows a platform to enforce a property it also injects: the injector adds the required annotation, and the validator rejects any object that reaches it without one, which catches the case where the injector was bypassed by a namespace selector.

Mutation is also where the platform’s implicit contract with tenants is written. An injected sidecar changes the resource request of the pod, the network identity of the workload, and the failure domain of the container. A mutating webhook that is unavailable either blocks all matching writes or silently stops injecting, depending on its failure policy, and the second case produces workloads that run without the proxy, the log shipper, or the credential sidecar that the rest of the platform assumes exists.

Ordering inside a single object’s admission is not the same as ordering across objects. Two webhooks that both mutate the same field produce a last-writer-wins result with no diagnostic, which is a common source of a field that “keeps reverting” to a value nobody wrote.

Failure policy is an availability decision

Every webhook declares what happens when it cannot be reached or returns an error. Fail rejects the request; Ignore allows it. Neither is a neutral default.

Fail makes the policy engine a hard dependency of the API server’s write path for every matching object. If the webhook’s backing service is down, or its certificate has expired, or its timeout is too short for its own latency, then no matching object can be created — including the objects that would restore the webhook’s dependencies. A policy service that requires a database, and whose own pods are subject to admission, has a circular dependency that must be broken deliberately, usually by excluding the policy’s own namespace from its matching rules.

Ignore inverts the problem: during an outage, objects that violate policy are admitted, and the violation is discovered later, if at all. The window is exactly as long as the outage, and the objects created in it are indistinguishable from legitimate ones afterwards unless the policy records its decisions.

Timeouts are the third lever. A webhook that calls an external system for every admission makes the API server’s write latency include that system’s latency, multiplied by whatever retry the client performs. Timeouts are configured per webhook in seconds and are bounded; a policy that cannot answer within that bound is a policy that will fail or be bypassed under load. This is the argument for evaluating rules in-process where the language allows it — the API server can evaluate a compiled expression without a network hop — and reserving webhooks for rules that genuinely need an external decision.

What admission cannot see, and the rules that belong elsewhere

Four classes of rule fail at admission for structural reasons.

Cross-object invariants. “The sum of replicas across all deployments in this namespace must not exceed the namespace’s allocation” requires reading every deployment. A webhook can list them, but the list is a snapshot taken outside any transaction, so two concurrent creates can each observe room for one and both be admitted. The sum is then over budget and no single request is at fault.

Rules about objects created by controllers. A deployment does not create pods; the deployment controller does, and those pods pass through admission as a different user with different labels. A policy that inspects the tenant’s submitted object sees the deployment, not the pods it will eventually produce. Enforcing a property of the pods requires either mutating the pod template at admission of the deployment, or admitting the pods themselves with a rule that recognises the controller as the requester.

Rules requiring external truth. Whether a container image is signed, whether a DNS name is available, whether a cloud quota has room — none of these are visible in the API server. A webhook can ask an external service, at the cost of coupling the write path to that service’s availability and of accepting that the answer may be stale by the time the object runs.

Rules about the future. Admission approves a declaration, not its consequences. A deployment that will fail to schedule, a service that will select no pods, a job that will never complete — all are admitted correctly and fail later. Detecting them requires a controller that observes the resulting state, which is the reconciliation loop described in level-triggered reconciliation.

The division of labour that follows is fairly stable. Admission handles properties of a single object that can be decided from the object plus its previous version: required labels, allowed registries, forbidden host paths, immutable fields, resource limits. Controllers handle anything that needs a second observation: quota accounting, cross-object sums, external resource existence, and the eventual state of a rollout. Policy languages that express the second class are still useful, but they are evaluated by the controller against its own view, not by the API server at write time — which is why the same policy text can behave differently in the two positions, a difference covered in policy evaluation semantics.

The audit problem

A rejected request leaves a trace in the API server’s audit log, but an allowed request leaves only the object. Six months later, the question “why was this admitted” has no answer unless the policy recorded its decision. Decision logs, emitted per evaluation with the input and the result, are the only durable evidence that a policy ran and what it saw. They are also the input to the shadow-mode comparison that makes it safe to turn a new rule from warn to enforce, which is the mechanism behind testing policy bundles before they gate writes.

The failure mode to watch for is a policy that becomes permissive without anyone editing it. A rule that returns no violations when a field is missing, combined with a mutation that stops adding that field, produces an allow where a deny was intended. Admission is a good place to enforce a rule; it is a poor place to discover that the rule stopped meaning anything.