Testing Policy Before It Gates the Cluster

A policy is a function from a document to a decision, which makes it testable in the ordinary way: supply a document, assert the decision. The difficulty is not writing the assertions. It is that the document a policy receives in production is produced by a chain of defaults, mutations, and API server behaviour that the author did not write and often has not seen, and a test that supplies a hand-built document proves only that the policy agrees with the author’s mental model of that chain.

Policy testing is therefore two problems. The first is ordinary unit testing of rules, which catches logic errors cheaply. The second is conformance between the test input and the real input, which is where the interesting failures live.

Assertions have to name a decision, not an absence

A policy that returns a set of violations is easy to test badly. The natural assertion is “the result is empty for a good object”, and it passes for a policy that is broken in the most damaging way: a rule that never fires because its body is undefined for every input also returns an empty set. The test cannot distinguish “no violations” from “no evaluation”.

The fix is to assert on both sides of the boundary and to assert the content, not just the emptiness. A denied object should produce a violation whose message names the reason, so a test that expects a specific message fails when a rule stops matching rather than passing because the set is empty. A rule that silently stops matching is the failure mode that matters, because the policy stays deployed and the enforcement quietly disappears.

package platform.images_test

import data.platform.images

cases := [
	{"image": "registry.internal/base:1.4", "denied": false},
	{"image": "docker.io/library/nginx:1.27", "denied": true},
	{"image": "", "denied": true},
]

fixture(image) := {"review": {"object": {"kind": "Pod", "spec": {"containers": [{"image": image}]}}}}

test_denied_images if {
	some tc in cases
	tc.denied
	violations := images.deny with input as fixture(tc.image)
	count(violations) > 0
}

test_allowed_images if {
	some tc in cases
	not tc.denied
	violations := images.deny with input as fixture(tc.image)
	count(violations) == 0
}

The empty-image case is not decoration. An empty string passes a prefix check only if the check is written as a negation of a prefix match, and whether it does is exactly the kind of question a table of cases answers once and permanently. Degenerate inputs belong in the table for the same reason they belong in any parser’s tests: they are the inputs an attacker or a misconfigured template will produce.

Tests run against the policy source with a test runner that treats a rule named with the test prefix as a test case: the rule passes if it evaluates to a defined, true value and fails otherwise. Fixtures for data can be substituted per expression with the same mechanism used for input, which is what allows a policy that consults an inventory document to be tested without the inventory.

Fixtures have to come from the enforcing position

The unit test above supplies an object shaped like a pod. In production, the policy may receive a pod that has already been defaulted by a limit range, mutated by an injector that added a sidecar, and stripped of fields the tenant wrote because they are not in the schema. Every one of those transformations changes the document the policy sees.

Three habits close most of the gap.

Capture the request, not the manifest. The input to an admission policy is an AdmissionReview, and the interesting fields are the ones the manifest does not contain: the operation, the user, the dryRun flag, and oldObject on updates. A fixture built from a YAML file will never exercise the update path, which is where immutability rules and “field may not be removed” rules live.

Test the bundle that will be deployed. A policy directory tested from source can differ from the compiled bundle in its data files and in its entrypoints. Building the bundle, testing the built artifact, and deploying the same artifact removes a class of difference that is invisible in review. Signing the bundle and verifying the signature at load time extends the same idea to integrity: the engine refuses a bundle that was not produced by the expected key.

Pin the engine version. Evaluation semantics are stable in the large and not in the small; a policy that relies on a builtin or on a language feature behaves according to the engine that evaluates it, and an engine upgrade is a change to every policy at once. The version belongs in the test environment and in the deployment, and the two should match.

None of this proves the policy is right. It proves the policy behaves as written against documents that resemble the ones it will see, which is a weaker and much more useful claim.

Differential testing against recorded decisions

The question that matters before enforcement is not “does the policy pass its tests” but “what will it deny that is currently allowed”. That question can be answered empirically if decisions are recorded.

A decision log entry contains the input document and the result. Replaying a set of recorded inputs through a new policy version and comparing the results produces a list of requests whose outcome changes. The list is the review artifact: every newly denied request is either a violation that should have been caught all along or a false positive that would have broken a tenant.

# Replay recorded inputs through a candidate bundle, one decision per line
jq -c '.input' decisions.ndjson | while read -r input; do
  printf '%s' "$input" | opa eval --bundle candidate.tar.gz --stdin-input \
    --format pretty 'data.platform.images.deny'
done

The mechanics are simple; the discipline is in the sample. Recorded inputs are a biased sample of what the policy sees, weighted toward whatever was happening during the recording window. A policy that adds a rule about a resource type nobody created during that window will show no new denials and will still break the first tenant who creates one. Differential testing finds regressions in the paths that are exercised, and it is silent about the paths that are not.

The other limitation is data dependence. A policy that consults an external document produces different decisions when that document changes, and a replay against the current document does not reproduce the decision that was made at the time. Reproducing it requires the document as it was, which is a versioning problem for policy data rather than for policy code.

Shadow enforcement, and the promotion gate

The empirical version of the same idea runs in the enforcing position without enforcing. The policy is deployed in a mode where it evaluates every request, records the decision, and returns allow regardless. After a window, the recorded denials are the set of requests that enforcement would have rejected, with the actual objects attached.

This is the only method that observes the real input stream, including the defaulting and mutation that unit tests approximate. Its cost is a period during which the rule is not enforced, and its characteristic failure is that the period never ends: the rule stays in shadow because nobody owns the promotion decision, and the violations accumulate.

A promotion gate fixes that by making the decision explicit and bounded. The rule ships in shadow with an owner, a review date, and a threshold for acceptable denials. When the threshold is met, the mode flips; when it is not, the rule is revised. The threshold itself is a judgement about which denials are legitimate, and the only honest way to make it is to read a sample of them, which is why the shadow output has to preserve enough of each request to be readable.

There is a subtlety in what shadow mode measures. A policy evaluated in shadow cannot see the effects of its own denials, because there are none. A rule that would have prevented a later request from being valid, or that would have caused a controller to retry, produces a different world when enforced. The measurement is of the first-order effect only.

What coverage reports and what they miss

Test runners can report which rules were evaluated by the test suite, and the number is useful for finding rules that no test touches at all. It is not a measure of whether the tests are meaningful. A rule can be evaluated by every test and still be asserted only in the direction that passes.

The check that catches that is deliberate mutation: remove a deny rule, or invert a comparison, and confirm that at least one test fails. A suite where removing the rule under test changes nothing is a suite that is asserting the absence of a decision, and it will keep passing after the policy stops working. This is the same reasoning that makes a default-deny top-level rule necessary in the first place, applied to the tests rather than to the policy: the interesting property is not that a good request is allowed, but that a bad one is refused for the stated reason.

The boundary of all of this is the input, not the logic. A policy suite can be complete with respect to its fixtures and still be wrong about the world, which is why the fixtures have to be captured rather than imagined, and why the enforcement position described in what an admission webhook can decide determines what the fixtures can contain. A test that supplies a document the API server would never produce is a test of a policy that will never run.