Drift Detection Without Reading Every Host: Sampling, Fingerprints, and Reconciliation

Drift is any difference between the declared state and the observed state that the apply loop did not produce. It is created by a person editing a file on a host, by a package upgrade that rewrote a configuration, by an out-of-band change through a cloud console, or by a process on the host itself. The apply loop cannot see drift until it inspects the resource, so the entire detection problem is a scheduling problem: how often to inspect, which resources to inspect, and what to do with what the inspection returns.

The cost model of inspection

The inspection unit is one resource, and its cost depends on what the provider must do to read the state.

A file’s metadata is one stat(2) per resource, but the tool must also know whether the content matches, which is one read and one hash over the file’s length when the content is declared. Hashing a 1 MiB file means reading 1 MiB, and a host with a thousand such files costs a gigabyte of reads per inspection pass, most of it from page cache but not all of it. A package’s state is a query to the package database, which is a parsed index; on a host with several hundred packages the parse dominates. A cloud resource is one API call with a round-trip of tens of milliseconds, and a fleet-wide pass over thousands of objects is thousands of round trips, which is both slow and a billable operation for providers that charge per request.

The pass therefore has a cost of roughly files × (stat + read) + packages × index_lookup + objects × rtt, which is dominated by whichever term is largest for the estate. Reducing the frequency of the full pass is the blunt instrument; reducing the cost per inspection is the effective one. The schedule that chooses the frequency is also the schedule that decides how long a host can be out of its declared state before anything notices.

Two levers reduce per-inspection cost without weakening the check. The first is metadata-first evaluation: if the declared content is a rendered template whose checksum is recorded in the catalogue, then a file whose mtime and size match the recorded values can be skipped, at the cost of missing a content change that preserved both. That is a trade of detection completeness for read volume, and it should be explicit rather than a default. The second is caching the inspection result within a pass, so that a resource inspected because it is a dependency is not inspected again.

Fingerprints over state, not hashes of files

A second approach abandons per-resource inspection for a hash over a description of the state subtree. The technique is the one Merkle trees use for authentication: a node’s fingerprint is a hash of its children’s fingerprints, so a comparison at the root proves equality of the whole subtree when the children are equal, and recursion finds the first differing leaf.

For configuration state, the tree is the directory the configuration system owns. Each leaf is a file’s content hash combined with its mode and owner; each internal node is a hash over the sorted list of (name, child fingerprint) pairs. The host computes its tree and sends one hash; the declared tree is computed from the catalogue or from the rendered outputs; the comparison either matches or it does not, and if it does not, the recursion identifies which subtree differs.

The strength of this approach is that only changed subtrees cost anything: hosts that have not drifted send one hash, and the cost is a full directory walk per pass plus the hash of the leaves, which is the same read volume as the direct inspection. The weakness is that the fingerprint proves equality with the declared tree only if both sides are computed the same way. Differences in the set of files considered part of the tree (an editor’s backup file, a lock file, a temporary created by a running process) produce a mismatch that is not drift, and the recursion then reports a leaf that no declaration covers. The tree definition, which is to say the exclusion rules, is therefore part of the specification and has at least as much effect on the result as the hash function.

There is also a distinction the fingerprint cannot make. A changed leaf is detected, but not what it changed to. For detection that is enough; for reconciliation, the tool still needs to render the declared state and compare it against the file, which is the per-resource diff. The fingerprint is a filter, not a substitute.

import hashlib, os, stat

def leaf(path):
    st = os.lstat(path)
    if stat.S_ISREG(st.st_mode):
        h = hashlib.sha256()
        with open(path, 'rb') as f:
            for chunk in iter(lambda: f.read(1 << 16), b''):
                h.update(chunk)
        payload = f"f:{st.st_mode & 0o7777:o}:{st.st_uid}:{st.st_gid}:{h.hexdigest()}"
    else:
        payload = f"o:{st.st_mode & 0o7777:o}"      # symlink or directory: type and mode only
    return hashlib.sha256(payload.encode()).hexdigest()

def fingerprint(root):
    entries = []
    for name in sorted(os.listdir(root)):            # sorted: directory order is not a contract
        p = os.path.join(root, name)
        entries.append(name + '=' + (fingerprint(p) if os.path.isdir(p) and not os.path.islink(p) else leaf(p)))
    return hashlib.sha256('\n'.join(entries).encode()).hexdigest()

The sorted call is load bearing: os.listdir returns entries in an order the filesystem chooses, and two hosts with identical content can return them differently. The symlink check is the other one: following a link into a directory would merge two subtrees and make the fingerprint depend on what the link points at.

Sampling and the latency of detection

Full passes at high frequency are expensive; low frequency means a long window in which drift is present and undetected. Between those, a schedule can be built from two observations.

Drift is not uniform. It concentrates on hosts that are actively administered, on resources that other agents write (package databases, container runtimes), and on paths that humans edit during incident response. A schedule that inspects the resources on the critical path more often than the rest gets most of the detection for a fraction of the cost, and the partition of resources into critical and non-critical is itself a declaration that belongs in the catalogue.

Drift detection is not the same as reconciliation. Detection can run continuously and cheaply if it only reports, and reconciliation can then be a separate, deliberate operation with its own change control. This split is the difference between a system that silently reverts an emergency change thirty seconds after it is made and one that records the deviation and lets an operator decide, which is the subject of emergency overrides and break-glass changes.

Sampling hosts rather than resources is the third lever and the weakest. Inspecting a random subset of hosts assumes drift is uniformly distributed, and it is not: a bad declaration applied to a hundred hosts produces drift on all of them, and a sample of ten finds it; a manual change on one host is found only if that host is sampled, which at a 10% sample takes ten passes in expectation. The exception is drift introduced by a failing shared dependency, which is exactly the case where per-host sampling works well.

Noise: drift that is not a defect

A detection system that reports too much becomes ignored, and most of the volume comes from a small number of predictable sources.

Files that the owning process rewrites. A certificate bundle, a generated index, a log file inside the declared directory. These differ from the declaration by design, and the declaration should exclude them rather than the detector suppress them.

Resources whose declaration is approximate. A configuration that declares a file’s content and also permits the program to write a state block into it will report drift on every pass. The fix is a separate resource for the mutable part, or a declaration that does not claim the whole file.

Time-dependent declared values. A declaration that renders a timestamp or an expiring token differs from any previously rendered value. The checksum comparison makes this visible; the timestamp mode hides it and makes the file never change. Both are wrong, and the fix is to remove the time dependence from the declaration.

Ownership and mode after a package upgrade. Installing a package can replace a configuration file with the package’s default, changing content and mode together. This is genuine drift, and it is the most valuable thing the detector finds, because the packaged default usually differs from the declared one in a way that matters.

The reconciliation half of the loop is a policy decision with three answers: revert to the declaration, accept the observed state by updating the declaration, or quarantine the host from the loop until a person resolves it. Automatic reversion gives an environment that returns to its declared state within one pass and also erases evidence of what was changed; acceptance ratifies whatever happened, which is correct when the change was an upgrade that the declaration should follow; quarantine stops the loop’s protection for that host entirely. Choosing per resource class, rather than globally, is what makes the loop’s behaviour predictable.

The measurement that matters for the whole loop is not the number of resources inspected but the time from a change to its detection. That is bounded above by the pass interval plus the inspection duration, and it is the number an operator should be able to state for each class of resource.