How a Configuration Value Resolves: Precedence, Merge Modes, and Missing Keys

A configuration value is rarely written once. It is written in a common layer, overridden for a group of machines, overridden again for a single machine, and possibly set from the command line. Something must decide which of those wins, and the rule that decides is the part of a configuration system that generates the most surprise, because the rule is almost always a total order over many sources rather than a small set of precedence levels.

A hierarchy is a lookup path, not a tree of files

The data layers are usually presented as a hierarchy, and the hierarchy is best understood as an ordered list of sources that the system consults, not as an inheritance tree in the object-oriented sense. The distinction matters because inheritance implies that a child can override a parent selectively; an ordered list implies that a source either supplies the value or does not participate.

Consider a lookup for a database endpoint. The system might consult, in order: a per-node source, a source keyed on the node’s role, a source keyed on the environment, and a global default. The first source that has the key supplies the value, and the rest are not consulted. Nothing below the winner is merged with it, and nothing above it can influence it. Changing the environment source has no effect on any node for which the per-node source defines the key, which is correct and is also the reason a fix applied at an inner layer can appear to be ignored.

This shape is why tools ship a precedence list as a documented, versioned artifact rather than as a convention. Ansible’s variable precedence is a numbered list of more than twenty categories, from role defaults (weakest) through inventory group variables, host variables, play variables, and set_fact results, with extra-vars at the top. Puppet’s Hiera uses an ordered hierarchy of lookup paths, and the node’s own classification is what determines which paths apply. The number twenty is not a wart; it is what happens when every mechanism that can carry data needs a defined rank relative to every other. The practical consequence is that a value’s origin can only be determined by consulting the list, and that the list is the authority on conflicts.

First-found, merge, and the choice that changes results

The lookup can stop at the first source that has the key, or it can continue and combine all sources that have it. Both modes are offered by mainstream tools, and they disagree in ways that are easy to construct.

With first-found, a list defined in two layers resolves to whichever layer ranks higher. With merge, the lists concatenate. For a value that represents a set of allowed origins, concatenation is nearly always wrong: an inner layer that wants to narrow the set cannot do so, because the outer layer’s entries remain. The author has to switch the key to first-found semantics, and the switch is a property of the lookup call, not of the data.

Hash values have a third option: a deep merge that recurses into nested keys. Deep merge is the right default for a settings dictionary where each layer adjusts a few keys, and it has an edge case worth knowing: a nested key that a lower layer sets cannot be removed by a higher layer, only overwritten with some other value. Removing a key requires either first-found semantics for that key or an explicit sentinel that the consuming code interprets as absence. There is no general merge semantics in which a higher layer can express “this key should not exist”, which is why configuration systems end up with an explicit null convention rather than relying on omission.

Arrays merge in one of several modes depending on the tool: concatenate with duplicates, concatenate and deduplicate, or replace. The choice is visible to consumers: a list of package names that deduplicates will silently hide a duplicate that a human intended as an error, and a list of mount points that concatenates in layer order will produce a mount order that depends on hierarchy depth rather than on any explicit priority.

Value kind	First-found	Deep merge	Practical default
Scalar	winner takes all	same	first-found
List of permissions	replace	concatenate	replace at the leaf
Settings dictionary	replace whole map	per-key override	deep merge
Nested dictionary with removal	replace whole map	cannot remove	first-found

Undefined, empty, and default are three states

A lookup for a key that no source defines is not the same as a lookup that finds an empty value. Most systems preserve the distinction at the data layer and lose it in the consumer.

A key that is absent resolves to a default supplied by the lookup call, or raises an error if none is supplied. A key present with an empty string, an empty list, or an explicit null resolves to that value and does not consult the default. This is the same semantics as a SQL column that is NULL versus one that contains an empty string, and it produces the same class of bug: a consumer that writes if not value treats an empty list and a missing key identically, and the two cases usually need different handling.

The language-level analogue is parameter expansion in POSIX shells, where ${var-default} substitutes when the variable is unset and ${var:-default} substitutes when it is unset or empty. Both forms exist precisely because the two conditions are different, and the choice between them is a decision the author has to make explicitly. Configuration languages that offer only one form have decided the question for the author, which is fine as long as it is documented and consistent.

Defaults that come from the code rather than from the data layer are the transitive case. A role whose manifest has a parameter with a default value supplies that default for any node whose data layer omits the key, and the default is not visible in the data at all. The declaration list for a node then depends on a value that appears in no data file, and changing the role’s default changes every node that did not set the key. Automatic parameter lookup makes this more common, since a role parameter that is not passed explicitly is resolved from the hierarchy, and the hierarchy’s silence means the manifest’s default applies.

Diagnosing resolution

The practical problem is not understanding the rules but determining which rule produced a particular value on a particular host. Three techniques cover most cases, and all three depend on the system being able to report its own decisions.

First, ask for the resolved value together with its source. Tools that support this return the winning layer’s identity, which immediately distinguishes “the override did not apply” from “the override applied and was later overridden”. Without source reporting, the diagnosis is a manual walk of the precedence list.

Second, evaluate the lookup where it is used rather than where it is defined. A value that is correct at the top level can be shadowed inside a scope, because several precedence categories are function-local: a variable set inside a block, a role, or a task outranks the same name set globally. Name collisions across scopes are the most common cause of a value that “is set but has no effect”.

Third, pin the ambiguity out of existence. A key that is only ever set in one layer cannot be misresolved, and a hierarchy with three layers has far fewer interactions than one with six. The cost of the hierarchy is paid at diagnosis time; it is a debt incurred by the author who added the layer that later nobody can explain.

The design constraint behind all of this is that layered data exists to express genuine structure: many hosts share most of their configuration and differ in a few values. The alternative, one data file per host, duplicates the shared parts and guarantees that a change to a shared value must be edited many times. The hierarchy removes that duplication and introduces the ordering question. Systems differ in how they answer it, and the ranking is what a reader should look up first when a value is not what the declaration appears to say. Where the resolved value is a per-host difference rather than a shared one, the comparison between hosts becomes its own problem, which is what environment parity as a measurable claim addresses.

A concrete failure shows how the three techniques combine. A service fails to start on one host in a group, with an error naming a port already in use. The declaration for the group sets the port, the per-host data for the failing member sets nothing, and the role default is the same as the group value. Asking the tool for the resolved value with its source returns the group layer, which rules out both the default and an override. Evaluating at the point of use shows a task-local variable of the same name set to a different port earlier in the run, which outranks the group layer and which no data file contains. The fix is either to rename the local variable or to stop setting the port in two places whose ranks are not adjacent in the list. The information that made the diagnosis possible was the rank of the local scope relative to the group scope, which is exactly what a documented precedence list supplies and an undocumented one does not.

The corresponding discipline for authors is to treat every new layer as a permanent addition to that list. A layer added to solve one override is consulted for every key on every host for as long as the configuration exists, and its rank determines whether it is consulted before or after every other layer. A hierarchy that grows one layer per incident ends up with ranks that no single person can enumerate, which is the point at which the resolution rules stop being a specification and become folklore. Keeping the number of layers small, and keeping each layer’s purpose distinct enough that its rank is obvious from its name, is the only defense that scales with the system.

The hierarchy is also only one of two places a process gets values from. Values supplied by the platform at start-up sit outside it entirely, with their own binding rules and their own defaults, which is where the boundary described in externalised configuration and immutable artifacts draws the line: the hierarchy resolves the configuration that is rendered into files at apply time, and the platform supplies the values that must differ without a rebuild. A value present in both is resolved twice, by two systems with different precedence rules, and the second resolution wins without appearing in any precedence list.