Observability · Logging

Severity Levels as a Filtering Contract

Why syslog orders severities from 7 downward, how numeric thresholds let a producer discard records before they are built, and what a level change does to a stream.

Every logging API has a level parameter, and almost every implementation gets one detail of it wrong at some point: the level is checked too late, after the message has been formatted. The consequence is that a disabled level still costs the string construction, the argument conversion, and often a heap allocation. A severity scheme is not just a taxonomy of importance; it is the mechanism that makes that early check possible, and its design determines how cheap a suppressed record can be.

The numeric ordering and why it is inverted

Syslog assigns severities 0 through 7, with 0 the most severe:

ValueNameMeaning in the specification
0Emergencysystem is unusable
1Alertaction must be taken immediately
2Criticalcritical conditions
3Errorerror conditions
4Warningwarning conditions
5Noticenormal but significant condition
6Informationalinformational messages
7Debugdebug-level messages

The ordering is inverted relative to the names: a higher number is less severe. This looks backwards until the filtering use is considered. A configuration says “log at severity 4 and above”, where “above” means numerically lower, and the check is severity <= threshold. The convention comes from the same lineage as errno and process exit codes, where zero means success or, in this case, maximum urgency. It survives because the arithmetic is convenient: a threshold is a single integer and the comparison is one instruction.

The practical consequence is that a severity threshold is inclusive of everything more severe. Setting a threshold to Warning keeps Emergency, Alert, Critical, Error, and Warning, and discards Notice, Informational, and Debug. The threshold never filters out more severe events, which is the property that makes it safe to raise in a hurry: raising the threshold can only lose detail, never lose an alarm.

RFC 5424 carries the severity inside the PRI field, computed as facility * 8 + severity, transmitted in angle brackets as a decimal integer. The facility occupies the high bits so that a receiver can extract both with a division and a mask. The framing note in Record Framing: Why a Log Line Needs a Delimiter Contract matters here: PRI is the first token of the record, so a receiver cannot parse severity until the record boundary is known.

The check must happen before the work

The essential implementation property is that the level test precedes argument evaluation. In a language with eager argument evaluation, a call such as log.debug("payload: " + serialize(obj)) performs the serialization regardless of the configured level. The fix is either a level test at the call site or a lazy argument.

Go’s log/slog package is a clear case because the API is explicit about it. A Logger has an Enabled method that reports whether a record at a given level would be processed, and the Debug, Info, and Warn methods consult the handler before building a record. Code that needs to guard an expensive argument writes:

if logger.Enabled(ctx, slog.LevelDebug) {
    logger.DebugContext(ctx, "request dump", "body", dumpBody(req))
}

The explicit guard exists because the variadic arguments are evaluated by the caller before the method is entered. The check is a single comparison against a stored threshold; the dump is a full traversal of the request body. Ordering these two operations is the entire performance question.

Languages with macros or lazy parameters avoid the call-site guard. A macro receives its arguments unexpanded and can compile the level check into a branch around the formatting code. A lazy parameter achieves the same by passing a closure that the callee invokes only after the check. Both approaches move the cost of the decision to the callee and keep the call site short; both also make it harder to see, when reading the call site, whether the expensive expression is conditional.

The comparison itself is cheap enough that a per-record branch is not a concern. What is not cheap is the string conversion, the map allocation, the timestamp read, and the lock or channel operation that publishes the record. A level check that saves only the comparison and not the formatting is worthless, which is why the check must be the first thing the logging function does, before any of those.

Where the threshold lives

A threshold can be fixed at compile time, read from configuration at startup, or changed while the process runs. Each has a different failure mode.

A compile-time constant produces code with no branch at all for disabled levels, because the compiler removes the dead path. It is the cheapest option and the least flexible; changing the level requires a rebuild and a redeploy, which is exactly the wrong property for an incident that needs more detail.

A startup configuration is the common default. The threshold is read once, validated, and stored. The failure mode is a value that is syntactically valid but semantically wrong: a threshold set to Debug on a service that handles high request volume can multiply log volume by an order of magnitude, and the effect is not visible until the disk or the ingestion quota fills. Validating that the configured level is one of the known names, rather than accepting an arbitrary string, catches typos but not the volume decision.

A runtime-adjustable threshold, usually exposed through an administrative endpoint or a signal, is the operationally useful form. Its cost is that the level is now shared mutable state read on every logging call, which requires either an atomic read or a synchronization mechanism. An atomic integer load is a single instruction and is acceptable on the hot path; a mutex acquisition per record is not. The second cost is that a change is not atomic with respect to in-flight records, so a stream around the change contains records at both levels. That is harmless for volume and confusing for a reader who expects a clean boundary.

The third cost is subtler: a threshold lowered to Debug captures the debug path of code that has not been exercised at that level recently, including its latent bugs. A debug statement that dereferences a nil pointer because it assumed an invariant holds will crash the process when the level is lowered. Debug statements are code, and code that runs only under an unusual configuration is code that has never been tested under that configuration.

Per-component levels and the multiplicative effect

Large programs rarely have one threshold. A noisy dependency and a critical subsystem need different levels, so the configuration becomes a mapping from component or module name to threshold. The lookup is a map access on the hot path, which is more expensive than an integer comparison but still cheap relative to formatting.

The design question is what a component name is and how it is matched. Options include the source file path, a declared category passed by the caller, and the package or module name. Matching by prefix against a hierarchical name allows a parent setting to apply to children, which is convenient and produces surprising results when two prefixes match the same component with different thresholds. The usual resolution is longest-prefix match, which must be stated explicitly because the alternative, first match in configuration order, makes the result depend on file ordering.

There is a compounding effect worth naming: thresholds are applied at every hop, and the effective level of a record is the intersection of all of them. A producer set to Debug, a shipper that samples at 1 in 100, and a storage backend that retains only Error produce a stream in which Debug records exist only as a sample and are then discarded. Reasoning about what is visible requires knowing all three, and a change at any one of them changes the visible set. This is why a level is best understood as a contract between adjacent hops rather than a global property.

Severity is not priority

The most common misuse is to treat severity as a proxy for operational urgency. They are different axes. A Warning about a deprecated configuration is not urgent. An Informational record about a certificate expiring in three days is urgent, and will be missed by any filter that discards Informational. A Debug record about a retry storm is urgent in a way that no severity scheme can express, because the severity of the individual record is low and only the rate is significant.

The reason severity cannot carry urgency is that it is a property of the event’s nature, assigned once at the call site, while urgency is a property of the system state at the time the event arrives. The same event can be routine at one moment and critical at another. Systems that need urgency express it separately: as a distinct field, as a derived condition over a rate, or as an alert rule that consumes the stream. Alerting on rate rather than on severity is the standard resolution, and it belongs to a different mechanism than the record’s level.

The second misuse is to give severity a meaning that depends on the reader. A record at Error that a consumer ignores because it is expected, and a record at Info that a consumer treats as fatal, both indicate that the level is being used to encode a policy rather than a property. The test for a correct level assignment is whether a reader who knows the system but not the specific incident would classify the record the same way. If not, the level is carrying local knowledge that belongs in a field.

Two guidelines follow from that test. Log at the point where the condition is detected, not where it is handled, because a condition handled three frames up has lost the context that made it severe. And log a condition once. A stack of handlers that each log the same error at Error produces one root cause and several consequences, and the consequences obscure the cause when the stream is read in order. Wrapping an error with context and logging once at the boundary is the shape that preserves the cause.

The retention consequences of a level decision are covered in Retention Policy as a Storage Budget, and the cost of a level change during an incident interacts with the collection path described in Backpressure and Loss at the Collection Boundary.

References

  1. RFC 5424: The Syslog Protocol, section 6.2.1 (Severity Values) — IETF
  2. POSIX.1-2017, Chapter 2, Definitions: syslog severity levels — IEEE Std 1003.1
  3. Go standard library: log/slog package documentation — Go project