Structured Fields Versus Formatted Message Strings

A conventional log statement formats its values into a sentence: failed to connect to db-17:5432 after 3 attempts. The sentence is readable and nearly useless for aggregation, because every distinct hostname, port, and attempt count produces a distinct string. A structured record keeps the same information as fields (host=db-17, port=5432, attempts=3, event=connect_failed) and renders the sentence only at display time. The difference is not cosmetic. It determines whether the record can be filtered by value, counted by dimension, and compared across producers.

What formatting destroys

A formatted string is a lossy projection of a structured event. Once values are interpolated, four properties are gone.

Type. attempts=3 is an integer; attempts=3 inside a sentence is a substring. A consumer that wants “attempts greater than 2” must parse the sentence, and parsing is ambiguous: a hostname may contain a colon, a message may contain a number that is not the attempt count, and a value may itself contain the delimiter used by the parser. Storing the value as a number preserves ordering and comparison without a grammar.

Boundaries. Field names tell a consumer which part of the record is which. In a sentence, the boundary between the constant text and the value is implicit in the template, and the template exists only in the source code of the producer. A consumer has no way to know that the token after “after ” is the attempt count.

Identity. Two records with the same event but different values have different strings. To count events by type, a consumer must group by the constant portion, which requires recovering the template. That recovery is the hard part: the values must be identified and removed, and without knowing which tokens are values, the removal is guesswork.

Additivity. Adding a field to a structured record does not disturb existing consumers; they ignore keys they do not use. Adding a field to a sentence changes the sentence, and any consumer that matched on the old text breaks.

The syslog family shows the intermediate design. RFC 5424 keeps the human-readable message in the MSG part, but adds a STRUCTURED-DATA element before it: a sequence of [id name="value" ...] groups, where id names a schema and the parameter names are defined by that schema. The structured part is machine-parsable and the message stays for readers. The constraint is that RFC 5424 requires parameter values to be UTF-8 with a restricted character set, in which ", \, and ] must be escaped, which is a reminder that even a structured encoding has to solve the delimiter problem described in Record Framing: Why a Log Line Needs a Delimiter Contract.

Templates and values as separate axes

The design that avoids both failure modes separates the record into a template and a set of typed values, and keeps the template identifier stable.

The template is the constant text with the variable positions marked. The values are a typed array. Rendering is a substitution over the template; parsing is unnecessary because the values are already separate. The template’s hash or assigned identifier becomes the event name, so grouping is a lookup rather than a text transformation.

import json, sys

def emit(event: str, template: str, **values) -> None:
    # the template id is stable; the rendered text is derived, never parsed
    sys.stdout.write(json.dumps({
        "event": event,
        "template": template,
        "values": values,
    }, separators=(",", ":")) + "\n")

emit("connect_failed", "failed to connect to {host}:{port} after {attempts} attempts",
     host="db-17", port=5432, attempts=3)

The rendered form is reproducible at any time from the template and values, so nothing is lost by storing the structured version. Note what the record does not contain: no pre-rendered sentence that a consumer might try to parse, and no formatting directives that depend on locale.

The cost of this design is a registry. Templates must be known to the query layer if the query layer wants to display them, which means either transmitting the template with every record (repetitive but self-contained) or registering it once and sending an identifier. The second is cheaper on the wire and introduces a dependency: a consumer that receives an identifier it does not know cannot render the record. Systems resolve this by shipping the registry as part of the ingestion pipeline’s configuration, and by treating an unknown identifier as an error rather than silently falling back to the raw values.

Cardinality is a property of the template, not the record

The reason templates matter more than any individual field is cardinality. A metric built from a log field has one time series per distinct value, and the number of distinct values is bounded by the domain of the field, not by the number of records.

Consider a field holding a user identifier. A service with a million users produces a million distinct values. A field holding an HTTP status code produces a few dozen. A field holding a hostname produces a number bounded by the fleet. A field holding a free-form error message produces a value for every distinct failure, which is unbounded.

The practical rule: fields that are dimensions must be low-cardinality and drawn from a known set; fields that are payload can be high-cardinality and must not be indexed as dimensions. A log record can carry both, but the storage layer must know which is which. Systems that index every field by default discover this when a high-cardinality field arrives and the index grows faster than the traffic.

Templates have the same property in a subtler form. A template with a value interpolated into the constant text produces a distinct template per value if the template is derived by string comparison rather than by declaration. This is why template identity must come from the source, as a literal string passed to the logging call, rather than from clustering rendered messages after the fact. Clustering is a recovery technique for uncooperative producers; it cannot distinguish a variable hostname from a variable error reason when both appear in the same position, and its output is approximate by construction.

Encoding choices and their costs

The field-value encoding is a separate decision from the framing, and the options trade readability against size and type fidelity.

Encoding	Type fidelity	Size per field	Readable without a decoder
JSON object	numbers, strings, booleans, null, arrays; no integer/float distinction beyond syntax	key plus quotes and separators	yes
CBOR map	integers of two widths, byte strings, tags	key plus a one- or two-byte header	no
Protocol Buffers	declared schema types, defaults elided	field number plus length	no
Logfmt	strings and untyped tokens	key, equals, value	yes

JSON is the default because it is readable and every language has a parser, but it has two real weaknesses. It has one number type, so a 64-bit integer beyond 2^53 loses precision in implementations that follow IEEE 754 doubles, which affects identifiers such as snowflake IDs and timestamps in nanoseconds. And it cannot represent a byte string, so binary payloads must be base64-encoded, inflating them by a third.

CBOR fixes the integer problem by encoding integers with explicit widths and fixes the binary problem with a byte-string type. Its cost is that it is not readable in a terminal and needs a decoder in every consumer. Protocol Buffers fix both problems and add schema evolution rules, at the cost of requiring the schema at both ends and producing a record whose fields are numbers in the wire format.

Logfmt is worth naming because it is often chosen for its simplicity: level=info msg="connected" attempts=3. It is easy to read and easy to parse with a small tokenizer, but it has no type system, so every value is a string and quoting rules for values containing spaces must be agreed by convention rather than by specification.

What a record should carry

Independently of encoding, a record needs enough context to be interpretable without the producer. The OpenTelemetry logs data model is a useful checklist because it separates the event from its context: a timestamp, an observed timestamp for when the collector saw it, a severity number and text, a body that may be structured, a resource describing the emitting entity, and a scope describing the instrumentation library. Trace and span identifiers are attributes rather than top-level fields, which allows a log record to be joined to a trace.

The reason to separate resource from record is size. The resource, comprising service name, host, region, and version, is constant for a long run of records, so an encoding that repeats it per record pays for it repeatedly. Columnar storage and per-stream dictionaries exploit this by storing the constant once; a JSON-per-line stream cannot, and a service emitting millions of records per hour pays for the repeated keys in bandwidth and in storage.

The fields that are cheapest to add and most expensive to remove are the ones that identify the code path: the source file and line, the event name, and the severity. They are known at the call site and are impossible to reconstruct later. Values that can be recomputed from other data, such as a derived duration, are better omitted than stored twice, because two copies of a derivable value can disagree, and the disagreement is discovered at the worst time.

Severity is treated as a filtering contract in Severity Levels as a Filtering Contract, and the storage consequences of these choices appear in Retention Policy as a Storage Budget.