Record Framing: Why a Log Line Needs a Delimiter Contract

A process writes a diagnostic message to a file descriptor. Somewhere downstream, another process reads a byte stream and needs to know where one message ends and the next begins. That second problem is framing, and it is logically prior to everything else a log system does. A record that cannot be delimited cannot be parsed, indexed, counted, or attributed to a level. Most of the awkwardness in log formats comes from the fact that framing and payload encoding are chosen independently and then have to be reconciled.

The ambiguity that forces a decision

Suppose records are separated by a newline byte, 0x0A. The writer emits a message, then a newline. A reader splits the stream on newlines. This works until a message itself contains a newline, which happens whenever a stack trace, a multi-line SQL statement, or a user-supplied string is logged. The reader now sees two records where the writer wrote one, and the second one begins in the middle of a sentence. Nothing in the stream marks the difference, because the delimiter is a byte value that is also a legal payload byte.

The general statement: a delimiter-based framing is unambiguous only if the delimiter cannot appear in the payload, or if the encoding guarantees that occurrences are transformed. Any framing scheme is therefore a choice about which of those two properties to buy, and what to pay for it.

There are three families in common use.

Delimiter with escaping. The payload is encoded so that the delimiter byte never appears in it. JSON Lines is the common example: a record is one JSON value per line, and RFC 8259 requires that control characters inside strings be escaped, so a literal newline inside a string is written as the two characters \ and n. The framing is unambiguous as long as the encoder is correct. The cost is a decode step and an expansion factor on strings containing many control characters.

Length-prefixed, or octet counting. Each record is preceded by its length. The reader reads the length, then reads exactly that many bytes. No byte value is reserved, so the payload is transmitted raw. The cost is that the reader must trust the length, and a corrupted length desynchronizes the stream until something else resynchronizes it.

Delimiter with an escape byte or a doubling rule. CSV uses this shape: fields are separated by commas, and a field containing a comma is enclosed in double quotes, with a literal double quote inside the field written as two double quotes (RFC 4180). The escape is not a backslash; it is a doubling, which keeps the encoding reversible without introducing a reserved character.

What syslog over TCP actually specifies

The syslog protocol is the clearest worked example, because the original design assumed datagrams and the stream transport was retrofitted.

RFC 5424 defines the message format but leaves framing to the transport. Over UDP, framing is implicit: one datagram is one message, and the transport preserves the boundary. RFC 5426 sets 2048 octets as the maximum message size for that transport, which is a hard ceiling on record length and the reason older syslog messages are terse.

Over TCP there are no message boundaries, so RFC 6587 defines two framings. Octet counting sends the message length in decimal ASCII, followed by a single space, followed by exactly that many octets. Non-transparent framing terminates each message with a trailer character, conventionally a newline, and requires that the sender ensure the trailer does not occur inside the message.

The interesting failure is in the second method: a syslog message begins with the <PRI> field, but a receiver that supports both framings cannot know which one it is looking at until it reads the first bytes. RFC 6587 addresses this by observing that a message beginning with a digit is treated as octet-counted, which means a non-transparent message whose payload happens to begin with a digit is ambiguous. The specification resolves it by requiring that such messages be sent with octet counting. This is a general lesson: when two framings share a stream, the first byte becomes a discriminator, and any payload that collides with the discriminator needs a rule.

The reader’s state machine

A correct reader is a small state machine, not a call to a split function. It accumulates bytes into a buffer, looks for a boundary, emits a record, and keeps the remainder. Two details separate a robust implementation from a naive one.

First, partial reads. A read on a socket returns whatever is available. A record may arrive in fragments, so the reader must retain the tail of an incomplete record across reads. The buffer for that tail is unbounded unless the reader imposes a maximum record size; without one, a sender that never emits a delimiter grows the reader’s memory without limit. Every practical implementation therefore has a maximum record size and a policy for exceeding it: truncate the record, or drop it and skip to the next delimiter.

Second, resynchronization. With newline framing, a truncated record is self-healing: the reader skips to the next newline and continues. With length-prefix framing it is not, because the next length field may be read from the middle of a payload. Recovery requires an out-of-band signal such as a connection reset, or a checksum and a scan for a plausible header.

A minimal octet-counting reader in Go:

func readFramed(r *bufio.Reader, max int) ([]byte, error) {
    // length is decimal ASCII terminated by a space
    n := 0
    for {
        b, err := r.ReadByte()
        if err != nil {
            return nil, err
        }
        if b == ' ' {
            break
        }
        if b < '0' || b > '9' || n > max {
            return nil, errors.New("bad length field")
        }
        n = n*10 + int(b-'0')
    }
    if n > max {
        return nil, errors.New("record exceeds limit")
    }
    msg := make([]byte, n)
    _, err := io.ReadFull(r, msg)
    return msg, err
}

The io.ReadFull call is the essential part. A single Read is allowed to return fewer bytes than requested, and code that assumes otherwise produces records that are silently truncated under load, when segment sizes are small.

Where framing leaks into the payload

Even a well-framed stream can be misread, because the record content is attacker-influenced in most systems.

Newline injection is the canonical case. If a request path, a header value, or a username containing a newline is interpolated into a log message, the resulting record contains a delimiter. In a newline-framed file, that forges an additional record: an attacker can write a line that looks like a legitimate entry from another component, with a chosen severity and timestamp. Encoders that escape control characters close this hole; hand-built format strings do not. The fix is not a filter on the input, because filtering requires enumerating every dangerous character across every consumer; it is an encoder that cannot emit a raw delimiter.

Terminal escape sequences are a second leak. A record is often displayed in a terminal, and a payload containing ESC [ sequences can move the cursor, recolor text, or overwrite previously displayed lines. A log viewer that writes raw bytes to a terminal is a rendering engine for untrusted input. Escaping non-printable bytes on display, rather than trusting the payload, is the only general answer.

Null bytes are a third. A record containing 0x00 is legal in a length-prefixed format but truncates in any consumer that treats the payload as a C string. This is a source of records that appear complete in one tool and half-missing in another.

Choosing a framing

The trade is between decoder complexity, payload fidelity, and recoverability.

Framing	Reserved bytes	Payload expansion	Recovery after corruption
Newline, unescaped	0x0A	none	skip to next newline
Newline, escaped (JSON Lines)	0x0A only outside strings	up to 6 bytes per control byte	skip to next newline
Octet counting	none	decimal length prefix, about 1 byte per 2 digits	none without a checksum
Trailer with byte stuffing	trailer and escape byte	1 byte per occurrence of either	scan for trailer

Newline-delimited JSON sits at a useful point: the delimiter is a single byte, the escape rule is already specified by the JSON grammar, corruption is self-healing, and the payload is readable with cat. Its cost is that every record must be a single JSON value and every string must be escaped, and that a large record is one very long line, which is awkward for line-oriented tools that assume a fixed maximum line length.

Binary formats such as CBOR remove the escaping cost and the numeric precision problems that JSON inherits, but they make the stream unreadable without a decoder and shift the framing problem to the container. This is why binary log formats usually come with their own record container rather than relying on a delimiter.

The choice is not permanent, but it is expensive to change: every producer, shipper, and parser encodes an assumption about where records end. That assumption is the contract, and the formats described in Structured Fields Versus Formatted Message Strings only work when the contract holds. Severity, discussed in Severity Levels as a Filtering Contract, is likewise a per-record field that requires the record boundary to be known before it can be read.

Limits worth remembering

A single record has a practical ceiling imposed by the transport, not by the format. UDP syslog caps at 2048 octets by specification, and anything longer is truncated or fragmented at the IP layer, where a lost fragment loses the whole datagram. Pipes and terminals have line-length conventions that vary by tool. Storage systems impose their own per-event limits, and a record larger than the limit is usually dropped at the ingestion boundary rather than stored partially.

The consequence is that a log call must have a size policy. A record assembled by concatenating an unbounded field, such as a request body or a full result set, will eventually exceed every limit in the path, and the failure is silent: the record is dropped at whichever hop first applies the cap. Truncating at the producer, with an explicit marker and the original length, keeps the record useful and makes the truncation visible.