Log Destruction, Rotation, and the Gaps It Leaves Behind

Deleting a log file is the cheapest anti-forensic action available to an intruder with administrative privilege, and the naive response, noticing an empty file, mistakes the artifact for the outcome. What matters is which records existed, where copies of them went, and whether the deletion itself is detectable. On a Linux host, all three questions have concrete answers rooted in how logging is implemented.

Three storage layers with different privileges

Modern Linux logging is a pipeline rather than a file. The kernel writes to the ring buffer (dmesg), which is a fixed-size circular structure in kernel memory. A logging daemon, traditionally rsyslog or syslog-ng and increasingly systemd-journald, reads from one or more sources and writes to disk. Applications write directly to the daemon’s socket, or to stderr, which the service manager may capture.

Each layer has distinct durability and distinct removal requirements.

The kernel ring buffer is bounded. Its size is a boot-time parameter, commonly measured in a few tens to a few hundred kilobytes, so old records are overwritten by new ones long before an attacker acts. It is also separate from the on-disk logs: wiping /var/log does not clear dmesg. Its content is readable while the host runs and is lost on reboot unless the daemon preserved it, which on systemd systems happens on shutdown and on some configurations during operation.

The on-disk text logs under /var/log are ordinary files owned by root and writable by root. An intruder with root can truncate them, unlink them, or rewrite them in place. Truncation and unlinking leave different residues: truncation preserves the inode, its timestamps, and its ownership, so the file’s metadata records the moment of the truncation; unlinking removes the directory entry but leaves the inode recoverable from the file-system image until the block group is reused, which is why the acquisition procedure in disk imaging and chain of custody matters more for this question than any live inspection.

The journal stores records in a binary format in a directory of files, with each file sealed by a hash chain when it is closed and rotated. journalctl --verify recomputes the chain and reports the first record where the chain fails or where the file is flagged as having been modified after sealing. The flag is set when a sealed file is subsequently opened for writing. The running daemon seals a file on rotation, and a modified sealed file is detectable. This is stronger than text-log integrity, but it protects against modification of closed files only. A file written to and then sealed by the daemon itself is indistinguishable from a legitimate one, and the daemon accepts records from any local process that can reach its socket.

What the accounting layer gives back

Whatever the log files say, the kernel and the file system maintain independent structures that record events without the logging daemon’s cooperation.

The audit subsystem, once a rule is loaded, emits records from the kernel to the audit daemon. If the daemon’s output file is destroyed, the in-kernel backlog may still hold recent events until it is drained, and the daemon’s own state reports whether it is running and whether events were lost. A host where the audit daemon was stopped and the log truncated retains the fact that the daemon stopped.

File-system metadata records modification times that a truncation updates and that a rewrite changes. A file creation time, where the file system supports one, is not settable by ordinary interfaces on all platforms; it is a weak but real cross-check against the mtime, and the interaction is developed fully in filesystem timestamps and timestomping.

Process accounting and last-style login records are written by different subsystems and therefore survive the destruction of one another. wtmp and btmp are append-only records written by login programs; an attacker who wipes syslog does not necessarily wipe them, and vice versa. Their formats are fixed-size and archaic, which is an advantage: appending fake entries requires knowing the exact structure, and gaps in the sequence are visible as a discontinuity in the record stream.

Forwarding is the only control that changes the outcome

Everything above describes recovery after the fact on a host the attacker controlled. The one control that changes the outcome is transmission off the host before the attacker acts.

Syslog over TCP with TLS (RFC 5425) delivers records to a remote collector with the stream framed and integrity protected in transit. Once written to the collector, the record is beyond the local attacker’s reach. Two properties of the transport matter for investigators:

Delivery is not acknowledged end to end. RFC 5424 defines the message format and the structured data element that can carry an origin identifier, but the protocol has no application-level acknowledgement that a record was persisted. A collector that accepts the TLS connection and then crashes loses records that the sender already considered sent.

Loss is visible in the collector. A gap in the collector’s record stream, bounded by the sending host’s sequence, is distinguishable from a period of no activity. This is why the origin metadata and a monotonic counter are more valuable than timestamps for this purpose: the counters make a gap an absence of specific records rather than an absence of events, and the clock skew argument applies to the timestamps that bracket the gap.

Local buffering complicates the picture. A daemon that cannot reach the collector spools to disk, and the spool is a local file with the same exposure as any other. If the attacker destroys the spool before the connection recovers, the records between the interruption and the destruction are lost with no remote trace, because the collector never saw a frame. Retention policy on the collector and a bounded spool size on the sender together determine the maximum recoverable window.

Distinguishing silence from absence

The investigative question after a log is found empty is whether the records were erased or never existed. Four checks answer it, in order of strength.

A rate check asks whether the host was otherwise chatty. A host whose collector shows a steady stream of records from neighbouring machines but nothing from this one during the same interval was either isolated, powered off, or suppressing. Other sources such as switch counters, authentication records at a central directory, or DHCP leases say which.

A collector-side gap check examines whether the sending host’s record stream is continuous immediately before and after the empty period, with a discrete break. A continuous stream that simply lacks the events in question is a much stronger statement than a break.

A counter check: journal and audit subsystems maintain sequence numbers and lost-event counts that a truncation of the output file does not reset if the internal state survives. A record numbering that jumps is an artifact of deletion; a numbering that is continuous and simply lacks entries is genuine absence.

A metadata check: inode timestamps on the log files themselves. A log file with an mtime recorded during the incident window, and a size of zero, is a truncation; the same file with a size of zero and a ctime from its last legitimate rotation is not.

None of these is conclusive alone, and the value of assembling them is precisely that they come from different subsystems with different privilege requirements. An intruder who wipes every one of them consistently is spending time and generating its own trail in the process, which is the property that makes detection engineering worth the effort: the destruction of records is itself a behavior, and behaviors can be detected.