Process Start Records as the Unit of Execution Evidence

An investigation that begins with “something ran on the host” needs three facts before any deeper analysis has value: what executed, what executed it, and whether the two were on the machine at the same moment. These sound trivial and are not. Operating systems keep the answers in places that are either deliberately disposable, expensive to record, or both, and the design of each record type determines which of the three questions can still be answered hours later.

The pivot is the process start record. Every process on a Unix-like system begins as the result of a fork (or clone, or posix_spawn built on them) followed by an execve. The kernel allocates a task structure, assigns it a pid, and records its parent’s pid and its start time in coarse units. That triplet, (pid, ppid, start_time), is the only stable handle to a process. Pids alone are worthless: the kernel reuses them, and on a busy host a pid can turn over to an unrelated process within minutes.

Why the pid alone cannot identify anything

The Linux kernel allocates pids from a bitmap and reuses the lowest available number. /proc/sys/kernel/pid_max defaults to the value that keeps pids in signed 16-bit range (32768); it can be raised to a much larger maximum on 64-bit systems. Either way, allocation is greedy from low to high, so on a machine that forks constantly, such as a shell running build tooling or a web server spawning workers, the wraparound period is short.

The consequence is that a log line reading pid=4812 is not a reference to a process, it is a reference to a number that happened to be attached to a process at the moment the line was written. Matching such a line against /proc/4812 later is a guess, and a wrong one the majority of the time. The kernel provides the disambiguator: /proc/<pid>/stat contains the start time of the process as field 22, the number of clock ticks after system boot at which the process started. Clock tick units are typically 1/100 second (the USER_HZ constant), so the value is coarse but far finer than pid reuse typically is.

Process lineage (ppid) is the second disambiguator, and it answers a different question: what caused this. A shell that spawned a network client is a different finding from a daemon that did so, because the first implies human interaction and the second implies a persistence mechanism. Lineage becomes genuinely useful when it is reconstructed as a chain, because attacker tooling frequently exhibits characteristic textures: a web server process that spawns a shell, a shell that spawns a base64-decoding interpreter. The anatomy of that texture is the subject of behavioral detection over indicator lists.

What the kernel keeps while the process lives

None of this needs a logging system to inspect while a process is running. Linux exposes it under /proc:

$ awk '{print "pid="$1, "ppid="$4, "start_ticks="$22}' /proc/2934/stat
pid=2934 ppid=1 start_ticks=98423107

The start_ticks value is only meaningful relative to the boot time, which /proc/stat does not give directly; the standard derivation combines the btime line of /proc/stat with the tick value to produce an absolute timestamp. That derivation is where evidence quality starts to degrade, because it depends on the current clock and on /proc/stat still describing the same boot.

Executable identity is a third piece. /proc/<pid>/exe is a symlink to the file that was execed, and it is tracked by inode, not by name. If the binary is replaced on disk after execution, the symlink acquires a (deleted) suffix while continuing to point at the now-unlinked inode. That detail matters: a process running from a deleted binary is a strong indication that the file was removed to obstruct inspection, and it also means any attempt to hash “the program” must read from /proc/<pid>/exe rather than from the path in the process’s command line.

The /proc view has a second temporal gap that the start-time field does not close. The triplet identifies when a process began, but the kernel keeps no record of when any still-running process last executed an execve relative to later activity: a process that called exec after starting has the start time of the original fork, because the pid and the start time belong to the task, not to the program image. A long-lived daemon that re-execs itself therefore appears older than the program it is currently running, and the only way to date the image change is an audit record for the execve itself. For the same reason, the start time in /proc/<pid>/stat is not evidence of when the currently running binary was written to disk; that question belongs to the file’s own metadata, with the caveats that filesystem timestamp manipulation documents.

Command-line arguments sit in /proc/<pid>/cmdline as a NUL-separated byte string. Because it is NUL-separated and not newline-separated, a naive text read concatenates arguments; parsing must split on \0.

The record that outlives the process

Once a process exits, the /proc entry vanishes and the (pid, ppid, start_time) triplet is gone. Nothing in the kernel retains it. The only durable record comes from an audit facility that observes execve at the syscall boundary.

On Linux, the audit subsystem can be configured with an execve rule that logs every program execution with the parent’s audit context, the full argument vector, and the executable’s inode and device. The kernel maintains a per-process audit context inherited across fork and preserved across exec, which links a chain of executions together even though pids change. That inheritance is the property that makes lineage reconstructible after the fact. Audit records are written to /var/log/audit/audit.log, which is a local file, and therefore a file that malware with sufficient privilege can truncate, and one that a container can hide entirely if the audit socket is not mounted into it.

macOS takes the opposite architectural choice. The Endpoint Security framework delivers events (ES_EVENT_TYPE_NOTIFY_EXEC among them) to subscribed system extensions, which must hold an entitlement granted by the operating system, and the monitoring process must itself be a signed system extension. The event carries the audit token of the new process, the parent’s audit token, and the resolved binary path. Apple’s OpenBSM-based audit trail (/var/audit) is the durable sink. Because the entitlement cannot be self-granted, a compromised userland process cannot subscribe to the stream and fabricate it, which raises the cost of deception relative to a log-file-only design.

Where the chain breaks

Three failure modes dominate.

The first is lossy retention. Audit logs rotate; kernel-side queues are bounded. If the audit daemon stalls, the kernel’s backlog buffer fills and events are dropped; on Linux the audit subsystem reports lost events as part of its status rather than silently discarding, but the events themselves are simply absent. Any timeline built from a log with a known gap cannot establish ordering across that gap, a problem treated directly in clock skew and event ordering.

The second is fork storms. Container runtimes and build systems create process trees thousands of nodes deep. Full execve auditing on such a host is expensive enough that operators routinely restrict rules to specific path prefixes or user ids, which is exactly the set an attacker will avoid.

The third is the missing parent. Processes reparented to init after their parent dies lose their apparent lineage: ppid becomes 1, and the original parent’s pid is not retained anywhere unless an audit record captured it. Double-forking daemons exploit this deliberately, and so does anything that wants a clean-looking parentage.

A fourth, increasingly common, is the pid namespace boundary. Inside a container, processes are numbered within the namespace, not within the host: pid 1 in the container is a host pid with an entirely different number, and /proc mounted inside the container shows the container’s numbering. A record collected inside the namespace and a record collected on the host therefore disagree about the identifier of the same process, and joining the two requires the namespace mapping, which only exists while the namespace exists. The audit subsystem logs in host pid terms, which makes host-side records the join point for any investigation that spans both views.

The defensible conclusion is that process start records are strong evidence when captured and weak evidence when reconstructed. An audit trail written at the execve boundary, inherited across fork, and shipped off-host is the only variant that answers all three questions after the process is gone. Everything else is a guess about a number that has almost certainly been recycled, and any timeline built on that guess inherits the timestamp problems of the file system underneath it.