What a Pipe Holds: Kernel Buffering, Atomic Writes, and SIGPIPE

The vertical bar is the most consequential character in shell syntax, and the object it creates is smaller than most users assume. A pipe is not a stream with unlimited capacity, not a file, and not a socket. It is a fixed-size buffer inside the kernel with two file descriptors attached, and every behaviour that pipelines exhibit follows from that buffer’s size and from what happens when it fills or when one end disappears.

Two descriptors, one direction, no seeking

pipe returns two descriptors in a single array: index 0 is the read end, index 1 is the write end. Data written at index 1 is readable at index 0 in the order written. Nothing else is available. There is no way to read the same byte twice, no way to seek backwards, and no way to write at the read end. The shell’s | performs this call, then forks, then arranges the descriptors with dup2 so that the left command’s standard output and the right command’s standard input are the two ends, then closes the originals.

Two consequences follow immediately. First, a pipeline is strictly unidirectional; anything the right-hand command wants to say back to the left has to go through a different channel, which is why xargs-style tools and co-processes exist. Second, the descriptors are ordinary file descriptors, so they inherit across exec like any other. That inheritance is how the stages of a pipeline receive their ends, and it is also how a pipe stays open long after the process that was supposed to write to it has exited, as described under descriptor inheritance.

The buffer is bounded, and the bound is the point

On Linux a pipe holds 65,536 bytes by default, which is sixteen pages on the common 4 KiB page size. The capacity can be queried and changed with fcntl using F_GETPIPE_SZ and F_SETPIPE_SZ, subject to a system-wide maximum; other kernels choose their own value, and POSIX specifies only that the buffer exists and that PIPE_BUF is at least 512 bytes.

The bounded capacity is not a limitation to be worked around. It is the flow-control mechanism. When the buffer is full, a write blocks until a reader drains space. When the buffer is empty, a read blocks until a writer adds data. The producer therefore cannot run arbitrarily far ahead of the consumer, and the amount of data resident in the pipeline at any moment is bounded by the number of stages times the buffer size. That is backpressure implemented in the kernel, with no protocol and no cooperation from either program, and it is why a slow consumer naturally throttles a fast producer instead of exhausting memory.

The trade is direct. A larger buffer absorbs bursts and reduces the number of times the producer blocks, at the cost of more memory per pipe and more data in flight, which means a consumer that dies has more work already committed that will never be read. A smaller buffer keeps latency low and memory small, at the cost of more context switches between producer and consumer.

Atomicity and the interleaving of writers

POSIX guarantees that a write of PIPE_BUF bytes or fewer is atomic: it either completes in full or does not happen, and no other writer’s bytes appear inside it. On Linux PIPE_BUF is 4096. A write larger than that may be split, and bytes from another writer may land between the fragments.

This is the whole reason log lines survive when several processes append to the same pipe. A line under the atomic limit arrives intact; a line over it can be cut in half by a concurrent writer. The guarantee applies per write call, not per logical message, so a program that emits a header with one write and a body with another has no atomicity across the pair, regardless of total size.

The same guarantee explains why a program that writes through the C library’s buffered output behaves differently from one that writes with write directly. printf may accumulate several logical lines and emit them in a single large write, or emit them one at a time depending on buffering mode. When the total exceeds PIPE_BUF, the atomicity that protected individual lines no longer applies, which is one of the failure modes discussed in buffering modes.

When the reader leaves: EPIPE and SIGPIPE

Closing the read end while a writer still holds the write end produces two effects from a single write. The write fails with EPIPE, and the kernel also delivers SIGPIPE to the writing process. The default action for SIGPIPE is termination, so in the common case the writer dies before it can examine the errno value. A shell reports such a death as 141, which is 128 + 13.

This design is deliberate and it predates the convention of checking every write. It lets the left side of producer | head -n 5 terminate promptly once head has its five lines and exits, without the producer needing any code to detect that its output is no longer wanted. The producer writes, the kernel raises the signal, and the process ends. A program that installs a handler or ignores the signal instead receives EPIPE and must handle it, which is the right choice for a program that needs to clean up before exiting but the wrong choice for a filter that should simply stop.

The inheritance rule makes this subtle. An ignored disposition survives exec, so a parent that ignores SIGPIPE hands that ignore to every child it spawns. Those children then see EPIPE instead of dying, and a filter that treats write errors as fatal will print a diagnostic and exit non-zero where the shell expected a silent signal death. The status a pipeline reports changes accordingly.

EOF requires every write end to close

A reader sees end-of-file only when all descriptors referring to the write end have been closed. There is no message that means “no more data”; the absence of writers is the message. This produces the most common pipeline hang: a background process inherits the write end, outlives the command that was supposed to close it, and the reader waits forever for an EOF that will not arrive until the background process exits.

The shell constructs pipelines so that each stage closes the ends it does not need, which is why an ordinary a | b terminates correctly. A program that creates a pipe and forgets to close the unused end in one of its children reintroduces the hang. O_CLOEXEC on every descriptor not meant to survive is the systematic defence, since it makes the leak impossible rather than merely unlikely.

Deadlock needs a cycle, not a big buffer

A single pipe cannot deadlock: one direction, one producer, one consumer, and the kernel arbitrates. Deadlock appears when two pipes are used in opposite directions and both processes decide to write before reading. Each fills its own pipe, each blocks waiting for the other to drain, and neither can proceed. The buffer size only determines how much data must be written before the deadlock becomes visible; it never prevents it.

The same shape appears with a single pipe when the consumer must read everything before producing anything. A stage that sorts its input cannot emit its first line until it has seen the last one, so the pipeline behaves as a barrier rather than a stream. That property, and the strategies for keeping such stages from stalling a long-running pipeline, belongs to backpressure in long pipelines.

What the interface does not carry

A pipe carries bytes, not records. There are no message boundaries, no priorities, and no way to ask how much data is waiting without reading it. A reader that issues a large read gets whatever is available, which may be half a line, several lines, or a single byte.

char buf[4096];
size_t bytes = 0, lines = 0;
ssize_t n;
while ((n = read(fd, buf, sizeof buf)) > 0) {
    /* n may be smaller than sizeof buf even when more data is coming */
    for (ssize_t i = 0; i < n; i++)
        if (buf[i] == '\n') lines++;
    bytes += (size_t)n;
}
/* n == 0 is end of file; n < 0 is an error, with EINTR possible */
printf("%zu bytes, %zu lines\n", bytes, lines);

The loop counts lines without assuming anything about how the producer wrote them, which is the only safe assumption.

Programs that treat each read as a complete unit produce intermittent, input-dependent bugs that are hard to reproduce because they depend on the producer’s write sizes and on scheduling.

The interface is also anonymous. There is no name to look up, no permission check beyond the descriptor, and no way to reconnect after the descriptors are closed. That is why named pipes exist as a separate object, and why any protocol that needs to survive the death of one endpoint has to be built on something other than |.