Pipe Buffers, PIPE_BUF, and the Absence of Message Boundaries

A pipe is two descriptors and a kernel buffer. Nothing in the interface records where one write ended and the next began, so a reader that asks for 4096 bytes may receive the tail of one record, all of the next, and the head of a third in a single call. Programs that treat a pipe as a sequence of messages are building that structure themselves, on top of a stream that does not preserve it, and the guarantee they need exists only in a narrow form.

The buffer, and what happens when it fills

The kernel allocates a fixed-size ring buffer when the pipe is created. On Linux the default capacity is 65,536 bytes, settable with fcntl and the F_SETPIPE_SZ command up to a system limit, and it was a single page of 4,096 bytes in earlier kernel versions. POSIX requires only that the capacity be at least PIPE_BUF, and fixes the minimum value of PIPE_BUF at 512 bytes through _POSIX_PIPE_BUF; the actual value is implementation-defined and is 4,096 on Linux. The capacity is not the same as PIPE_BUF; the confusion between them is the source of most incorrect assumptions about pipes.

A writer that fills the buffer blocks until a reader drains space. The granularity of that blocking is not specified: a write larger than the free space may block until all of it can be written, or may write part of it and return a short count, depending on the implementation and on whether the descriptor is non-blocking. POSIX permits a blocking write of more than PIPE_BUF bytes to be interleaved with writes from other writers, which is precisely the guarantee it withholds. A write of PIPE_BUF bytes or fewer on a blocking descriptor is required to be atomic: it either writes all of the bytes without interleaving, or it blocks until it can, and the bytes from different writers do not mix.

That atomicity is the only message-boundary property the interface offers, and it is a property of the write, not of the read. Two writes of PIPE_BUF bytes from different processes arrive as two contiguous regions in the buffer, in some order, and never interleaved. A reader calling read with a buffer twice that size receives both regions in one call if both are present, and the boundary between them is not marked. The reader must know the record size in advance or parse a length prefix it defined itself.

#include <stdio.h>
#include <string.h>
#include <unistd.h>

int main(void) {
    int fds[2];
    if (pipe(fds) < 0) return 1;

    const char *rec = "0123456789abcdef";   /* 16-byte record */
    for (int i = 0; i < 3; i++)
        write(fds[1], rec, 16);             /* each write <= PIPE_BUF: atomic */

    char buf[64];
    ssize_t n = read(fds[0], buf, sizeof buf);
    printf("%zd bytes in one read\n", n);   /* 48 on Linux: boundaries lost */
    return 0;
}

The read returns 48 bytes, the concatenation of three writes. Nothing distinguishes record one from record two. A reader that wants records must read exactly 16 bytes at a time, and even then it must handle the case where read returns fewer, which happens when the writer is slower than the reader.

Short reads and the loop that is always required

read on a pipe returns as soon as any data is available. It does not wait for the requested count. A read for 4096 bytes when 100 are in the buffer returns 100, and the caller must decide whether to loop or to process a partial record. This is the same contract as read on a terminal, on a socket, and on a file, and it is the reason every robust reader is a loop:

static int read_full(int fd, void *buf, size_t len) {
    size_t done = 0;
    while (done < len) {
        ssize_t n = read(fd, (char *)buf + done, len - done);
        if (n < 0) return -1;          /* caller checks errno for EINTR */
        if (n == 0) return 0;          /* EOF: all writers closed */
        done += (size_t)n;
    }
    return 1;
}

The zero return is the important case. A pipe reports end-of-file only when every descriptor referring to the write end has been closed. While any process holds the write end open, a reader that has drained the buffer blocks rather than seeing EOF, even if the process that was supposed to write has exited, because the descriptor may have been inherited by a child or duplicated into a daemon. This is the failure that pipe-based protocols run into most often: a server forks a child that inherits the write end, the parent closes its own copy, and the reader never sees EOF because the child still holds it. The fix is to close the unused end in every process that does not need it, immediately after the fork, which is exactly the discipline that the descriptor inheritance model requires and that the close-on-exec discussion formalises.

The write end and SIGPIPE

Writing to a pipe whose read end has been closed raises SIGPIPE in the writing process, and the default action for SIGPIPE is termination. The write also returns -1 with errno set to EPIPE, but the signal is delivered first, so a program that has not handled or ignored SIGPIPE dies before it can inspect the error. This is deliberate: it makes a pipeline terminate when its consumer disappears, so that a shell running yes | head does not leave yes spinning forever. head exits, the pipe’s read end closes, the next write from yes raises SIGPIPE, and yes dies.

Programs that want to handle the condition rather than die must ignore SIGPIPE (with signal(SIGPIPE, SIG_IGN) or by installing a handler) and then treat EPIPE as an ordinary error. Ignoring it changes the disposition for the process and is inherited across exec only in the sense that ignored signals stay ignored; handled signals reset to default on exec, which is one of the asymmetries in the inheritance rules.

SIGPIPE also fires on a socket whose peer has closed the reading direction, which is why network servers ignore it globally at startup. The signal is a property of writing to a broken stream, not of pipes specifically.

Duplex and the half-close problem

A pipe is unidirectional. Bidirectional communication needs two pipes, and two pipes have a failure mode that one does not: each side must close the end it is not using, and the two directions have independent lifetimes. A protocol that says “send request, read response, close” must decide whether closing the write direction signals the end of the request or the end of the conversation. With two pipes, closing one direction is a half-close: the peer sees EOF on its read side while its own write side remains open. Whether that is meaningful depends on the protocol, and the interface does not provide a way to say “I am done sending but still listening” other than by closing one of the two descriptors.

socketpair provides a bidirectional channel with a single descriptor per side and supports shutdown for half-closing, which is the cleaner primitive for request-response protocols. It also supports passing file descriptors between the two ends, which pipes cannot do; that mechanism, SCM_RIGHTS, is the way a process hands an open description to another process without going through the filesystem, and it is the reason the descriptor model’s sharing rules can be extended across process boundaries.

The popen library function, which runs a command and connects a stream to it, is a thin layer over fork, exec and a pipe, and its known limitation is that it offers only one direction at a time and that its pclose returns the child’s status through a wait that can be confused by other children. Both limitations are consequences of the underlying primitives rather than of the wrapper.

Buffer sizing as a control knob

The capacity of a pipe is a throughput/latency knob. A larger buffer lets a writer run further ahead of a reader, which raises throughput when the reader is bursty and lowers the number of context switches, at the cost of memory and of the delay before a reader sees data that a writer has produced. The default of 64 KiB on Linux is large enough that a single write of a modest record does not block, and small enough that a pipeline of many stages does not consume large amounts of memory.

The interaction with PIPE_BUF is what matters for correctness. A program that writes records no larger than PIPE_BUF and relies on atomicity is correct at any capacity, because the guarantee is about the write, not the buffer. A program that writes records larger than PIPE_BUF and assumes they arrive intact is incorrect regardless of capacity, and enlarging the buffer makes the bug less likely to appear rather than fixing it. That is the worst kind of failure: one that is hidden by the current default and appears when the default changes, when the load increases, or when the program moves to a system with a different PIPE_BUF.

O_NONBLOCK changes the failure from blocking to a short write or EAGAIN, and a non-blocking write of more than PIPE_BUF bytes may write a partial record, which reintroduces interleaving. The combination of non-blocking descriptors and records larger than PIPE_BUF has no correct implementation that does not add its own framing.