Buffering Modes and Why Pipeline Output Appears Out of Order

A program that prints a diagnostic and then exits can produce output in the wrong order without containing a single logic error. The lines were written in the intended sequence; they reached the descriptor in a different one. The cause is that two separate queues sit between a printf call and the bytes on a pipe, and only one of them preserves order across both streams.

Three policies, chosen at open time

The C library gives each stream a buffering mode. Unbuffered mode sends every write straight through. Line buffered mode accumulates until a newline is written, then flushes. Fully buffered mode accumulates until the buffer fills or the program flushes explicitly. setvbuf selects the mode and, optionally, the buffer, and it must be called before any I/O on the stream.

The default is chosen when the stream is first used, and the choice depends on what the descriptor refers to. A stream connected to a terminal is line buffered, because interactive output must appear as it is produced. A stream connected to anything else, such as a pipe, a regular file, or a socket, is fully buffered, because the cost of a system call dominates and batching is nearly free. Standard error is the exception: it is unbuffered or line buffered by default depending on the implementation, and the C standard only requires that it not be fully buffered.

What the descriptor refers to is not quite the same as what the terminal driver does with the bytes. The library decides on the basis of isatty, while the driver independently decides whether to edit the stream, echo it, and translate its line endings. Both layers inspect the same descriptor and both apply their own policy, and a program that adapts to only one of them will behave inconsistently.

This is why the same program behaves differently under prog and prog | cat. The descriptor changes from a terminal to a pipe, the mode changes from line to full, and output that appeared promptly now arrives in blocks. The program has not changed; its environment has.

The buffer size is a block, not a line

A fully buffered stream’s capacity defaults to a value related to the file’s block size, or to BUFSIZ when no better information is available. The C standard requires BUFSIZ to be at least 256; glibc defines it as 8192. A stream writing short lines therefore holds thousands of bytes before emitting anything.

#include <stdio.h>

int main(void) {
    fprintf(stdout, "out 1\n");
    fprintf(stderr, "err 1\n");
    fprintf(stdout, "out 2\n");
    fprintf(stderr, "err 2\n");
    return 0;   /* exit flushes stdout here */
}

Run at a terminal, the four lines appear in written order, because stdout is line buffered and flushes at each newline. Run with stdout redirected to a file and stderr left on the terminal, the two error lines appear first and both output lines appear afterwards, when the buffer is flushed at exit. The interleaving changed because one stream was queued and the other was not.

Descriptor order is preserved; stream order is not

The kernel does not reorder anything. Writes to a file descriptor are applied in the order the calls are made, and a pipe delivers bytes in the order written. If a program called write(1, ...) and then write(2, ...), and both descriptors referred to the same pipe, the bytes would arrive in that order. The reordering happens entirely in user space, before either call is made.

That distinction explains the failure of the common attempt to fix ordering with 2>&1. The redirection changes which file description descriptor 2 refers to; it does not change the buffering of the stream attached to descriptor 1. After prog > log 2>&1, both streams write to the same file, and stdout is still fully buffered, so its lines still land after the error lines that were produced earlier. The redirection made the output contiguous; it did not make it ordered.

Ordering can only be imposed by flushing. Calling fflush(stdout) before writing to stderr forces the queued bytes out, and the sequence then matches the program’s intent. A program that writes diagnostics to stderr and progress to stdout has to flush at every transition, which is why many tools flush unconditionally before emitting a warning.

What flush and exit actually guarantee

fflush writes the stream’s pending bytes with write and returns once the kernel has accepted them. It does not guarantee that the data has reached durable storage, which requires fsync on the descriptor. For a pipe, acceptance by the kernel means the bytes are in the pipe buffer and will be delivered unless the reader closes, as described in pipe buffering.

exit flushes all open streams and runs functions registered with atexit, in reverse order of registration. _exit and _Exit do neither. A program that terminates through _exit after a signal handler, or through abort, loses whatever was queued. That is the mechanism behind truncated output from a program that crashed: the last block of its standard output never left the process.

Buffered data also interacts with process creation. fork copies the buffer into the child, so a parent with queued output and a child that calls exit produce the queued bytes twice. The standard remedy is to flush before forking, and the symptom is duplicated output in a program that uses both fork and printf.

Input is buffered too, and that loses bytes

The same mechanism runs in the other direction and causes a sharper defect. When a program calls fgets or scanf, the C library does not read one line from the descriptor; it reads a block and returns the first line from its buffer, leaving the remainder queued inside the stream. Those queued bytes are invisible to any subsequent read on the same descriptor.

A program that mixes stream input with direct descriptor input therefore drops data. The classic shape is a prompt read with scanf followed by a read from descriptor 0: the read returns nothing because the bytes it was meant to see are sitting in the stream’s buffer, and the program waits for input that has already arrived. The reverse order, read first and then scanf, is safe only because the descriptor has nothing buffered yet.

This is also why an interactive shell’s read builtin reads one byte at a time rather than a block. A shell must not consume input that belongs to the command it is about to run, and the only way to guarantee that with a shared terminal descriptor is to read exactly as much as the current line requires. The cost is a system call per byte, which is acceptable for interactive input and unacceptable for bulk processing. It is the same latency-versus-throughput trade that governs output, applied in the opposite direction.

Changing another program’s buffering

stdbuf alters the buffering of a program that has already been compiled, by loading a library that intercepts the C library’s stream initialisation and applies the requested mode. It has no effect on statically linked programs, and the dynamic loader ignores the mechanism for setuid binaries, so the setting is silently dropped in both cases. It also cannot help a program that manages its own buffers or writes with write directly.

The portable alternative is to design the program to flush at the points where ordering matters, or to make its output line oriented and accept the cost. A filter that must interleave with a chatty log has no other reliable option, because the decision about buffering belongs to the process doing the writing and cannot be imposed from outside in general.

The cost the batching buys

Each write is a system call: a transition into the kernel, a copy of the bytes into kernel space, and a return. A memcpy within the process is orders of magnitude cheaper. For a program emitting many short lines, fully buffered output can reduce the number of calls by a factor equal to the number of lines that fit in the buffer, which for 8 KiB and 40-byte lines is roughly two hundred.

The same batching is a liability for a long-running process whose output is consumed interactively. A progress indicator written to a pipe will not appear until 8 KiB have accumulated, which may be never. The two requirements, throughput for bulk output and latency for interactive output, pull in opposite directions, and the resolution is to choose the mode per stream: bulk data fully buffered, status line unbuffered or explicitly flushed.

There is a third cost that is easy to miss. Buffering delays the discovery of write errors. A program writing to a closed pipe learns about EPIPE only when the buffer flushes, which may be at exit, after it has done work whose output can never be delivered. Unbuffered output moves that discovery to the point of the write, which is why programs that must react to a closed consumer, and to the signal semantics that go with it, avoid deep buffering on the streams they care about.