Readiness Notification: What select, poll, and epoll Actually Report

A server that handles many connections without a thread per connection needs to know, before it calls read, whether the call would block. The kernel offers several ways to ask that question, and all of them answer it with the same weak guarantee: a descriptor reported ready means an operation of the requested kind will not block at the moment of the report. It does not mean the operation will succeed, and it does not mean the data will still be there when the operation runs. Everything else about the interfaces — the data structures they use, the cost per call, the number of descriptors they can carry — is a consequence of how each one organises the search.

The question the interfaces share

Readiness is a property of an open file description, evaluated against a mask of conditions. For a socket those conditions are readable, writable, and an error or hangup pending. For a pipe, readable means data or end-of-file. For a terminal, readable means a line is available. The kernel keeps this state in the object itself; the interfaces differ only in how they locate the objects whose state has changed.

The guarantee has a sharp edge. Between the moment epoll_wait returns a descriptor as readable and the moment the program calls read, another thread may have consumed the data, or the peer may have reset the connection. The subsequent read returns EAGAIN or ECONNRESET. A readiness loop is therefore always built on non-blocking descriptors, and always treats EAGAIN on a descriptor reported ready as a normal event rather than an error.

There is also a class of object for which readiness is meaningless. A regular file on disk is always ready: read on it never blocks on network or pipe state, only on the page cache and the device. Registering one with an edge-triggered interface produces a descriptor that is reported once and then never again, because its state never transitions. On Linux, epoll_ctl rejects some file types outright with EPERM.

The bit vector and its two costs

select takes three sets of descriptors — read, write, exceptional — plus a timeout. Each set is a fixed-size bit vector indexed by descriptor number, fd_set, and FD_SETSIZE is the number of bits. In glibc it is 1024. The number is a compile-time constant baked into the structure layout, so a descriptor numbered above 1023 cannot be placed in a set at all, regardless of how many descriptors the process actually holds.

Two costs follow. First, the kernel cannot know which bits are set without scanning, so each call walks all FD_SETSIZE bits of each set, giving a cost proportional to the highest descriptor number rather than to the number of descriptors of interest. Second, the caller rebuilds the sets on every call, because select overwrites them in place with the ready subset. A loop that waits on fifty descriptors pays for copying and clearing 1024 bits each iteration even when nothing happens.

select also mutates its timeout argument on Linux, replacing it with the time remaining. Code that passes the same struct timeval to successive calls therefore waits for a decreasing interval and eventually spins. POSIX leaves the state unspecified, so the portable discipline is to reinitialise the timeout before each call.

poll removes the constant but keeps the scan

poll replaces the bit vector with an array of struct pollfd, each holding a descriptor, a requested event mask, and a returned event mask. There is no fixed upper bound on descriptor numbers, and the caller reads results out of revents without losing the request in events, so the array does not have to be rebuilt between calls. A descriptor with a negative value in the fd field is skipped, which gives a portable way to ignore an entry without compacting the array.

#include <poll.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
    struct pollfd fds[2];
    fds[0].fd = STDIN_FILENO;  fds[0].events = POLLIN;
    fds[1].fd = -1;            fds[1].events = POLLIN;  /* ignored */

    int n = poll(fds, 2, 1000);   /* 1 second, in milliseconds */
    if (n < 0) { perror("poll"); return 1; }
    if (n == 0) { puts("timeout"); return 0; }

    /* revents carries POLLIN, POLLHUP, POLLERR, POLLNVAL */
    if (fds[0].revents & POLLIN) puts("stdin readable");
    if (fds[0].revents & POLLHUP) puts("hangup");
    return 0;
}

The cost is now proportional to the number of entries in the array, and the kernel must copy the array in and out on every call and scan it to find which entries to test. That is linear in the number of descriptors watched, per call. For a hundred connections and frequent events it is fine. For ten thousand mostly idle connections it means ten thousand entries copied and examined each time one connection has data. The array also lives in user memory, so the kernel has no persistent record of what the process cares about.

The persistent interest list

epoll splits registration from waiting. epoll_create1 returns a descriptor referring to a kernel object holding two structures: an interest list, and a ready list. epoll_ctl adds, modifies, or removes an entry in the interest list, and the kernel attaches a callback to each registered file description. When an event occurs on that object, the callback places the entry on the ready list. epoll_wait then returns whatever is on the ready list, up to a requested maximum.

#include <errno.h>
#include <sys/epoll.h>

void serve(int listen_fd) {
    int ep = epoll_create1(EPOLL_CLOEXEC);

    struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
    epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);

    struct epoll_event ready[64];
    for (;;) {
        int n = epoll_wait(ep, ready, 64, -1);
        if (n < 0) { if (errno == EINTR) continue; return; }
        for (int i = 0; i < n; i++)
            handle(ready[i].data.fd, ready[i].events);
    }
}

The cost per wait is now proportional to the number of ready descriptors, not to the number registered. Registration is a separate, occasional call. The interest list persists across waits, so there is nothing to copy in, and the ready list is drained rather than scanned.

The data field is the load-bearing part of the design. It is a 64-bit value the kernel stores untouched and returns with the event, so a program can put an index into its own connection array there instead of a descriptor number. That avoids a lookup from descriptor to connection state on every event, and it survives the descriptor being closed and its number reused.

Level and edge triggering

The default is level-triggered: if a descriptor still has unread data when epoll_wait is called again, it is reported again. That is the same semantics as poll, and it tolerates a program that reads only part of what is available.

With EPOLLET the report happens only on a transition — data arriving on an empty queue, or space appearing in a full send buffer. The program must then drain the descriptor completely on each report, looping on read until EAGAIN. Missing that loop does not produce an error; it produces a connection that stops being serviced while the event loop continues to run. Edge triggering reduces the number of wakeups when a descriptor is frequently ready, at the cost of requiring exact drain logic.

EPOLLONESHOT disables the registration after one report, so the entry must be re-armed with EPOLL_CTL_MOD. It exists for the case where several threads share one epoll descriptor: without it, two threads can be woken for the same socket and race for the same data.

Timeouts, units, and the atomic-wait variants

The three interfaces accept a timeout, and the units differ in a way that catches every program that moves between them: select and pselect use a struct timeval with seconds and microseconds, poll and ppoll use milliseconds as a plain integer, and epoll_wait uses milliseconds with resolution bounded by the kernel’s timer granularity. A value of -1 waits indefinitely and 0 polls once and returns immediately, in both poll and epoll_wait; select expresses the same two cases as a null pointer and a zeroed structure.

A return of zero does not always mean the full timeout elapsed. A signal delivered to the waiting thread interrupts the call, and the kernel returns either EINTR or a zero count depending on the signal’s disposition and on the specific interface. SA_RESTART does not rescue poll, select, or epoll_wait, so a loop that treats a zero return as “nothing happened, wait again” must also treat EINTR as a reason to go around rather than to fail.

pselect and ppoll exist for the gap between checking a flag and blocking. Both take a signal mask that the kernel applies atomically with the wait, so a signal arriving in that gap is delivered rather than lost. A program that waits on a descriptor and on a flag set by a handler cannot get this property by blocking signals around a plain call, because the unblock and the wait are two separate operations. The same problem appears in sigsuspend, and it is treated with signal delivery in signal handling and async-signal safety.

Lifetime and inheritance

An epoll registration is attached to the open file description, not to the descriptor number. Closing the registered descriptor removes the entry automatically, which is convenient, but the rule interacts with duplication: if the descriptor was duplicated and one copy closed, the registration survives because the description is still open. Only when the last descriptor referring to that description closes does the entry disappear. The same description-level identity is what makes descriptor inheritance across fork behave as it does; see fork, exec, and descriptor inheritance.

The epoll object itself is a file description and can be duplicated across fork. Two processes then share one interest list and one ready list, and both may call epoll_wait on it. Whether that is useful depends on the workload: it distributes events but makes per-connection state sharing mandatory. The common alternative is one epoll object per worker, with the listening socket registered in each.

The interfaces described here report readiness for descriptors. When the event of interest is not a descriptor at all — a signal, a timer, a completion from a thread — the same loop needs a way to turn it into one, which is where signalfd and its equivalents fit. The completion-based alternative, in which the kernel is given the operation rather than asked about readiness, is covered in io_uring submission and completion rings.