Thread-per-Connection, Event Loops, and the Cost of Blocking

A connection spends almost all of its life waiting. Between the request bytes arriving, the database answering, and the client reading the response, the server’s involvement is a few microseconds of computation separated by milliseconds of idle time. The choice of concurrency model is a choice about what occupies a machine while that waiting happens, and every model trades one kind of cost for another.

The two poles are simple to state. A thread per connection gives each connection its own stack and lets it block in ordinary function calls; the operating system deschedules it until the descriptor is ready. An event loop gives one thread a set of descriptors and asks the kernel which are ready; the thread never blocks on any single connection, and each connection stores its progress in an explicit state machine instead of on a stack.

What a blocked read actually holds

On Linux, a thread’s stack is a mapping sized by the stack limit, commonly 8 MiB of address space, though only the pages actually touched are resident. A thread that has handled a request will have touched a few pages, so the resident cost is measured in tens of kilobytes, not megabytes, but the virtual reservation and the guard page are real and the kernel’s bookkeeping per task is not free. Thread creation and teardown cost on the order of tens of microseconds, which matters only if connections are short.

The descriptor costs more than the thread in many deployments. Each connection holds a socket with a send and a receive buffer, each sized by the kernel’s autotuning within configured bounds and often tens of kilobytes per direction. At ten thousand idle connections the buffers dominate memory unless they are explicitly reduced, and the stack reservation dominates address space.

Blocking is nevertheless an efficient use of the scheduler. A thread waiting on a socket is not runnable; it is on a wait queue and costs the scheduler nothing until data arrives. The wakeup it receives is precise, because the kernel knows exactly which task is waiting on that descriptor. There is no polling, no scan, and no shared structure to contend on.

Readiness notification and the loop

The event model replaces the wait with a query. The thread registers descriptors once and calls a function that returns the subset that is ready. The historical interfaces differ in cost: select() is limited by FD_SETSIZE, which is 1024 on typical builds, and requires rebuilding and scanning bitmaps whose cost grows with the highest descriptor number; poll() removes the descriptor limit but scans the whole array on every call. Linux’s epoll keeps the interest set in the kernel and returns only ready entries, so the per-call cost is proportional to the number of ready descriptors plus a small constant, not to the number registered.

That last property is what makes a single loop viable for tens of thousands of connections. A loop that processes readiness events and returns to the kernel is doing one syscall per batch, not one per connection, and the batch amortises the syscall entry cost across every descriptor that became ready in the same interval.

The price is paid in the structure of the code. A blocking read returns the bytes or an error. A non-blocking read returns whatever is available, possibly nothing, and the program has to remember what it was doing. A write may accept only part of a buffer, so every connection needs an output queue and an offset, and the loop must register interest in writability only while that queue is non-empty, because a socket is almost always writable and a permanent write interest produces a busy loop. That output queue is memory proportional to the number of connections with pending output, which quietly removes part of the advantage over threads when responses are large and clients are slow.

/* one iteration of a single-threaded loop; c is the connection state */
for (;;) {
    int n = epoll_wait(epfd, evs, MAXEV, -1);
    for (int i = 0; i < n; i++) {
        conn *c = evs[i].data.ptr;
        if (evs[i].events & (EPOLLERR | EPOLLHUP)) { close_conn(c); continue; }
        if (evs[i].events & EPOLLIN)  read_into(c);   /* may produce output */
        if (evs[i].events & EPOLLOUT) flush_out(c);   /* partial writes handled here */
        /* re-arm write interest only while c->out has bytes left */
        set_interest(epfd, c);
    }
}

The call that blocks anyway

An event loop’s correctness depends on every operation completing without blocking. The failures are mundane and specific: a name lookup that uses the system resolver blocks; a file read on a local filesystem blocks, because regular files are always reported ready by readiness interfaces, so the loop cannot know whether the read will return immediately; a page fault on a mapped file blocks; a lock held by another thread blocks. Any of these stalls every connection the loop owns, and the stall duration is whatever the operation took.

This is why event-driven servers push such work to a pool of threads and treat the completion as another event. It is also why the model is easiest to get right in a language with an asynchronous runtime that owns the pool.

The mirror-image failure exists in the thread-per-connection model. With no cap on concurrent connections, a burst of slow clients creates threads until the process hits its limit, at which point pthread_create fails or the accept loop cannot keep up. The remedy is a cap on in-flight connections and a queue for the excess, which is the same admission control an event loop needs; neither model removes the need to decide what happens when the arrival rate exceeds what the machine can serve.

Staged designs and intermediate forms

SEDA, described by Welsh, Culler, and Brewer in 2001, made the queue the central object: each stage of request processing is a bounded queue served by a small thread pool, and the queue depth, not the thread count, absorbs bursts. The design makes overload visible and lets each stage be tuned independently, at the cost of context switching between stages and of latency added by queuing even when the system is idle.

Most contemporary servers land somewhere between the poles. Go’s runtime runs goroutines on a small set of operating system threads and parks those blocked on the network in the runtime’s own poller, so a goroutine can be written as straight-line blocking code while the underlying thread never blocks on that socket. Java’s virtual threads, delivered as a standard feature in JDK 21, achieve the same shape: the virtual thread unmounts from its carrier when it blocks on a supported operation. Both approaches keep the programming model of threads and the scheduling economy of an event loop, and both require that a blocking operation be one the runtime knows about; a call into native code that blocks still occupies a carrier thread.

The goroutine or virtual thread stack is small to begin with, a few kilobytes that grow on demand, which is what makes hundreds of thousands of them feasible where operating system threads are not. The cost moves to the scheduler: work stealing, run queue contention, and stack growth checks add per-call overhead that an explicit state machine does not pay.

Choosing by measurement rather than by reputation

The variables that decide the question are the number of concurrently open connections, the fraction of time each spends waiting, and the size of the per-connection state the application needs. A service holding a few hundred connections, each with a database transaction open, is well served by threads; the memory cost is trivial and the code is straightforward to debug. A gateway holding a hundred thousand mostly idle connections, each forwarding a few hundred bytes occasionally, cannot afford a stack per connection and gains nothing from the simpler code.

Between those extremes the differences are small enough that operational factors dominate: which model the language’s runtime supports well, how easy it is to enforce per-connection limits, and whether the team can reason about the failure mode when one operation blocks. The remaining cost is the same in both models, and it is the one measured in the next layer up: how much work each request does while the connection is open, which streaming request bodies and header limits both assume can be bounded.