When a Blocking Call Enters an Asynchronous Runtime

An asynchronous runtime’s scheduling guarantee is conditional: a task that suspends yields its carrier thread, and the carrier runs another task. The condition is that the suspension goes through the runtime. A task that calls a blocking function without the runtime’s knowledge does not suspend; it occupies the carrier thread inside the kernel, and every task that carrier would have run waits.

On a runtime with one carrier per core, one such call removes one core’s worth of capacity. With enough of them, the runtime has no runnable carriers and stops, even though every task in the system is merely waiting. This is the failure that makes blocking calls a correctness problem for an async runtime rather than a performance note, and every runtime that supports arbitrary user code has to decide how to handle it.

The three places a blocking call can come from

The first is the program’s own code: a call to a sleep, a file read, a name lookup, or a lock that may be held. A runtime can address this by providing non-blocking replacements and documenting that they must be used. That works as long as the program’s authors control all the code.

The second is a library the program calls, written before the runtime existed or written for a different concurrency model. A database driver, an image decoder, a compression routine that reads from a file descriptor, or a cryptographic library that calls getrandom. The program cannot rewrite these, and the runtime cannot see inside them.

The third is the language’s own runtime: garbage collection stop-the-world pauses, class loading, JIT compilation, and lazy initialisation that touches the filesystem. These are the runtime’s own blocking, and they are usually the largest single source of unexpected latency in a managed language, because they happen at unpredictable points and are not attributable to any task.

The three sources need different treatments, and confusing them produces designs that solve the wrong problem.

Detection is a function of who owns the syscall

If the runtime intercepts every call, detection is trivial: the interception point knows whether the call may block and can route it. Go does this because the compiler emits the calls, so a file read goes through runtime code that knows how to release the logical processor. A runtime that only provides a scheduler around ordinary functions has no such point.

Where interception is impossible, detection can still be approximated by watching the carrier thread. A runtime that controls its own threads can arm a timer before dispatching a task and check after the task returns: if the task took far longer than its budget without yielding, the carrier was probably blocked. This is a diagnosis, not a fix, because by the time the timer fires the carrier is already stuck, and it produces false positives for tasks that compute for a long time without suspending, which is the cooperative scheduling problem rather than the blocking problem.

Some runtimes expose a way to mark a call as blocking, so that the scheduler moves the task to a pool before making it. This is a manual declaration, and its accuracy depends on the programmer knowing which calls block. A call that is documented as non-blocking but performs a lazy disk read on first use will slip through, which is the case for many seemingly safe library functions.

What a blocking pool buys and what it costs

The standard remedy is a pool of threads that are allowed to block. A task that must make a blocking call is handed to the pool; the pool thread performs the call and wakes the task when it returns. The scheduler’s carriers are never blocked, and the pool’s size bounds the concurrency of blocking operations.

The cost has several parts. Each pool thread is a kernel thread with a stack, and on Linux the default 8 MiB stack means a pool of thirty-two threads reserves a quarter of a gigabyte of address space; the pages are touched only as each stack grows, so the resident cost is a fraction of that, but the reservation is what makes a very large pool expensive. The handoff costs two wakeups and a queue insertion. The task’s data must be accessible to the pool thread, which for a stackless runtime means the frame must not move or must be pinned while the pool thread holds a pointer into it. And the pool’s size is a policy decision with no safe default: too small and blocking operations queue behind each other, too large and the runtime creates hundreds of threads that the kernel must schedule against each other.

import asyncio

async def fetch(path):
    loop = asyncio.get_running_loop()
    # Runs in the default executor thread; the loop stays free.
    data = await loop.run_in_executor(None, blocking_read, path)
    return parse(data)

The pattern is the same in every runtime: a call that names a function, a pool, and a completion. The runtime cannot check that blocking_read really blocks, and it cannot check that parse does not; the boundary is a claim by the programmer.

There is a second-order cost that is easy to miss. A pool thread that blocks holds a lock the program may need. If the blocking call is flock or a database transaction, the pool thread’s existence changes the program’s lock ordering, and the deadlock that results involves a thread the program never created. Runtimes that expose the pool therefore also expose its size, so that the program can bound the concurrency of operations that contend.

File descriptors and the readiness assumption

The readiness model treats sockets and files alike in the API and very differently in the kernel. A socket becomes readable when data arrives, and epoll reports it. A regular file is always reported as readable, because the page cache can satisfy a read without blocking in the common case; a read that misses the cache blocks inside the kernel until the page arrives, and no readiness notification can warn about it.

This is why file I/O in a readiness-based runtime is usually routed to a blocking pool regardless of the descriptor. The alternative is a completion interface, where a file read is an operation the kernel performs and reports, which is the reason readiness and completion models matter to runtime design and not only to kernel programming. A runtime that uses completion for files and readiness for sockets has two paths, and the choice is made per descriptor type rather than per call.

The same asymmetry applies to the name resolver. getaddrinfo blocks, and the C library’s resolver may perform several round trips; a runtime that calls it directly on a carrier will stall for the duration of a DNS query, which is measured in milliseconds on a warm cache and seconds when the resolver retries. The remedies are a pool thread, a resolver library that exposes an asynchronous interface, or a runtime-provided cache. All three exist in shipping runtimes, and none of them makes getaddrinfo itself safe to call.

The carrier-count question

A runtime with a pool of size P and carriers of size N can have N + P threads runnable. If P is unbounded, the runtime’s concurrency is bounded only by the operating system, and the scheduler’s careful accounting of carriers becomes meaningless. If P is zero, blocking calls are forbidden and the runtime cannot run ordinary code.

The usual compromise is a small default pool with the ability to grow, plus a separate mechanism for operations that must not be delayed. The growth policy is where the runtime’s assumptions show: a pool that grows on queue depth responds to load, while a pool that grows on starvation responds to latency. A pool that grows without bound eventually hits the process’s thread limit, and the failure mode is a program that stops creating threads and deadlocks instead of reporting an error.

A distinct approach is to let the carrier block but hand its scheduling state to a replacement thread. Go’s runtime does this: the logical processor is detached from the blocked thread and attached to a new one, so the number of runnable carriers stays constant. The blocked thread remains parked in the kernel until its call returns, at which point the runtime either resumes it or parks the task. The cost is one thread per concurrently blocked call, which is the same resource as a pool thread, but the accounting is automatic rather than a policy the programmer must set.

Where the boundary belongs

The honest conclusion is that a runtime cannot make arbitrary blocking code safe, and pretending otherwise produces the worst outcome: a program that works under light load and stalls under heavy load, with no indication of which call caused it. The runtimes that handle this well do three things. They provide asynchronous versions of the operations that matter. They route everything else to a pool with a size the program can see and configure. And they instrument the carriers so that a task which exceeds its budget without yielding is reported, with enough information to identify the call.

The third is the one most often omitted and the most valuable in practice. A runtime that can say which task blocked, on which carrier, and for how long turns an intermittent stall into a specific fixable line of code. That instrumentation is also the input to the fairness machinery described in fairness and tail latency in task schedulers, because a blocked carrier is indistinguishable from a carrier running a long task until the runtime records the difference.

None of this changes the underlying constraint. A blocking call is a decision to occupy a thread, and a runtime’s thread budget is the resource it schedules. The only way to make blocking free is to not have threads, which is what a single-threaded event loop does, and it pays for that with the same limitation in a sharper form: the loop in readiness notification and the single-threaded event loop has exactly one thread to lose.