Thread Stacks, Guard Pages, and Thread-Local Storage

The main thread’s stack is created by the kernel as part of the initial address space and grows on demand: touching an address below the current stack pointer faults, the kernel notices the address is within the stack’s allowed range, and it maps another page. A thread created later gets none of that. Its stack must be allocated before the thread starts, and once it exists the kernel will not extend it, because the kernel has no way to distinguish a legitimate growth from a wild pointer.

Reserving a Stack Without Committing It

The library that creates a thread asks the kernel for a mapping of the requested size, then adjusts it. The mapping is established as inaccessible or uncommitted, and a portion at the top is made readable and writable for the thread to use. The untouched remainder costs address space and page-table entries but no physical memory, which is why a default stack size of several megabytes does not translate into several megabytes of resident memory.

The size is a compromise with no universally correct value. Too small and a deep recursion or a large local array overflows. Too large and a program that creates many threads exhausts the address space or the commit limit. The C library exposes the attribute so the application can set it, and the practical approach is to measure the deepest legitimate call chain and add a margin rather than to accept the default.

The stack must also be aligned. Most architectures require the stack pointer to be aligned to a boundary at every call, and some require more for vector state. The library rounds the base and the size to the required boundary, and a program that supplies its own stack buffer must do the same or risk faults on the first vector instruction that spills to the stack.

A subtle constraint is that the stack must be placed where the thread-local storage block can be addressed from it. On several architectures the thread pointer register points into the thread’s control block, and the offset from that register to a thread-local variable is a link-time constant. If the block is placed at the top of the stack mapping, the two are adjacent and the addressing works with a single base register. This is why the layout is not arbitrary: the ABI fixes where the block sits relative to the stack.

The Guard Page

At the end of the stack that grows downward, the library leaves one page unmapped. This is the guard page. A stack that overflows past the last usable page reaches the guard, and the access faults. Without the guard, the overflow would silently write into whatever mapping happened to be adjacent, corrupting another thread’s stack or the heap with no diagnostic.

The fault is reported as an invalid access at an address that is close to the stack pointer but below the mapped region. The handler that reports it can recognize the pattern and produce a message naming a stack overflow, which is far more useful than the eventual corruption that an unguarded overflow produces.

The guard is one page, which is a deliberate choice about the trade between detection and waste. A page is four kilobytes on common architectures, small enough that the waste per thread is negligible, large enough that a modest overflow lands in it. A large local array that jumps over the guard in a single access will not be caught, and neither will an overflow that occurs through a pointer rather than through the stack pointer. The guard detects the common case, not every case.

Some systems add a second guard region for a different purpose: protection against a stack that grows upward past its top, or against a mapping placed immediately below. The direction of growth is fixed by the ABI, so the guard goes on the side the stack grows toward.

The alternative design, used by some runtimes, is to allocate a stack large enough that overflow is considered impossible and to omit the guard. This trades a bounded waste of address space for the loss of a diagnostic, and it works only when the maximum depth is statically known, which is not true for general-purpose code.

Addressing Per-Thread Data Without a Lookup

Thread-local storage is the mechanism that gives each thread its own instance of a variable declared with thread-local duration. The naive implementation would be a map from thread identifier to a block of values, consulted on every access, which would make a thread-local read as expensive as a hash lookup. The actual implementation is an offset from a register.

The compiler knows, at link time, the offset of each thread-local variable within the per-thread block. An access becomes a load from the thread pointer register plus a constant. The register is set when the thread starts and is not changed during the thread’s life, so the access is a single instruction with no lookup and no lock.

The details differ by architecture and by whether the variable is defined in the executable or in a shared library. A variable in the main executable can use a fixed offset known at link time, the simplest form. A variable in a shared library cannot have a fixed offset, because the library may be loaded after other libraries and the layout of the block depends on load order; the access goes through a per-module offset that the dynamic linker fills in at load time. The cost of that indirection is one extra load, and it is paid only for variables in shared objects.

#include <pthread.h>
#include <stdio.h>

static _Thread_local int counter;   /* one instance per thread */

static void *worker(void *arg) {
    counter = (int)(long)arg;
    counter += 1;
    printf("thread sees %d\n", counter);
    return NULL;
}

int main(void) {
    pthread_t a, b;
    pthread_create(&a, NULL, worker, (void *)10L);
    pthread_create(&b, NULL, worker, (void *)20L);
    pthread_join(a, NULL);
    pthread_join(b, NULL);
    return 0;   /* each thread printed its own value plus one */
}

The initialization rules are the part that surprises people. A thread-local variable with a constant initializer is initialized for each thread when that thread starts, and the initialization is performed by the runtime rather than by the kernel, which means a thread created by a mechanism that bypasses the runtime may see an uninitialized value. A thread-local variable with a dynamic initializer requires a constructor call per thread, and the runtime arranges for it to run before the thread’s start function.

Destructors exist for the reverse direction. A thread-local object with a destructor is destroyed when the thread exits, and the runtime runs the destructors in a loop that allows a destructor to create a new thread-local object, up to a fixed number of iterations. The limit exists because a destructor that recreates its own object would otherwise loop forever.

The Cost of a Thread’s Existence

A thread’s address-space footprint is the stack mapping plus the thread control block plus the thread-local storage block. The first dominates: a default stack is measured in megabytes of address space even when only a few pages are touched. The others are small, on the order of a few hundred bytes to a few kilobytes.

The kernel-side footprint is a task descriptor and a kernel stack. The kernel stack is a fixed allocation, sized to the architecture’s maximum kernel call depth, typically on the order of several pages. It is not growable and not swappable in the ordinary sense, so a large number of threads consumes wired or unreclaimable kernel memory in proportion to the count. It is also the memory that holds a blocked thread’s saved registers, which is why a thread’s stack and its saved context are the same object, as described in The Anatomy of a Context Switch.

Creation cost is dominated by the kernel work: allocating the descriptor and the kernel stack, duplicating or sharing the structures named by the flags, and placing the new task on a run queue. The user-side work is the stack mapping and the runtime’s initialization. Destruction cost includes the reverse, plus the cost of joining, which requires the exiting thread to remain in a state where its result can be collected.

The trade against processes is stark at the extremes. A process has an address space of its own, which costs a page-table tree and a translation-cache flush on every switch, but gives full isolation. A thread has a stack and a control block, switches without flushing translations, and gives no isolation at all. A program that needs many independent units of work with small stacks is choosing between the per-thread cost of a kernel stack and the per-process cost of an address space, and the arithmetic usually favors threads until the count reaches the tens of thousands, at which point the kernel stack allocation becomes the binding constraint and a user-level scheduling design becomes attractive.

That design, in which many logical threads multiplex over few kernel threads, is the subject of User-Level Threads and the M:N Scheduling Problem. Its stack strategy is the interesting part: a user-level thread can start with a small stack and grow it by copying, which is impossible for a kernel thread whose stack base is fixed at creation.