Thread-Local Storage Access Models

A thread-local variable is a variable with one instance per thread. The compiler must turn a reference to it into an address, and the address differs per thread, so the reference cannot be a constant. The four access models that ELF defines are four different answers to “where is this thread’s copy”, arranged by how much the compiler and linker can know ahead of time.

The cheapest model resolves to a constant offset from a register that the ABI dedicates to the thread pointer. The most expensive performs a function call into the runtime on every access. The models exist because a module loaded with dlopen at run time cannot know its own TLS offset when it is compiled, while the main executable can know it exactly. Choosing the model is choosing which of those situations the code must support.

The thread pointer and the two block layouts

Every thread has a thread control block and, adjacent to it, the static TLS area: the concatenation of the TLS segments of all modules loaded at startup. The ABI designates a register as the thread pointer, and each module’s TLS variables sit at fixed offsets from it once the module’s position in the static area is assigned.

The sign of those offsets is architecture-dependent and worth knowing before reading generated code. On x86-64 the thread pointer points into the control block, and the static TLS blocks lie at negative offsets from it, so a local-exec access is a load from %fs: with a negative displacement. On several RISC ABIs the thread pointer points at the start of the TLS area and offsets are non-negative. The psABI for each architecture fixes the layout, and the compiler emits accordingly; the sequence below is the x86-64 form.

Modules loaded after startup cannot be given space in the static area, because the area’s size was fixed before the process created any threads beyond the first. Such a module gets a dynamic TLS block, allocated per thread on first use and reachable only through a runtime lookup. That is the boundary that separates the two cheap models from the two expensive ones.

Local exec: a constant displacement

The local-exec model applies to the main executable and to anything statically linked into it. The variable’s offset from the thread pointer is known at static link time and does not change, so the access is a single instruction with a constant displacement.

# local exec: %fs base is the thread pointer, offset fixed at link time
movl %fs:tls_var@tpoff, %eax

One instruction, no memory indirection beyond the access itself, no call. It is the cheapest possible form and the reason a statically linked program’s thread-local variables are effectively free compared with globals. The linker computes the offset by laying out all TLS segments in the output and assigning each a position; the TPOFF relocation then reduces to the constant.

The constraint is absolute: the module must be part of the initial static TLS area. A shared object cannot use this model, because the loader may place it anywhere in the static area depending on what else is loaded, and a dlopened object cannot use it at all because it is not in the static area.

Initial exec: one indirection through the GOT

The initial-exec model applies to shared objects that are expected to be loaded at startup. The offset is not known at compile time, but it is known at load time, so the loader stores it in a GOT slot and the access reads the slot and then indexes from the thread pointer.

# initial exec: load the offset from the GOT, then index from the thread pointer
movq tls_var@gottpoff(%rip), %rax
movl %fs:(%rax), %eax

Two memory operations instead of one, the first of which usually hits in cache because the GOT slot is read on every access. The relocation for the slot is R_X86_64_TPOFF64 and is resolved at load time, which is why the object must be in the static area.

The model is not the default for shared objects. A compiler targeting a shared library defaults to general dynamic, because it cannot know whether the object will be linked at startup or loaded later; initial exec is requested explicitly, by a per-variable attribute or a module-wide flag. C libraries use it for their own thread-local state, where the object is always present at startup and the extra indirection per access would be paid millions of times.

Its failure mode is precise and well known. A module built with initial-exec relocations that is dlopened after the process has started may fail to load, because the static area has no room reserved for it. Some loaders reserve surplus space in the static area to accommodate a few such modules, which is why the failure is not universal, but relying on that reserve is relying on an implementation’s slack rather than on the ABI.

The practical rule follows: a library that may be dlopened must use general-dynamic TLS, and a library that will always be linked at startup may use initial-exec.

General dynamic: a call with a module identifier

The general-dynamic model works for any module in any load situation. The compiler cannot know the module’s TLS offset, and it cannot even know the module’s identity, so it passes a pair to a runtime function that looks both up.

# general dynamic: pass (module id, offset) to the runtime, which returns the address
leaq tls_var@tlsgd(%rip), %rdi
call __tls_get_addr@PLT
movl (%rax), %eax

The argument is the address of a two-word descriptor in the GOT: a module identifier assigned by the loader and the variable’s offset within that module’s TLS block. __tls_get_addr reads the current thread’s control block, indexes the module’s dynamic block pointer array by the identifier, adds the offset, and returns the address.

The cost is a function call per access, plus the descriptor load, plus the lookup inside the runtime. On the first access from a given thread to a given module the runtime must allocate the dynamic block, which involves the allocator and is far more expensive than the subsequent lookups. The result is cached in the thread’s block pointer array, so later accesses are a couple of loads and a call.

__tls_get_addr is part of the platform’s runtime, not the C library’s public interface, and code that calls it directly is depending on a private contract. The compiler emits the call; hand-written assembly that mimics it must reproduce the calling convention exactly, including the fact that the argument is a pointer to the descriptor rather than the descriptor itself on x86-64.

Local dynamic: the same lookup without the GOT traffic

A module with several thread-local variables pays the general-dynamic cost per variable, including a separate descriptor and a separate call. Local dynamic reduces this for variables in the same module: the compiler calls the runtime once to obtain the module’s TLS block base, then addresses each variable at a constant offset from that base.

# local dynamic: one runtime call yields the module's block base
leaq tls_block@tlsld(%rip), %rdi
call __tls_get_addr@PLT
movl tls_a@dtpoff(%rax), %ecx     # first variable
movl tls_b@dtpoff(%rax), %edx     # second variable, same base

The saving is one call and one descriptor per group of accesses rather than per variable, which matters in a function that touches several thread-local fields. The constraint is that all the variables must belong to the same module’s TLS block, and a compiler emits the model for variables whose definition it can see in the translation unit it is compiling.

Compilers select among these models based on flags that describe the intended load situation, and the selection is per variable rather than per module. A variable can be annotated to request a specific model, and the compiler will emit a sequence for it, but if the annotation promises more than the linker can deliver — local-exec in a shared object, for instance — the link fails with a relocation error rather than silently producing a slower sequence.

Allocation, alignment, and the cost of the first touch

Each module’s TLS segment declares an alignment, and the loader must place it so that every variable satisfies its own alignment. On x86-64, a variable requiring 16-byte alignment forces the block to be 16-byte aligned, and the layout code in the loader handles this by rounding offsets up as it assigns them. The thread control block itself has a fixed layout the ABI defines, including a pointer to itself at a known offset so that a thread can find its own control block from the thread pointer.

The static area is allocated when the thread is created, as part of the thread’s stack allocation or a separate mapping, depending on the implementation. Its size is fixed at the moment the first thread is created, which is the precise reason a later dlopen cannot add to it. The dynamic blocks are allocated lazily and stored in the control block’s array of per-module pointers, indexed by the module identifier the loader assigned.

Destruction is the mirror image. When a thread exits, the runtime calls destructors for thread-local objects that have them, in the reverse of the order those destructors were registered. A destructor that accesses another module’s thread-local variable can force allocation of that module’s block during destruction, and the runtime must handle that case rather than assuming the set of blocks is frozen. The interaction is one of the reasons the TLS runtime is more intricate than its interface suggests.

/* Alignment and layout are visible in the object file: each TLS variable
   contributes to .tdata (initialised) or .tbss (zero), and the segment's
   alignment is the maximum required by any of them. */
#include <stdalign.h>

_Thread_local int counter;                       /* .tbss */
_Thread_local char name[64] = "unnamed";         /* .tdata */
alignas(64) _Thread_local double matrix[8][8];   /* forces 64-byte alignment */

int bump(void) { return ++counter; }

The distinction between .tdata and .tbss mirrors .data and .bss: initialised thread-locals occupy file space in every module, while zero-initialised ones occupy only the memory size. Since the block is per thread, .tdata bytes are copied for every thread created, which makes a large initialised thread-local array an expensive declaration in a program that creates many threads.

Where the models leak into program behaviour

The most consequential leak is the one already noted: an initial-exec module that is dlopened can fail to load, and the error arrives as a load failure rather than as a TLS-specific diagnostic. Build systems that produce a single shared object used both as a startup dependency and as a plugin must choose general-dynamic, accepting the per-access call.

The second is performance predictability. A general-dynamic access inside a tight loop is a call per iteration, which no amount of inlining removes because the call is emitted by the compiler as part of the access sequence. Moving the address computation out of the loop — taking the address of the thread-local once and indexing through the pointer — converts the repeated lookup into one, and that transformation is legal because the address of a thread-local variable is stable for the life of the thread.

The third is interaction with the loader’s ordering. A TLS variable’s initial value comes from the module’s .tdata image, copied at block creation, so a constructor that writes the variable before any other thread exists initialises only the initial thread’s copy. The other threads get the .tdata value, not the constructor’s write. Code that expects a constructor to establish a per-thread default must set it in a way that each thread runs, or accept the segment’s initial bytes as the default. That ordering constraint, and the more general question of when initialisation code runs, connects back to the load sequence described in From execve to _start and to the deferred resolution in Lazy Binding and the Procedure Linkage Table.