Constant Memory and the Read-Only Load Path

Two of the fastest paths into a GPU do not look like memory accesses at all. One is a small bank of values written by the host before the kernel starts and read by every thread; the other is an ordinary global load that the compiler has been told will never see a conflicting write. Both exist because the common case in numeric code is a read of something uniform or something immutable, and both cost less than the general case for exactly that reason. They also fail in ways that the general path does not, which is why they are worth separating from ordinary loads.

A bank written from the host, read by the grid

Constant memory is 64 KB of device memory reserved for values the host uploads before a launch. Kernels read it and cannot write it. The upload uses an explicit copy call that names the symbol on the device side, so the binding is by symbol rather than by pointer, and the pointer-based alternatives are not interchangeable with it.

The reason for the separate space is the cache it feeds. A small constant cache sits next to the instruction pipeline on each SM, and a read that is uniform across a warp is served as a single broadcast, much as an immediate operand is fed from the instruction stream. Uniformity is the key property: because every lane wants the same word, there is nothing to select and nothing to queue, and the value can be delivered as if it were part of the instruction.

The compiler exploits a second property of the same bank. When the index into constant memory is known at compile time, the value can be embedded directly in the instruction stream as an immediate constant, which removes the memory access entirely. Loops that read a fixed coefficient array with an unrolled and constant index therefore cost nothing at all, and it may not be obvious from the source that this happened. The instruction set distinguishes an immediate operand from a constant-bank operand, and the compiler chooses between them rather than the programmer.

The 64 KB limit is a hard partition of that address space. It is large enough for coefficient tables and small lookup tables, and too small for anything resembling a dataset.

The broadcast rule and what breaks it

A warp-wide read of one constant address is one access. A warp whose lanes read different constant addresses is divided by the hardware into as many serialized accesses as there are distinct addresses, because the cache returns one word per pass along this path. Thirty-two lanes with thirty-two different indices therefore cost thirty-two passes for one instruction, which is worse than the same pattern would cost in global memory, where the sectors would at least coalesce by proximity.

This produces a characteristic mistake with the space: indexing a constant table by thread ID. The code reads naturally, looks like a lookup, and converts the cheapest memory in the machine into the most expensive one. The remedy is either to make the index uniform by restructuring the computation so every lane of a warp works from the same table entry at that step, or to move the table to global memory and accept the sector cost of the scattered reads, which is usually smaller.

The read-only path below tolerates divergence much better, and the reason is architectural rather than accidental. Constant memory is built for a single broadcast word per pass; a cache that handles general loads is built to return a run of bytes.

Loads that promise not to observe a write

The general global load path must handle the possibility that any address it reads has been modified by some other thread. That requirement forces the hardware to include the load in the coherence protocol and to invalidate cached copies when a peer writes. A large class of reads in real kernels never need that guarantee. Input arrays are read by many threads and written by none, which means a cached copy can never become stale and no coherence action is required.

The instruction set exposes this as a distinct load form, described as non-coherent, and the compiler emits it only when it can prove the property from the source. The proof comes from the type qualifiers: a pointer declared as pointing to constant data and marked as not aliasing any other pointer in the kernel is enough. With that information, the compiler can route the load through a cache that does not participate in coherence, which is cheaper to maintain and in early implementations was physically separate, a remnant of the texture unit that graphics workloads had always used for read-only fetches. Later architectures merged the storage of that cache with the level-one data cache while keeping the distinct access path and its non-coherent semantics.

The practical difference is throughput and cache pressure. Reads that the compiler can classify as non-coherent do not create coherence traffic, and they can be served from a structure dedicated to them, which matters most when a kernel is read-heavy and the general path is saturated. For kernels that read a large array exactly once, the distinction buys almost nothing, because there is no reuse to cache and the DRAM traffic is unchanged.

When the promise is broken

The non-coherent path is an unchecked promise. The qualifiers that enable it are assertions about the program, and the compiler does not verify them. If a kernel writes to an array through an ordinary pointer while another path reads the same memory through a read-only one, the reader may return a stale value; whether it does depends on cache state and timing, so the failure is intermittent and may not appear in a short test.

The dangerous cases are aliasing through unions or casts, a read-only view of a buffer that a later phase of the same kernel modifies, and a kernel that both reads and writes a reduction target. The qualifiers are intended to exclude all of these by construction, and code that defeats them by casting away const acquires the risk along with the optimization. The safe rule is that the read-only path may be used for memory that no device code writes for the entire lifetime of the kernel, which for the usual input-buffer case is true by construction and for the usual in-place case is false.

There is a related hazard specific to constant memory. Because the upload happens on the host between launches, a kernel launched while a copy to the constant bank is in flight on another stream can observe a partially updated table. The copy and the read have to be ordered, either by placing them on the same stream or by an explicit synchronization, and the compiler has no way to see the conflict because the table is not a parameter.

Choosing between the paths

The decision is driven by three properties: uniformity across the warp, reuse, and immutability for the kernel’s lifetime.

A value that is uniform and reused is the ideal constant. A value that is uniform and used once gains little from the constant path, and the ordinary path handles it with the same coalescing as any other load. A value that varies per lane is the wrong constant and may be the right read-only load, since the read-only path coalesces its accesses as a general load does. A value that varies per lane and is not reused is simply a global load, and the qualifiers that route it elsewhere mostly change cache accounting rather than the access itself.

The choice also interacts with the register budget. A table small enough to sit in constant memory is small enough that a kernel might instead compute its entries, and a computed value costs no memory traffic but costs arithmetic and registers, which is the trade described in register pressure and the spill into local memory. The address spaces are not alternatives to each other so much as a set of price lists, and the read-only and constant paths are the two cases where the price is unusually low provided the access pattern holds still.