How a Warp Turns Thirty-Two Loads into Memory Transactions

A single instruction in a kernel is not a single memory operation. When a warp executes a load, thirty-two threads each supply an address, and the memory system must decide how many DRAM transfers those thirty-two addresses require. The answer ranges from one to thirty-two depending on nothing but how the addresses relate to each other. This is one of the few places in GPU programming where a purely mechanical property of the code — the arithmetic that produces an index — can outweigh the algorithm itself in its effect on throughput.

The request a warp actually issues

Threads of a warp are scheduled together, so one load instruction covers the whole warp. The address unit receives up to thirty-two addresses at that moment and forms them into a set of requests. Adjacent addresses that fall in the same aligned block are merged; addresses in different blocks are not. The merged set is what leaves the SM for the level-one cache and, on a miss, for the level-two cache and DRAM.

The hardware does not care about the order of the addresses, only their positions. A warp whose threads read addresses in a scrambled order but covering one contiguous run of bytes forms exactly the same requests as a warp that reads them in order. Reversing an array index inside a warp costs nothing. Interleaving threads across a large stride costs a great deal.

// Thread t reads element t: one contiguous 128-byte run per warp.
out[t] = in[t];

// Thread t reads element t*4: the same 32 loads spread over 512 bytes,
// two addresses per 32-byte sector.
out[t] = in[t * 4];

// An odd multiplier permutes the lanes inside the same 32-element run.
out[t] = in[(t * 17) & 31];

The third case defeats the assumption that coalescing requires ascending addresses. It requires a dense footprint. Any permutation of a warp’s addresses within the same aligned 128-byte window touches the same four sectors, because the hardware forms requests from positions rather than from order. A stride that spreads those same 32 accesses across several hundred bytes is a different matter, and the cost scales with the number of sectors the set touches.

Sectors are the unit, not lines

The cache line on NVIDIA GPUs is 128 bytes, and it is composed of four 32-byte sectors. The sector is the granularity at which the memory system moves data: a request that touches eight bytes of a sector transfers the whole 32 bytes, and a request that touches all four sectors of a line transfers 128. The distinction matters because a load that covers 128 contiguous bytes costs one line but the line is fetched as four sectors, and a load that touches one byte per sector across a 128-byte span still costs four sectors even though it moves only four useful bytes.

So the efficiency of an access is the ratio of bytes the warp asked for to bytes the transfer actually moves, and bytes moved is the sector count multiplied by 32. A warp of 32 threads reading 4-byte words contiguously asks for 128 bytes and touches four sectors: full efficiency. The same warp reading with a stride of 32 bytes asks for 128 bytes spread over a kilobyte and touches thirty-two sectors, one per thread: 128 useful bytes out of 1024 moved, or one eighth.

On the DRAM side the same arithmetic applies with a coarser unit. DRAM transfers occur in bursts tied to the bus width, and the memory controller schedules them per bank and row. A stride that lands in a new row on every access pays a row activation as well as the transfer, which is why scattered access patterns degrade more than the sector arithmetic alone predicts.

Why a partial write must read first

Reads have no such complication: fetching a sector that is only partly needed wastes bandwidth but requires no extra step. Writes do. If a warp stores four bytes into a sector, the memory system cannot simply push those four bytes to DRAM, because the other twenty-eight bytes of the sector must retain their previous values. The line is fetched, the bytes are merged, and the line is later written back. A store that misses can therefore cost a read as well as a write.

This is the mechanism behind the familiar advice to make stores contiguous and aligned. A warp storing 4-byte words at consecutive addresses writes four complete sectors and needs no fetch at all, provided the cache can allocate the line without reading it — hardware supports this for full-sector writes. A warp storing one byte per thread at a stride of 32 bytes touches thirty-two sectors, each with a byte written and thirty-one untouched, and every one of those sectors must be read before it can be modified.

The same rule applies to structures. Writing one field of a multi-field struct from each thread leaves the rest of the struct’s cache lines untouched by that store, and if the struct is 64 bytes wide the warp’s stores span many sectors with holes in them. Struct-of-arrays layouts exist partly to avoid this: when each thread writes one element of a dense array, the warp’s stores tile the sectors exactly.

Divergence splits the request

When a branch diverges, the two paths are executed in sequence on the same warp — interleaved at instruction granularity on newer architectures, but never at the same time — and each path issues its own load instruction for the threads that are active on it. The memory system sees two requests instead of one, each covering a subset of the warp. If the two subsets happen to fall in the same sectors, the second request hits in cache and costs little beyond the instruction issue. If they fall in different sectors, the cost doubles.

This is why the standard advice for divergent code is to arrange the branches so that each side is spatially clustered — for instance by sorting work by category before the kernel, or by assigning categories to blocks rather than to lanes within a warp. The divergence itself is unavoidable in general; the extra transactions are avoidable, and they can dominate the cost.

A subtler case is predication. A short conditional body is compiled into predicated instructions rather than a branch, so all thirty-two lanes issue the load and the inactive ones are masked. If the masked-off lanes still supply addresses, the hardware may or may not suppress those accesses depending on whether the architecture forms the request from active lanes only. Architectures have differed here, and code that depends on masking to reduce traffic can behave differently across generations. The reliable construction is to keep the address computation inside the branch so the request is never formed for inactive lanes at all.

Predicting the sector count from the source

The ratio of useful bytes to transferred sectors can be derived from first principles by simulating the address pattern, without a profiling run.

def sectors(addrs, size, sector=32):
    """Count distinct 32-byte sectors touched by one warp-wide access."""
    hits = set()
    for a in addrs:
        for b in range(a, a + size):
            hits.add(b // sector)
    return len(hits)

base = 0
row = [base + 4 * t for t in range(32)]          # contiguous
col = [base + 4 * 64 * t for t in range(32)]     # stride of one 256-byte row
print(sectors(row, 4), sectors(col, 4))          # 4 sectors, 32 sectors

The function is a model, not the hardware: real implementations may apply different rules for the read-only path or for shared memory. But for global loads and stores it predicts transaction counts closely enough to guide layout decisions, and it makes the eight-to-one difference between the two access patterns explicit rather than anecdotal.

Where the model stops

Three qualifications keep this arithmetic honest. First, the level-one cache absorbs repeats: a stride that misses on the first pass may hit on the second if the footprint fits, which is why a poorly coalesced access inside a loop over a small array is often cheaper than the sector count suggests. Second, L2 is shared and partitioned by address, and a request that misses L1 does not necessarily reach DRAM. Third, the sector count is a lower bound on cost, not a timing prediction; the achieved bandwidth also depends on how many requests are in flight, which is a function of occupancy and of how many independent loads the compiler issued before the first use.

The practical ordering is still reliable. Contiguous, aligned accesses cost the fewest sectors, strided accesses cost more in proportion to the stride, and writes cost more than reads when they are partial. Every layout transformation described in stride and layout transformation — transposing through a scratchpad, padding a shared array, splitting a struct — is an attempt to move a kernel from the strided or partial-write case into the contiguous one, and the sector arithmetic above is the measure of whether it worked.