Row-Buffer Locality Inside GPU DRAM

Below the last-level cache the memory system changes character. Cache misses are independent events that a banked SRAM can serve in any order, but a DRAM access is a sequence of timed electrical steps against a physical array, and the order of those steps determines how much of the bus is usefully occupied. A GPU presents the controller with far more concurrent demand than a CPU does, and it reaches a high fraction of its theoretical bandwidth only while its kernels stay spatially local. When it does not, the achieved bandwidth falls without any cache or scratchpad being at fault.

The row buffer is a cache of one line

A DRAM chip is divided into banks, and each bank contains rows that are several hundred to a couple of kilobytes wide. Reading a location inside a row begins by transferring that entire row into the bank’s row buffer, an operation that takes tens of nanoseconds and closes whatever row was previously open. Reading a second location from the same row, with the row still open, is answered from the row buffer at a fraction of the cost. Reading a location in a different row of the same bank requires closing the open row, which itself costs time, before the new row can be activated.

The bank is therefore a resource with one slot, and every access either hits the open row, misses and pays the full sequence, or is reordered by the controller so that it becomes a hit. The controller runs a scheduling policy over the queued requests: it looks for requests targeting an open row, issues those first, and defers the ones that would force a closure. Memory controllers therefore reorder aggressively, which converts DRAM latency from a fixed number into a distribution whose spread depends on the traffic mix.

The unit moved per command is matched to the cache sector. A DRAM device transfers a burst sized by the bus width and the burst length, on the order of a few tens of bytes, and the caches in front of it request data in 32-byte sectors, as described in the transaction discussion. The matching is deliberate: a request that the caches would discard is a request the DRAM interface should never issue.

Bank groups and command spacing

Modern DRAM standards organize each channel’s banks into small groups and apply different minimum spacings between consecutive commands depending on whether they stay within a group or move between groups. A command issued to a bank in a different group can follow sooner than one issued to another bank in the same group, because the internal buses of a group are shared among its banks. The practical effect for a scheduler is that spreading requests across bank groups buys throughput that spreading them within one group cannot.

This is one of the reasons the address interleaving used by GPU memory controllers distributes consecutive sectors across groups and channels rather than filling one bank at a time. Sequential access is not merely good because of row hits; it is good because the interleaving guarantees that consecutive requests land in different banks and groups, so the activations can overlap and the bus does not wait on a single array.

There is a second periodic cost. DRAM cells are capacitors and leak, so the controller must refresh every row within a window of tens of milliseconds, issuing refresh commands continuously and refreshing a small fraction of the rows at a time. Refresh consumes bus time and forces rows closed, so a kernel that is already at the bandwidth limit pays for it out of the same budget as its data. The overhead is a small percentage of the bus in the common case and a larger one under high bank pressure, when refresh closes rows that the scheduler had carefully kept open.

How a GPU stack is wired

The high-bandwidth memory used on accelerator cards is a stack of DRAM dies connected by vertical vias to a logic die, with the whole stack mounted beside the processor and connected over a wide interface. The interface is wide and relatively slow compared with the narrow, fast interfaces used for graphics memory, and a stack is divided into several independent channels, each of which has its own banks and bank groups and functions as a smaller DRAM subsystem in its own right.

The width is the design decision. Roughly a thousand data pins per stack, spread across a handful of channels, give the peak bandwidth from many parallel arrays rather than from a fast bus. That has a direct consequence for bank-level behavior: each channel carries only a fraction of the stack’s total width, so more requests are needed to move the same number of bytes and the controller has more opportunity to interleave them across banks. The cost is that every one of those requests must be scheduled and tracked, which is why the controller’s queue depth and policy matter as much as the array’s raw timing.

The same distribution is why the level-two cache is partitioned rather than centralized: each slice sits beside a subset of the memory controllers, a layout that predates stacked memory. The address hash described in the L2 slice discussion is what keeps the traffic balanced across the slices. A hash that spread requests evenly over slices also spreads them evenly over channels, which is the property the DRAM scheduler needs to keep several rows open at once.

What locality looks like from the kernel

Two properties decide whether the DRAM scheduler can do its job.

The first is spatial: the addresses a kernel touches over a short interval should fall into few rows. A streaming read of a large array satisfies this trivially. A gather from many scattered locations does not, and no amount of coalescing within each warp fixes it, because coalescing operates within a warp and the scheduler operates across the whole device’s outstanding requests.

The second is the number of concurrent streams, and it is the property that catches people out. A kernel in which each thread walks its own long strided sequence generates a stream per thread, and the controller must interleave them. Since each bank holds one open row, a set of streams that map to the same bank can thrash it. The bandwidth falls not because the data is uncoalesced but because the open-row rate drops.

The remedy on the kernel side is to reduce the number of distinct address streams in flight and to make each of them sequential. Tiling does this directly: a block that reads a contiguous tile of an array produces a short, dense burst rather than a long stroll, and the same tile loaded into a scratchpad is then reused many times without returning to DRAM at all, which is the reuse argument made in the tiling discussion. Fewer, larger, more sequential transfers are easier for the controller to schedule than many small ones, even when both move the same bytes.

Where this stops being controllable

The hash and the scheduling policy are not documented and they change between generations. A kernel tuned to a stride that happened to work well on one part may behave differently on the next, and the microbenchmarking literature exists largely because the mapping must be measured rather than read. The durable guidance is structural: consecutive addresses, a small number of streams, reuse within a block, and transfers wide enough to cover a full sector and aligned to it. Those properties hold regardless of what the controller does with the queue.

There is also a floor to how much analysis helps. If a kernel must gather scattered data, the DRAM system will pay for the scatter somewhere, either as row misses, as wasted sector transfers, or as both, and the design question becomes whether to reorganize the data once so that the gather disappears. That is the same trade that layout transformation makes one level up, evaluated here against row misses instead of wasted sectors.