L2 Slices, Address Hashing, and Partition Camping

The cache that sits between the streaming multiprocessors and device memory is one logical cache with a single name, but no single piece of silicon holds it. It is built as a set of slices, each sitting next to a fraction of the memory controllers, and every address in the device’s address space belongs to exactly one slice. A load that misses in the per-SM level-one cache reaches a specific slice, determined by a function of the address that the hardware chooses and does not publish. When that function distributes a kernel’s traffic evenly, the whole subsystem is used; when a stride makes the function land repeatedly on a subset, the kernel saturates a fraction of the machine and leaves the rest idle.

One cache, many physical banks

Each slice holds tags, data banks, and the miss-status machinery needed to track outstanding requests to the DRAM channels behind it. Because the slices are independent, they can serve requests concurrently, and the aggregate bandwidth of the level-two cache scales with their number. A device with sixteen slices can sustain roughly sixteen times the request rate of one slice, provided the requests are spread across them.

The channels behind the slices impose their own constraint. Each controller drives a subset of the DRAM devices, and those devices have banks and rows with the access properties described in the row-buffer discussion, so a slice whose requests all target one DRAM row is limited by the bank cycle time rather than by the bus. The design goal of the mapping is to make a sequential scan of memory touch every slice in turn and, within a slice, rotate through the DRAM banks.

L2 capacity on recent parts ranges from a few megabytes to tens of megabytes; the exact figure is a property of the part and is reported by the runtime. Its bandwidth is larger than DRAM bandwidth by a small factor, which is what makes it worth staging traffic through, and it is the reason a kernel whose working set fits in L2 can exceed the bandwidth DRAM alone would provide.

Why hashing beats modulo

The obvious way to assign an address to a slice is to take a few bits from the physical address and use them as an index, which is what most CPU last-level caches do. That scheme has a known weakness: an access pattern whose stride is the slice count multiplied by the line size, or any multiple of it, places every request on one slice. Successive power-of-two strides are exactly the patterns that high-performance code likes, because they let the compiler replace multiplication with shifts.

GPU memory systems therefore use a hash of several address bits rather than a direct field extract. The hash is constructed so that a range of consecutive addresses touches all slices, and so that common power-of-two strides still spread. The specific function is not documented; it differs between generations and is one of the parts of the machine that microbenchmarking papers go to some trouble to reverse-engineer, and even then only in terms of observable behaviour rather than the circuitry.

Hashing removes the pathological cases that a linear mapping has, but it does not remove the possibility of resonance. A stride that happens to interact badly with the hash, or an access pattern whose concurrent requests occupy a small fraction of the address space, can still concentrate traffic. This is what partition camping means, and it is usually diagnosed by the shape of the achieved bandwidth curve as a function of stride rather than by a counter alone.

Patterns that resonate

The classic mechanism is concurrency, not stride alone. Every SM tends to issue its loads at the same offset within a block at the same time, so a kernel whose blocks are aligned to a fixed multiple of a large power of two issues a burst of requests whose addresses differ by that multiple. If those requests all map to the same slice, the burst is serialized while the rest of the memory system idles, and the peak bandwidth of the kernel is a fraction of the device’s capability. The effect is transient inside each burst, which makes it appear as a sustained throughput loss rather than as an obvious stall.

Three constructions avoid it. The first is to offset each block’s starting address so that the concurrent addresses differ by a value coprime with the hash’s effective period, which can usually be done from the block index alone. The second is to make the array’s row pitch, or the tile pitch inside a scratchpad, an odd multiple of the line size rather than a power of two, the same padding trick that fixes bank conflicts in shared memory. The third is to change the assignment of work to blocks so that neighbouring blocks do not touch neighbouring addresses at the same moment, which spreads the concurrent working set across the address space without any padding at all.

None of the three requires the programmer to know the hash function. All three work by ensuring that the set of addresses in flight at any instant covers a wide range, which is the only property the hash can preserve.

The cache is also the point of coherence

The last-level cache is where device-wide coherence happens. Global memory writes go to L2 and become visible to other threads when the line is written or invalidated there, not when it reaches DRAM. This is what makes the memory model’s visibility rules expressible at all: a release operation orders prior writes so that they are visible at the L2 level, and an acquire operation ensures that subsequent reads do not bypass it.

It also explains why the atomics discussed in the contention discussion cost what they cost. An atomic operation on a global address executes at the slice that owns the address, and the slice serializes operations to the same location, performing them as read-modify-write sequences against its own bank. Atomic throughput therefore scales with the number of distinct addresses being targeted and stops scaling entirely when every thread of a grid increments one counter.

Because L2 is sectored, a partial write costs more than a full one. A store that covers only part of a 32-byte sector requires the previous contents to be preserved, which means the sector must be present before the write completes. Atomics are applied against the sector at the owning slice, so the operation’s width matters as much as the number of threads issuing it.

Reserving part of the cache for reuse

A streaming kernel that reads an array once and writes an array once has no use for cache capacity beyond the write-allocate it is forced to do, and it will evict whatever else is resident. A kernel that runs alongside it and depends on a small hot buffer being resident is therefore at the mercy of the neighbour’s traffic, even though the hot buffer would comfortably fit.

Recent architectures let a kernel designate an address range as persistent and specify what fraction of L2 it may occupy. The normal, non-persistent traffic is then restricted to the remainder, so a repeated read of a hot structure stops competing with bulk streaming traffic. The mechanism is a hint, not a guarantee: the reservation is a maximum on what the designated range may take and a ceiling on what everything else may take, and the hardware does not promise that any particular line survives. It works well for lookups and index structures with genuine reuse and does nothing at all for a stream.

The cost of the feature is in the configuration, not the runtime: the size of the reserved region is chosen by the application through a device limit whose effect depends on the driver and the part, so code that sets a persistence window needs to check the values the driver reports rather than assuming the request was honoured. As with the register budget, the achievable setting is a property of the part, and treating it as a constant is the way to get a kernel that behaves inconsistently across a fleet.