Sector Counting and Why Cache Hit Rate Misleads

A hit rate is a ratio, and a ratio with a small denominator can be excellent while the absolute number of misses is ruinous. The counter that matters is not the fraction of requests served, it is the number of sectors moved, and the two diverge whenever the requests are not shaped the way the hardware wants.

The sector as the unit of transfer

Memory is fetched in sectors, not in bytes and not in cache lines. A cache line is divided into a small number of sectors, commonly four sectors of 32 bytes for a 128-byte line. A request from a warp is split into the set of sectors it touches, and each touched sector is fetched or hit independently.

This granularity exists because fetching a full 128-byte line to satisfy a 4-byte request would waste 97% of the transfer. Sector granularity lets a scattered access pattern pay for only the 32 bytes it actually uses, at the cost of more transactions and more tag lookups. The design is a compromise: fine enough that scattered access is not catastrophic, coarse enough that sequential access amortizes the tag overhead over 32 bytes.

The consequence for accounting is that a kernel’s traffic is the number of distinct sectors it touches, multiplied by 32 bytes. Two kernels issuing the same number of load instructions can differ in traffic by a factor equal to the ratio of sectors touched per instruction.

One warp, two access patterns

Consider a warp of 32 threads each loading a 4-byte word. If the 32 words are consecutive, they span 128 bytes, which is four sectors, and the warp’s load becomes four sector requests. If the 32 words are strided so that each lands in a different 128-byte line, they touch 32 lines and at least 32 sectors, and the traffic is eight times larger for the same instruction count.

// Contiguous: 32 threads x 4 bytes = 128 bytes = 4 sectors.
out[i] = in[base + tid];

// Strided: each thread lands in its own 128-byte line.
out[i] = in[base + tid * 32];   // 32 sectors, 8x the traffic

The strided case is not merely slower in proportion to the extra traffic. Each warp instruction now generates 32 separate memory transactions rather than 4, and the transaction rate itself is a limited resource. A memory pipeline that can process a bounded number of transactions per cycle becomes the bottleneck long before the byte bandwidth is exhausted. This is why some access patterns degrade by much more than the byte ratio suggests: the limit has moved from bytes per second to transactions per second.

The counter that reveals this is the ratio of sectors requested to sectors that would have been requested by a perfectly coalesced access of the same logical size. A kernel reading 4 bytes per thread with a ratio of one sector per thread is touching eight times the minimum.

Pattern	Sectors per warp-load	Relative traffic
32 consecutive 4-byte words	4	1.0
32 words with stride 2	8	2.0
32 words with stride 8	32	8.0
32 words scattered across lines	32	8.0
32 words each 32 bytes apart	32	8.0

The stride-2 case is worth noting because it is common in practice: an array of two-component structs read one component at a time produces exactly this pattern, and the fix is to read both components together or to split the struct into separate arrays.

Why the hit rate is the wrong headline

A cache hit rate is computed as hits divided by hits plus misses, where the unit is a request. Because a request may touch one sector or several, and because the number of requests is determined by the access pattern rather than by the data volume, the denominator moves with the code shape.

Two kernels illustrate the problem. Kernel A reads a large array sequentially. Almost every request misses in L2 because there is no reuse, and the hit rate is low. Its traffic is minimal, because every byte fetched is used. Kernel B reads a small array with a stride that touches each line repeatedly. Its hit rate is high, because most requests hit, but the traffic is eight times what the useful data requires and the kernel is slower.

Kernel A has a bad hit rate and is efficient. Kernel B has a good hit rate and is wasteful. Tuning for hit rate would push A in the wrong direction, since the only way to raise A’s hit rate is to introduce reuse that does not exist, and it would leave B alone, since B’s hit rate is already excellent.

The usable metrics are absolute: sectors fetched from DRAM, sectors fetched from L2, sectors requested by the instruction stream. The hit rate is a derived quantity and is only informative when the request pattern is fixed. The traffic counters are informative always.

Where a high hit rate still costs

Hits are not free. An L2 hit costs a lookup and, on a miss in the lower level, a transfer that occupies the same interconnect the DRAM path uses. A kernel whose hit rate is high because it re-reads the same lines many times is spending interconnect bandwidth on every one of those reads, even though it is not spending DRAM bandwidth.

The distinction shows up when a kernel is scaled up. A working set that fits in L2 with one block resident may not fit with eight blocks resident, and the hit rate falls as the grid grows. The measured rate then depends on the launch configuration in a way that looks like a scaling failure but is really a capacity transition. The mechanism is the same one described under occupancy pressure in occupancy as a latency-hiding budget, and the diagnostic is to plot hit rate against resident block count and look for the knee.

There is a second, subtler cost: a hit still occupies a request slot and a return path. A kernel that issues a huge number of hits because it re-reads shared data from L2 could instead stage that data in shared memory once and read it many times from there, spending no interconnect bandwidth at all. Whether that trade is worth it depends on the reuse count and on the shared-memory capacity available, which is the classic tiling decision.

Write traffic and the allocation question

Writes are accounted differently from reads and are a frequent source of unexplained traffic. A write that covers a full sector can be performed without reading the old contents. A write that covers part of a sector may require the sector to be fetched first, so that the untouched bytes can be merged. The fetch is real DRAM traffic and appears in the counters even though the kernel performed no read.

A warp writing 32 consecutive 4-byte words covers four full sectors and generates no fetch. A warp writing 4 bytes into each of 32 different sectors generates 32 fetches plus 32 writes, doubling the traffic and more. The same source-level statement produces either pattern depending on the address calculation.

The mitigation is to ensure writes are full-sector and, where possible, full-line. Padding a data structure so that each thread’s output lands in a distinct full sector, or restructuring so that a warp’s writes are contiguous, removes the fetch traffic. In some cases a partial write is unavoidable, and the honest accounting is to include the fetch in the kernel’s byte count when computing intensity, which can move a kernel’s position on the roofline substantially.

Reading the counters together

The sector counters are only interpretable in combination. A workable set:

• Sectors requested by the instruction stream. This is the demand, and it is fixed by the access pattern.
• Sectors fetched from L2. This is what the L1 and the request coalescing absorbed.
• Sectors fetched from DRAM. This is the traffic that matters for the bandwidth ceiling.
• The ratio of the last two to the first.

A kernel where the demand is much larger than the L2 traffic has good locality and is served by caches. A kernel where the demand is equal to the L2 traffic has no reuse at all, and the L2 is acting as a pass-through. A kernel where the DRAM traffic is a large fraction of the L2 traffic has an L2 capacity problem. A kernel where the demand itself is far above the logical data size has a coalescing problem that no cache can fix.

The last case is the one that a hit-rate metric hides. If each thread requests eight times the sectors it needs, the demand is eight times the data, and even a perfect cache serving all of it from L2 leaves the kernel issuing eight times the transactions. The fix is in the address calculation, not in the cache.

Why this constrains the layout, not just the loop

Because the sector is the transfer unit, data layout determines traffic as much as the loop does. An array of structures where each thread reads one field touches a sector for every element and uses 4 of its 32 bytes. The same data as a structure of arrays lets a warp read 32 consecutive floats in four sectors, using every byte.

The transformation is mechanical but has costs. Splitting a struct into parallel arrays changes the memory access pattern for every consumer, and a consumer that needs all fields of one element now issues several loads instead of one. The trade is favourable when the dominant consumer reads one field across many elements, and unfavourable when the dominant consumer reads all fields of few elements. There is no layout that is right for both, which is why the decision has to be made against the actual access pattern rather than by convention.

The related question of how the shared-memory side of the hierarchy behaves, including bank structure and its own conflict accounting, belongs to the memory cluster; the DRAM-side accounting here is what determines whether a kernel can reach its bandwidth ceiling at all.