Data Movement as the Real Bottleneck in Offload Design

Every offload decision is a bet that moving work to a unit costs less than leaving it where it is. The work itself is usually arithmetic, and arithmetic is cheap in every direction: a multiply-add costs a few hundred femtojoules and a cycle or two. What differs between a core and a unit is not the arithmetic but where the operands come from and where the results go. An offload that saves arithmetic and adds movement loses, and an offload that reduces movement wins even when it does more arithmetic. Getting this right requires accounting for bytes rather than for operations.

The arithmetic-intensity ladder

The useful measure is arithmetic intensity: operations performed per byte moved. A computation’s intensity is fixed by the algorithm; what varies between implementations is the level of the memory hierarchy at which the bytes are counted. The same kernel can be described as compute-bound or memory-bound depending on whether the bytes are counted at registers, at cache, or at DRAM.

Offloading changes which level the bytes come from, and that is the whole story. A host core running a kernel reads its operands from cache and writes results back, and the DRAM traffic is whatever the cache could not hold. An accelerator with a private memory must be given the operands explicitly, so the DRAM traffic becomes whatever the transfer moved regardless of what the cache could have held. An offload that duplicates data into a device memory and back can therefore have higher DRAM traffic than the on-core version it replaces, even though the kernel itself is faster.

The roofline formulation makes this quantitative. For a machine with peak floating-point rate $P$ and memory bandwidth $B$, the attainable performance for a kernel of intensity $I$ is $\min(P, I \cdot B)$. A kernel with intensity below $P/B$ runs at $I \cdot B$ no matter how fast the arithmetic units are. The value of interest when comparing offloads is not the peak but the point where the two lines cross, and whether the kernel sits on the memory-limited side of it.

def attainable(peak_ops, bandwidth, intensity):
    """Bytes/s * ops/byte gives ops/s; the min is the roofline."""
    return min(peak_ops, intensity * bandwidth)

def crossing(peak_ops, bandwidth):
    """Intensity at which the kernel stops being memory-bound."""
    return peak_ops / bandwidth

# a memory-limited kernel gains nothing from a wider datapath
print(attainable(1e12, 25e9, 8), crossing(1e12, 25e9))

The second output is the number worth remembering: for a machine with teraflop-scale arithmetic and tens of gigabytes per second of bandwidth, the crossing intensity is in the tens of operations per byte. Kernels below that intensity are limited by movement on every machine of that class, including the accelerator, and adding more arithmetic units to the accelerator changes nothing.

Counting the full path

The transfer cost of an offload is not the data’s size. It is the sum of every movement the offload adds, and the list is longer than it first appears.

Transfer in and out. The operands must reach the device and the results must return. If the device has memory of its own, this is a copy across the interconnect, and the traffic is twice the data size for a computation that reads and writes. If the device reads host memory directly, the copy disappears but the traffic does not — it moves over the interconnect instead, at a different bandwidth and with a different latency.

Descriptor and control traffic. Each submission carries a descriptor, an index update, and a notification. For large transfers this is negligible; for small ones it can exceed the payload. A descriptor is on the order of tens of bytes and a notification involves several accesses, so a transfer of a few hundred bytes has an overhead of a similar order to its data.

Translation traffic. Address translation for the device is a page-table walk per miss, and a device that issues many small transactions with poor locality generates walk traffic disproportionate to its payload. The mechanisms that manage this cost are described in Device DMA and the IOMMU Boundary; the point here is that the walk is part of the offload’s traffic and belongs in the accounting.

Synchronization traffic. Waiting for completion involves a poll or an interrupt, and the poll reads a memory location the device wrote, which bounces a cache line between the two. A tight polling loop on a completion index generates a stream of coherence traffic that is invisible in the data counters and real in the bandwidth budget.

State save and restore. If the device is shared and preempted, the state of an interrupted context moves through memory. This is discussed in Scheduling and Preemption on Accelerator Queues, and it is a traffic cost proportional to the number of contexts and the switch rate rather than to the work.

The accounting rule is to count bytes at every boundary the offload introduces and to compare the total against the bytes the on-core version moved. The offload wins on traffic only if it moves fewer bytes, which for a data-parallel kernel it often does — because the accelerator’s local memory holds operands across many uses, whereas the core re-reads them — and for a pointer-chasing kernel it usually does not.

Energy follows the same ladder

The energy hierarchy is steeper than the latency hierarchy. Moving 32 bits from a register costs a fraction of a picojoule, from an on-chip cache on the order of ten picojoules, from DRAM about a nanojoule, and across an interconnect to a device’s memory more still, because the interconnect’s physical layer costs dominate at that distance.

The consequence is that an offload’s energy is decided by its crossings. A kernel that runs 10× more efficiently in a device, but requires the data to cross a boundary twice, may consume more energy than the core version if the boundary dominates the total. The condition for an energy win is that the savings inside the device exceed the cost of the crossings, and the savings scale with the intensity of the kernel while the crossing cost scales with the size of the data. High-intensity kernels therefore amortize a crossing; low-intensity ones do not.

This asymmetry is why the same offload can be a throughput win and an energy loss, and why the two metrics must be evaluated separately rather than assumed to move together. It also explains a design pattern that looks odd from a pure-performance view: units placed close to their data — a checksum engine in the network interface rather than in a coprocessor, an encryption unit beside the storage controller — win overwhelmingly, because they eliminate a crossing rather than adding one.

Where the boundary should sit

Given a computation and a set of possible placements, the boundary should be drawn where the data already is. Three cases make the principle concrete.

A computation that consumes data produced by the host and consumed again by the host should stay on the host unless the computation is expensive relative to the data’s size. The crossing is pure overhead in this case, and it is only worth paying if the local speedup is large.

A computation that consumes data produced by a device, or produces data a device will consume next, should be placed on that device. A filter that runs immediately after a decode, a transform that runs before a display, a compression that runs before a store — these move the data’s production and consumption adjacent and remove a crossing that would otherwise exist regardless of where the computation ran.

A computation that consumes data from many sources and is expensive per byte is the case where a copy is justified. The copy is amortized because the computation holds the data long enough to do enough work with it, and this is the regime where pooling data in a device’s memory and running many kernels over it pays.

The version of this reasoning that governs local reuse is the operand-reuse analysis of Systolic Arrays and Dataflow Scheduling, where intensity is raised by the hardware’s own structure rather than by the placement of the boundary. The two are complementary: a structure that reuses operands well is exactly what makes the boundary worth crossing, because the crossing is amortized over the reuse.

Pipelining, and the cost of a stage that stalls

Once the boundary is drawn, the offload’s throughput is set by its slowest stage. A pipeline of transfer, compute, and transfer back can be overlapped so that the device computes on one batch while the host transfers the next, and the steady-state rate is then the maximum of the stage times rather than their sum. Double buffering is what makes this possible: two buffers, one being filled while the other is being computed on.

The gain from overlapping is large when the stages are comparable and small when one dominates. If the transfer takes ten times the compute, overlapping saves at most a tenth of the total and the offload is really a bandwidth exercise; if compute dominates, overlapping is what prevents the transfer from appearing as a serial gap. The stage times also determine the batch size that keeps the pipeline full: a batch must be large enough that its compute time hides the transfer latency of the adjacent stage.

The failure mode of pipelining is the stage that occasionally stalls. A single slow transfer — one that misses on a resource, crosses a page boundary, or contends with another user of the interconnect — drains the pipeline, and the steady-state rate is determined by the tail rather than the mean. This is why offload designs care about worst-case latency even when they are optimized for throughput: the batch size must be large enough that an occasional stall in one stage is absorbed by the buffer in the next.

What the accounting buys

The value of doing this arithmetic is that it identifies the offloads worth building before any hardware exists. A computation with high intensity, data that is already adjacent to the device, and a batch large enough to amortize the crossing will benefit. A computation with low intensity, data that must be copied in and out, and a batch too small to hide the crossing will not, no matter how much faster its arithmetic.

The numbers in the accounting are approximate and that is acceptable, because the decisions they drive are not close. The difference between a kernel that runs at 0.5 operations per byte and one that runs at 50 is a difference in kind, not in degree, and it determines whether the offload’s performance is limited by arithmetic or by movement. Designs that get this wrong are not marginally slower; they are slower by the ratio of the two intensities, which is why movement deserves to be the first thing measured and the last thing optimized.