Heterogeneous Cores and the Area-Performance Frontier

A processor built from identical cores is easy to schedule and easy to verify. It is also, for most workloads, a poor use of the area and power budget. Single-thread performance grows slowly with the area invested in one core, while throughput grows nearly linearly with the number of cores, so a uniform design must choose one of the two objectives and give up the other. Building two kinds of core and assigning work to whichever fits is a way to keep both, at the cost of a scheduler that has to understand the difference.

Two curves that a uniform design cannot satisfy

The first curve relates the performance of one core to its area. As a rule of thumb, going from a simple in-order core to a wide out-of-order core with a large private cache multiplies the area by several and the single-thread performance by a factor far smaller than that multiple. The frequently quoted form of this observation is that performance from microarchitectural sophistication grows roughly with the square root of the increase in complexity, so tripling the transistor count of a core buys something closer to 1.7 times the performance than 3 times. The exact exponent varies by design and by workload, and the useful part is the direction: single-thread performance is concave in area.

The second curve relates aggregate throughput to area. If $n$ small cores each deliver performance $p$ on a parallel workload, total throughput is close to $n p$ until a shared resource saturates, so throughput is nearly convex, in fact nearly linear, in area. The shared resource is usually the memory hierarchy, the interconnect, or the power budget, and the point where it bites sets the useful core count.

The two curves cross. For a workload with no parallelism, one large core is best. For a workload with abundant parallelism, many small cores are best. Real workloads are neither, and the mix changes during execution: a compile has serial phases in the front end and parallel phases in optimisation; a web server has parallel request handling and serial shared-state updates.

A uniform machine must pick a point on the trade for all of it. A heterogeneous machine keeps both ends of the curve and lets the scheduler move work between them, by Power Density and the Dark Silicon Constraint reasoning: at any instant only some of the cores are active, so the die can hold more cores than the power budget allows to run flat out, and the small cores are the ones that fit in the remaining budget.

The area-efficiency crossover is quantitative

The choice of how many large and how many small cores to build follows from the two curves plus a model of the workload.

Let a large core have area $A_L$ and single-thread performance 1, and let a small core have area $A_S$ and performance $r < 1$ on the same single thread. Throughput-oriented work of $n$ threads runs $n r$ times as fast on $n$ small cores as one thread would on one small core, provided memory allows.

Given a fixed area budget $A$, a design can hold either $A / A_L$ large cores or $A / A_S$ small ones. If $A_L / A_S = 4$ and $r = 0.5$, then the same area buys one large core or four small ones, and on perfectly parallel work the small cores deliver twice the throughput while on perfectly serial work the large core delivers twice the throughput. The optimum mix for a workload that is 90 percent parallel is heavily skewed toward small cores, and for a workload that is 20 percent parallel it is skewed the other way.

# Amdahl applied to area: the same area holds one large core or k small
# ones. Small cores are r times as fast per thread, and the serial
# fraction cannot spread over them.
def speedup_small(parallel_fraction, k, r):
    serial = (1.0 - parallel_fraction) / r          # one small core, at rate r
    par = parallel_fraction / (k * r)               # spreads over k cores
    return 1.0 / (serial + par)

for f in (0.2, 0.5, 0.9, 0.99):
    print(f, round(speedup_small(f, 4, 0.5), 2))

At $f = 0.2$ the four small cores reach about 0.59 of the large core’s speed; at $f = 0.9$ they reach about 1.54 times it. The crossover sits where the parallel gain from four cores exactly cancels the 2× per-thread deficit, which for these numbers is $f = 2/3$.

The model above is deliberately crude, and two refinements matter. Small cores often run at a lower voltage for the same frequency, so their energy per instruction is better than the performance ratio would suggest, which pushes the optimum further toward small cores. And the serial work often cannot move to the small core at full speed, so the assumption that serial work always runs on the fast core is optimistic for any schedule that must place a whole thread.

Scheduling is where the design lives or dies

Hardware that provides two kinds of core without a scheduler that understands them provides the cost of heterogeneity and none of the benefit. The scheduler’s job is to decide, for each runnable thread, which core it should occupy, and to move it when the answer changes.

Three mechanisms are used. The first is a simple asymmetry-aware placement policy: estimate each thread’s demand from its recent behaviour and place it on a core of the appropriate class, keeping the large cores for threads that are actually compute bound. The estimate comes from hardware counters, typically the ratio of stalled cycles to issued instructions, which distinguishes memory-bound threads from arithmetic-bound ones.

The second is periodic rebalancing. Every so often the scheduler compares the load on the two classes and moves work if one class is saturated while the other is not. This handles workloads whose phase changes slowly, such as a service that becomes parallel under load and serial when quiet.

The third is migration on demand. A thread that has been descheduled, or that has just woken from a long block, is a candidate for placement on a different class, because its warm state has already been lost. Migration is only worthwhile when the thread’s cache footprint is small or reusable; for a thread with a large working set, moving it discards the cache contents and the new core must refill them from memory, which costs more than the placement gain.

The migration cost is the crux, and it is the same trade as the one in The Area Budget: the resource being spent is not silicon but the time and bandwidth of refilling a working set. A thread with a 32 MiB working set moved between cores of different classes may refill most of it, and if the thread runs for less time than the refill takes, the move is a loss.

What heterogeneity does to the microarchitecture

Building two cores is not the same as building two sizes of the same core, and the design tends to diverge.

The large core is optimised for latency: deep speculation, aggressive prefetching, a wide window, and a large private cache. Its energy per instruction is high because it does work speculatively that sometimes gets discarded, and because its wide structures switch whether or not the extra width is used, the effect described in Issue Width and Its Diminishing Returns.

The small core is optimised for energy per instruction: in-order or lightly speculative execution, a narrow window, a smaller private cache, and a lower supply voltage at the same frequency. Its performance on serial code is poor, and its performance on parallel throughput code can be surprisingly good, because throughput work tends to have few unpredictable branches and a lot of independent arithmetic, which is exactly the pattern that in-order execution handles well.

The two cores share an instruction set, which is what makes migration possible at all, but they need not share a microarchitecture, and they generally do not. What they must share is the memory model and the interface to the shared cache, or migration breaks programs rather than merely slowing them.

A common refinement is a hierarchy of more than two classes, or a single physical core that can be configured as wide at low frequency or narrow at high frequency. The latter reduces the area overhead of having two sets of structures but gives a coarser control, because the configuration change costs a pipeline drain and a frequency transition, which is a Voltage and Frequency transition with the added cost of flushing speculative state.

Where the arrangement fails

The failure modes are all versions of the same mismatch.

A workload that is entirely serial cannot use the small cores at all, so the area spent on them is pure overhead while the large cores are the only productive ones. A workload that is entirely parallel and homogeneous leaves the large cores idle. Mixed workloads are where the arrangement pays, and mixed workloads are the common case for user-facing systems but not for every system.

Thermal behaviour complicates the accounting. The large cores have a high power density and their sustained operating point is limited by the package, so the number that can be active at once is smaller than the number on the die; the small cores have lower density and can run in larger numbers. The effective mix at any instant therefore depends on the thermal state, not on the area ratio, and the scheduler’s model has to include it or it will place work where the power limit cannot sustain it.

Finally, verification cost grows with the number of distinct core designs, and the software that must schedule across them has to be tested against both. That cost does not appear in a perf-per-watt curve, but it is real and it is the reason homogeneous designs remain attractive for markets where the workload mix is too variable to forecast.