Issue Width and the Structural Ceiling on Instructions per Cycle

A superscalar core is defined by a number: the issue width, the count of operations it can begin in a single cycle. The number describes structural capacity, not an achieved rate. Between the two sit fetch bandwidth, decode, operand read ports, functional unit availability, and the dependence graph of the program being run. A four-wide core that retires three instructions per cycle on ordinary integer code is behaving normally, not badly; the fourth slot is capacity for the moments when four independent operations happen to be ready.

The interesting engineering question is which resource binds first. In most designs it is not the width number printed in the datasheet but a narrower stage feeding it, or the register file feeding the functional units. Finding that resource explains most of the gap between peak and sustained instructions per cycle.

Five widths that are rarely equal

Every superscalar pipeline contains several distinct width parameters, and the sustained rate cannot exceed the smallest of them.

Fetch width is the number of instruction bytes the front end can pull from the instruction cache per cycle. It is usually expressed in bytes or in cache-line fractions, and a 64-byte cache line is the common unit of transfer. Four instructions of four bytes each occupy 16 bytes, comfortably inside a line, but instructions are not packed into fetch blocks on demand: a taken branch consumes the remainder of the block. General-purpose code contains roughly one branch per six to ten instructions, so a fetch block of four instructions frequently contains a branch that redirects the next fetch, and the slots behind it are discarded.

Decode width is the number of instructions the decoders can translate per cycle. Fixed-length instruction sets make this a matter of instantiating N identical decoders. Variable-length sets make it harder, because instruction boundaries are not known until the length is decoded; implementations typically use a predecode step at fill time, or a set of length decoders plus a rotation network to align the following instructions. When one instruction in a group turns out to be long or complex, the group’s decode slots go unused.

Rename width is the number of instructions that can be assigned physical registers and a scheduler entry per cycle. It is often smaller than decode width, because rename requires a free-list read, a map table read and write, and allocation of a reorder-buffer entry for each instruction in the group. Rename is where the machine commits to tracking an instruction to completion, so it is a natural place to economise.

Dispatch width is the number of instructions that can be sent to functional units per cycle, and it is bounded by the number of dispatch ports, discussed below. Retirement width is the number of instructions that can leave the reorder buffer per cycle; retirement is in order, so it must be at least as wide as dispatch or the buffer fills.

The sustained rate is the minimum of these, and the minimum is often two or three even when the headline width is four or six.

Dispatch ports and operand bandwidth

A dispatch port is a physical connection from the scheduler to one functional unit or group of units. Two integer ALUs, one load/store unit, and one floating-point unit give four ports, and the mix of ports determines what a group of four instructions can contain. Four integer additions dispatch in one cycle; three integer additions and a load also dispatch, but four loads cannot, because there is one address generation path.

The subtler limit is operand bandwidth. A two-source operation reads two registers and writes one. Dispatching W such operations per cycle requires 2W read ports and W write ports on the register file. Four-wide dispatch with two-source operations therefore needs eight read ports and four write ports. A register file bit cell with N read ports needs separate access transistors and separate bitlines for each port, and the wordline driver must drive all of them; area grows roughly with the number of ports, and access latency grows with bitline capacitance and decoder fan-out. Since the register read sits inside a pipeline stage whose length sets the clock period, adding ports can slow the whole core.

Two standard responses exist. One is banking: the register file is split into banks selected by low bits of the register number, so two operands that fall in different banks read in the same cycle. A conflict occurs when both operands land in one bank, which costs a dispatch slot. The other is replication: several copies of the file, each wired to a subset of the functional units, with writes broadcast or copied into every replica. Register File Ports and Operand Delivery works through the arithmetic of both.

There is a third path, used by machines that schedule dynamically: read operands late, into a small collector structure near the functional units, so the wide, expensive read happens once rather than at dispatch. This moves the port pressure but does not remove it, since the collector still needs enough bandwidth to feed the units.

Structural hazards that survive perfect prediction

Structural hazards are conflicts over hardware, and they are distinct from data hazards, which come from dependences in the program. A machine with perfect branch prediction and a program with unlimited parallelism would still stall on structure.

The instruction cache is the first. A miss in the fetch stage empties the pipeline for the miss latency, and a four-wide machine loses four instructions per cycle of the stall. Because fetch blocks are consumed in order, a miss at the start of a hot loop hurts more than one in a straight-line region.

The load/store unit is the second. Outstanding misses are tracked in miss status holding registers, and the count is finite. A wide machine issues loads four times as fast as a scalar one, so it needs four times the memory-level parallelism to keep the same fraction of its slots busy. When the queue is full, dispatch of further loads stops, and because dispatch is in order within a group, unrelated arithmetic behind the load also stops.

Functional unit contention is the third. A four-wide core with one integer multiplier dispatches at most one multiply per cycle; a loop with a multiply in every iteration runs at one instruction per cycle regardless of width. Similarly, a single floating-point divider with a multi-cycle initiation interval serialises dependent and independent divides alike.

Bank and set conflicts are the fourth. Two loads whose addresses map to the same cache set can conflict, and two register operands in the same register-file bank cannot be read together. These are probabilistic, so they show up as a fraction of cycles lost rather than a clean stall.

Why the return on width diminishes

Suppose a program’s critical chain is a sequence of dependent operations, each with latency L cycles, and the machine can start W operations per cycle. The chain alone permits at most one operation per L cycles. If the surrounding code offers k independent operations per chain element, the achievable rate is bounded by min(W, k/L). Widening the machine past k/L changes nothing.

The bound is easiest to see in a loop with a loop-carried dependence:

/* each iteration depends on the previous store: rate limited to one per latency */
for (int i = 1; i < n; i++)
    a[i] = a[i - 1] * c + d;

/* the same recurrence, split into four disjoint slices:
   four independent chains instead of one */
int q = n / 4;
for (int i = 1; i < q; i++) {
    a[i]         = a[i - 1]         * c + d;
    a[q + i]     = a[q + i - 1]     * c + d;
    a[2 * q + i] = a[2 * q + i - 1] * c + d;
    a[3 * q + i] = a[3 * q + i - 1] * c + d;
}

The second form is not faster because the machine is wider. It is faster because the source was rewritten so the dependence graph has four independent chains, which is a change to the program, not to the hardware. Compilers do this transformation within limits, and those limits are set by alias analysis and by the need to preserve the order of memory operations.

Because ordinary integer code exposes only a few independent operations per cycle in a reasonable window, designers stopped pushing width alone. Width costs area superlinearly: the bypass network that forwards results from each functional unit to each operand input of every other unit grows with the square of the width, and the wakeup logic that matches result tags against waiting instructions grows the same way. Wider also tends to mean slower, since more ports and more wires in a stage lengthen the clock period, and a wider machine at a lower frequency can lose on real code.

Two other routes to parallelism are therefore central. One moves the scheduling problem into the compiler, so the hardware can spend its area on functional units rather than on dependence checking; that is the subject of VLIW Bundles and the Compiler as Scheduler. The other obtains parallelism from data rather than from the instruction stream, applying one instruction to many elements; Fixed-Width SIMD Lanes Versus Scalable Vector Registers takes up that split. A third, orthogonal route fills the idle slots with work from other threads, covered in Simultaneous Multithreading and the Slots It Fills.

What the width number is good for

Issue width still matters, but as a statement about the machine’s structural envelope rather than about expected performance. It sets the maximum instruction fetch and retirement rate, the number of functional units that can be kept busy, and the size of the structures needed to track instructions in flight: a W-wide machine with a pipeline depth of D needs a reorder buffer and scheduler sized for roughly W times D entries to cover a memory latency, and those structures cost area and energy whether or not the slots are used.

The practical consequence is that width is chosen together with the workload. A core intended for server code with many independent memory operations can use a wide, deeply buffered design profitably. A core intended for pointer-chasing integer code gains less, and pays the same area and power. This is why the same instruction set appears in implementations with very different widths, and why the width difference between them predicts performance much less reliably than the sizes of the caches and the depth of the out-of-order window.