Simultaneous Multithreading and the Slots It Fills

A four-wide core running a single integer thread on ordinary code leaves a large share of its issue slots empty: the thread has a limited number of independent operations available, and when it stalls on a cache miss or a mispredicted branch, every slot goes empty at once. Simultaneous multithreading fills those slots with instructions from other threads. The threads share the same physical core, the same functional units, and, in the common design, the same reorder buffer and scheduler; only the architectural register state and the fetch/decode control are replicated per thread.

The result is not a second core. It is the same core with a second source of instructions, and the amount it gains depends on which structure was the bottleneck for a single thread. If a single thread already saturates the issue width, a second thread gains nothing and costs area. If a single thread uses half the slots because it cannot find enough independence, a second thread can supply work for many of the remainder.

Which structures are replicated

The architectural state per thread is the register files, the program counter, and the control registers. Everything else is shared or partitioned.

The instruction fetch unit is replicated in the sense that each thread needs its own program counter and its own next-fetch-address logic, and usually its own return-address stack and branch history, since a branch predictor trained by two interleaved threads predicts worse than one trained by either. The instruction cache is shared, which is a benefit: two threads running the same code share fetch bandwidth, and two threads running different code evict each other’s lines.

The reorder buffer is shared in the classic implementation, with entries tagged by thread. This is the crucial choice. Sharing means a thread that stalls on a long-latency load occupies reorder buffer entries but does not prevent the other thread from allocating new ones; it simply consumes capacity. Partitioning the buffer per thread wastes entries when one thread is idle, which is exactly the case the mechanism was designed to fix.

The scheduler and functional units are shared. Registration queue entries are held by instructions from either thread, and dispatch selects among all ready instructions regardless of origin. This is where the gain comes from: an instruction from thread B can be dispatched into a slot that thread A’s dependence graph leaves empty.

The load/store queue is shared with per-thread tags. Memory ordering between threads is handled by the memory consistency model, not by the queue, so the queue only needs to preserve each thread’s own program order. Store-to-load forwarding, however, must not forward a value across threads, and the queue’s comparison logic must check the thread tag on every forward.

The register file can be partitioned statically, with each thread owning a fixed alias range of physical registers, or shared dynamically, so a thread can use more physical registers than its architectural count when another thread is idle. Static partitioning is simpler and bounds the worst case; dynamic sharing recovers registers for a busy thread but complicates rename, since the free list is global. The port and banking constraints that bound how many of those registers can be read in a cycle are treated in Register File Ports and Operand Delivery.

Fetch policy as the real design problem

With N threads sharing a fetch unit that delivers four instructions per cycle, the policy question is how to allocate fetch slots. The obvious answer, allocate per thread in a fixed rotation, ignores the fact that some threads are stalled and some are in a long run of independent instructions.

The behaviour that shaped the design is that fetching from a thread that is about to stall wastes fetch bandwidth, and modern designs use feedback: a thread that recently had no instructions ready to issue gets its fetch priority reduced, and a thread that recently issued successfully gets more. This is a control loop over fetch allocation, not a static split, and its parameters are microarchitectural, not architectural.

The fetch policy also interacts with the instruction cache. Fetching from multiple threads at once means the fetch unit needs to track multiple streams, and the line fill policy must handle a miss for one stream without disturbing the other. Implementations commonly use a per-thread fetch pointer into the same cache, with the line boundary crossing handled by the next thread’s turn.

Why the gain is less than the thread count

Two threads on one core rarely double throughput. Three effects reduce the gain.

The first is resource competition. Both threads share the scheduler, the reorder buffer, the load/store queue, and the caches. A second thread can fill idle slots, but it also consumes entries the first thread would have used to expose more of its own parallelism. On a wide core with a large window, a single thread may already extract most of the available instruction-level parallelism, and the second thread’s instructions displace some of the first’s rather than filling truly idle slots.

The second is that the two threads contend for the same units. If each thread’s critical path runs through the integer multiplier and there is one multiplier, the second thread waits behind the first regardless of how many issue slots exist. The units that are plentiful, such as integer ALUs, are not the ones that limit a thread; the ones that limit are shared equally.

The third is cache. Two threads share an L1 data cache and an L2. If their working sets overlap, the total footprint is smaller than the sum and SMT is nearly free on the memory side. If the working sets are disjoint, the effective cache per thread halves, which raises miss rates and can offset the issue-slot gain. This is why SMT gains are largest on workloads with large shared footprints, such as many instances of the same server process, and smallest on workloads with disjoint streaming data.

A concrete way to see the bound: if a single thread achieves an IPC of 1.5 on a four-wide core because its average parallelism is 1.5, and a second identical thread is added, the pair completes at most 3 instructions per cycle, the sum of the two threads’ independent operations, which is still short of the width of four. The scheduler can dispatch only four instructions per cycle, and the combined dependence graph of the two threads offers no more than that sum, so throughput rises toward the width but cannot exceed it. The quantity that improved is utilisation, not peak.

Vector state and the context-switch problem

A thread on a machine with vector registers carries a large architectural state. On a core with sixteen 512-bit vector registers, that state is 1 KiB per thread; with eight threads sharing the core, the aggregate is 8 KiB that must be saved and restored across context switches on the same hardware that is trying to amortise every other structure.

Operating systems handle this with lazy state management: a flag in the thread control block marks whether the thread has executed a vector instruction, and the save happens on the first switch away from a thread that has. Threads that never touch vector code never pay. Threads that do pay a memory transaction of a kilobyte, which is not free but is small relative to a page fault and comparable to a cache miss on a cold line.

The sharing also changes what vector execution means. Vector units are per-core, so two threads on one core compete for the same vector pipeline. A thread using wide vectors occupies the vector unit for the whole instruction, and the other thread’s scalar instructions can still issue in the same cycle only if the front end and dispatch ports allow it. On cores where vector instructions occupy multiple dispatch slots or enforce serialisation with the vector unit, SMT gains on vector code are much smaller than on scalar code, because the vector unit is the contended resource and it is not being shared efficiently.

Where SMT does and does not pay

SMT pays when the workload is a mix of threads with low individual parallelism and the core’s width is already paid for. It pays most on server workloads with many similar threads and substantial shared memory footprints, where the second thread’s occupancy of the caches costs little.

It does not pay when the core is narrow, because there are few idle slots to fill. It does not pay when the workload is memory-bandwidth-bound, because the bottleneck is off-core and the extra threads only add queueing delay. It does not pay when security requires isolation, since sharing a reorder buffer, a load/store queue, and caches between threads creates timing channels that neither software nor the ISA can fully close; the response has been to disable SMT or to flush shared structures on a switch, both of which reduce the benefit.

The engineering conclusion is that SMT is a way to recover waste in an already wide machine, not a way to build parallelism. The parallelism has to exist in the program, either as independent threads or as data parallel elements. The latter route, applying one instruction to many elements, is treated in Fixed-Width SIMD Lanes Versus Scalable Vector Registers and is the one that raises the work done per slot rather than the number of slots filled.