Per-Lane Program Counters and Independent Thread Scheduling

For most of the SIMT model’s history, a warp had one program counter and divergence was resolved by strict serialization: run one side, then the other, then rejoin. The mask stack described in divergence and the reconvergence stack is the mechanism, and it makes divergent cost predictable at the price of a hard constraint: lanes that diverge cannot make progress at different rates, only in different phases.

Architectures that give each lane its own program counter remove that constraint and, in doing so, change the cost model and the correctness rules at the same time. Divergence becomes something the scheduler can interleave rather than serialize. Implicit reconvergence stops being a guarantee. Code that relied on lanes being at the same instruction because they were in the same warp can become incorrect without any change to the source.

What the hardware adds

The change is a per-lane program counter and a per-lane call stack, plus the bookkeeping to keep them coherent. A warp is still the scheduling unit: it is still the entity that occupies a resident slot, is selected by a scheduler, and issues one instruction per cycle. What changed is the set of lanes the issued instruction applies to and the point in the program from which it is taken.

Each lane has its own program counter, and the hardware tracks the set of distinct addresses the lanes occupy. When the warp issues, it selects one address from that set and the subset of lanes currently at it. This is the key structural difference from the mask stack: the older design had one address at a time and a stack of masks describing lanes waiting at other addresses, whereas the newer design has several addresses live at once and can choose among them each time the warp is scheduled.

The practical effect is that lanes on different paths can be advanced in an interleaved fashion rather than in strict sequence. If half the warp is in a long loop and half has exited it, the exited half can proceed past the loop while the other half continues iterating. The two groups are simply at different program counters, and the scheduler picks whichever group’s next instruction is ready. Under strict serialization, the exited lanes would wait at the reconvergence point until the loop finished.

The gain is bounded by the warp’s issue bandwidth, since the warp still issues one instruction per cycle and a warp split across two addresses now needs twice as many issues to advance both groups by the same amount. The benefit is not extra issue capacity; it is that the warp’s progress is no longer gated on its slowest lane before the join. Latency hiding improves because the fast lanes can reach and issue their subsequent loads earlier, rather than idling behind the slow group.

The stack becomes per-lane, and calls with it

With a per-lane call stack, function calls from divergent lanes no longer need to be serialized into a single warp-wide call sequence. Each lane can call and return independently, so a lane that takes a rare path through a function does not force the rest of the warp to wait for it to return before proceeding. The caller lanes continue while the callee lanes execute their call frames.

This matters for code with data-dependent recursion or with a conditional call to a helper that is expensive. Under the single-stack model, the call is a divergence point whose reconvergence is the return, and every lane pays the call’s full cost. Under per-lane stacks, the cost is paid by the lanes that actually call, in proportion to their own progress.

The bookkeeping cost is real. A per-lane stack needs storage proportional to the number of lanes times the maximum depth, and the hardware must arbitrate between lanes whose stacks are at different depths. That storage is why the feature did not appear earlier and why it is not universal across architectures.

Reconvergence is no longer automatic

The correctness consequence is the more important one. In the older model, lanes of a warp were guaranteed to be at the same instruction after a structured branch, because the stack popped at the join point. Warp-level operations (the vote functions that return a mask, the shuffles that exchange register values between lanes) were therefore safe to place anywhere after a branch, on the assumption that all lanes participating would arrive together.

With independent scheduling, that assumption fails. Lanes may be at different instructions, and a shuffle whose participating lanes are at different points cannot produce the value the source expects. The architecture addresses this by making the compiler insert an explicit reconvergence point before operations that require lanes to be together, and by making warp-level primitives take a mask of participating lanes so the hardware knows which lanes must be gathered.

The mask argument is not decoration. A primitive that names its participating lanes can gather those lanes even when the rest of the warp is elsewhere, and the semantics are defined in terms of the named lanes. Code that passes a mask computed from the current active set gets the expected result; code that assumes an implicit full-warp participation gets whatever the scheduler’s interleaving produced.

This is the sense in which the model changed rather than merely improved. The programmer’s mental model of a warp as a group of thirty-two threads in lockstep was always a simplification, but it was a simplification the hardware enforced at join points. Now the enforcement is explicit and, where it is absent, the simplification is simply wrong.

Assumptions that no longer hold

Several idioms were correct under strict serialization and are not correct in general under independent scheduling. They share a structure: one lane writes a value that another lane reads, with no synchronization, relying on the two lanes being at known relative positions in the program.

A producer lane writes to shared memory and a consumer lane in the same warp reads it, with a warp-level barrier or an implicit assumption of lockstep providing the ordering. Without a synchronization primitive, the consumer may execute its read before the producer’s write is visible, because the lanes are no longer in step. The fix is to use the block-level barrier, or a warp-level barrier where the architecture provides one with the required semantics, and to reason about visibility rather than about instruction order.

A lane that spins on a flag set by another lane of the same warp is the sharper case. Under strict lockstep the setter cannot run while the spinner occupies the instruction stream, so the wait never ends. Independent scheduling removes the strict interlock, but it does not by itself promise that a spinning lane will make progress: unless the architecture documents a forward-progress guarantee, termination depends on the scheduler’s choices. A bounded spin with a fallback, or a documented yielding primitive, is the form that does not depend on that choice.

The general rule is that any communication between lanes needs a synchronization operation whose semantics are documented, and that warp membership is not a substitute for one. Warp-level primitives that do carry the guarantee are the exception, and they say so.

What it does to the cost model

The efficiency arithmetic in active-mask accounting assumes each divergent region is issued once with a partial mask, and the sum of the mask widths over the issued instructions gives the lane-work. That remains the correct accounting, but the sequence of issued instructions is no longer determined by the source’s control-flow structure alone.

Two consequences follow. First, the serialization penalty in time shrinks: a divergent branch whose paths are unbalanced no longer holds the short path’s lanes idle until the long path drains. The instruction count does not fall, because both paths still issue in full, but the latency of the region is no longer the sum of the two phases. Second, the reconvergence overhead can grow: explicit reconvergence points inserted before warp-level primitives are instructions that must be executed and, depending on the implementation, may force lanes to be gathered, which reintroduces waiting exactly where the source requires it.

The net effect on a given kernel depends on the kernel rather than following from structure alone. Kernels whose divergence is unbalanced and whose paths contain independent memory accesses benefit; kernels that are dense with warp-level primitives see the benefit largely cancelled by the gather points. Kernels with neither see little change, because their divergence was already cheap.

Where the model still holds

None of this dissolves the warp. Thirty-two lanes still share a scheduler slot, still issue one instruction per cycle as a group, still occupy registers and a resident slot as a unit. The compiler still vectorizes by mapping threads onto lanes, still coalesces memory transactions per warp, and still pays for divergence in issue slots.

What changed is the granularity of progress within the warp. The execution model is no longer “thirty-two threads that must be at the same instruction” but “thirty-two threads that share an issue port”. That is a weaker and more accurate description, and it makes the warp’s remaining constraints the ones that matter: the shared issue slot, the shared register reservation, and the shared transaction formation. Warp specialization, described in producer and consumer warps, is the design response that takes those constraints seriously: instead of trying to keep lanes converged, it assigns different warps different code so that each warp’s mask is naturally full.