Mask Registers, Tail Handling, and the Cost of the Last Iteration

A vectorised loop almost never has a trip count that is an exact multiple of the lane count. Of the two ways to handle the remainder, one writes a second scalar loop and the other disables the lanes that have no work. The first is simple and costs instruction footprint plus a second loop to fetch and branch through. The second uses a mask: a bit per lane, held in a dedicated register or derived from a comparison, that makes each masked operation a no-operation on the lanes where the bit is clear.

Masks are the more general mechanism, and the generality is why they cost something. A masked instruction cannot blindly write back its results; the write-back must select, per lane, between the new value and the old one. That selection is a multiplexer across the full width of the datapath, on every lane, and it is done whether the mask is all-ones or not.

How a mask reaches the datapath

A mask register holds one bit per lane. On a machine with 128-bit registers and 32-bit elements, that is four bits; on a 512-bit machine, sixteen. The register is written by a compare instruction or by an explicit load of a mask.

An operation under a mask reads the mask, reads the destination’s previous value, computes the new value, and selects per lane. The selection can be done at the end of the pipeline, just before the write-back, which keeps the arithmetic path unchanged but adds a multiplexer and a destination read port. In a single-cycle-per-lane pipelined unit, this means the destination vector register must be readable in the same cycle the result is written, which doubles the read pressure on that register for masked operations.

An alternative is to suppress side effects rather than results. For a load, an inactive lane must not access memory and must not fault; for a store, it must not write. The address generation for inactive lanes can proceed harmlessly provided the memory access is squashed before it reaches the cache, so the mask is applied at the point where the memory request is formed. Arithmetic has no side effects, so its only requirement is the write-back selection. The distinction matters because it means floating-point exceptions and memory faults are only raised for active lanes, which is a semantic requirement, not an optimisation.

Masked operations in a pipelined unit also change the stall behaviour for trapping instructions. Because the mask determines which lanes can trap, the pipeline cannot report a precise exception at the point of the instruction without knowing the mask, and a masked load that misses must remember which lanes were active so that the fill writes only those lanes on return. That state travels with the miss in the miss status holding registers.

Tail agnosticism and the value in unwritten elements

When the last iteration has fewer active elements than lanes, the inactive lanes receive no new value. What they hold is a policy decision, and the two sensible policies have different costs.

Under an undisturbed policy, inactive lanes retain their previous value. This lets software accumulate into a register across iterations and read the tail elements afterwards without masking, because the untouched lanes still hold data from earlier. It costs the write-back multiplexer on every operation, and it costs a dependency: the masked operation reads the destination, so it cannot be renamed or scheduled as freely as one that writes all lanes.

Under an agnostic policy, inactive lanes may hold any value. This frees the hardware from the dependency, which simplifies renaming and can reduce the power spent driving multiplexers, at the cost of making software unable to rely on those lanes. A compiler emitting agnostic-mode code must mask any later read that reaches the tail elements, which shifts the work back to software.

The choice is per-instruction in designs that expose it, because a loop that accumulates wants undisturbed lanes and a loop that computes independent results does not care. The distinction shows up in the port pressure discussed in Register File Ports and Operand Delivery: undisturbed masked operations need a destination read port that agnostic ones do not.

Vector length control and the strip-mined loop

A scalable vector ISA adds a second mechanism: the number of active elements can be set as a value in a control register, not only through a mask. The loop then strips by the granted length and the last iteration sets a shorter length, so the hardware processes fewer elements.

    vsetvli t0, a0, e32, m2, ta, ma   # request up to VLEN/16 elements,
                                      # 32-bit, LMUL 2, tail and mask agnostic
    vle32.v  v0, (a1)
    vadd.vv  v2, v0, v1
    vse32.v  v2, (a2)
    sub      a0, a0, t0
    ...

The advantage is that the last iteration does not need a mask at all in the common case: the length register is smaller and the hardware processes exactly that many elements. The inactive lanes still need a defined tail state, so the same multiplexer exists, but software never has to construct a mask, and the loop body stays branch-free.

The cost is that every iteration executes the length-setting instruction, and that instruction may serialise the pipeline in some implementations because it changes the operating parameters of subsequent vector instructions. Designs that make length setting cheap keep the parameter in a register that the front end reads as the instruction is issued; designs that let it stall issue pay a cycle or more per iteration on short vectors.

Where masks are unavoidable

The tail is only one use of masks. Three others make masking a first-class requirement.

Data-dependent control flow inside a loop cannot be branched around without vectorising the branch, so a conditional update becomes a compare that writes a mask followed by a masked operation. This is the vector analogue of the predication discussed in VLIW Bundles and the Compiler as Scheduler, with the difference that the predicate is a vector because the condition differs per element.

Irregular memory access is the second, and it is where masking interacts with the memory system: a gather whose addresses are valid for some lanes and not others must not fault on the invalid ones, so the gather is masked and the inactive lanes need not even compute an address. That case is treated in Strided, Indexed, and Gather Memory Access.

Compression is the third. When a mask marks a subset of lanes, a compress operation packs the active elements into a contiguous prefix, which converts a masked scatter into a dense store. Without it, storing a sparse subset means either a masked store to the original positions, which writes into whatever else is there, or a scatter. Compression costs a crossbar across lanes and is a meaningful area item in a mask-capable vector unit.

The cost account

Masking is not free, and the costs are specific.

The write-back multiplexer costs area proportional to the vector width and delay in the write-back stage, though it is off the arithmetic critical path in most implementations.

Maintaining the mask in a vector register costs one architectural register if masks live in the vector file, and a separate predicate file with its own read ports if they do not. A separate predicate file with 8 registers of 64 bits is small, but it needs a read port per operand per unit, which is the same port problem at reduced width.

The compare that generates a mask adds an instruction per condition, and the compare’s result is a mask rather than a vector, so it occupies a mask write port and cannot be chained into the arithmetic path the way a vector result is in Vector Chaining and What It Costs in the Register File. A predicated operation therefore waits for the entire mask rather than starting on the first active element, which is a real serialisation on top of the arithmetic chain.

Finally, masks consume issue slots on a superscalar front end. Each mask-generating compare is an instruction the front end must fetch and dispatch, and on a loop that is already close to the front end’s limit, the compares can be the difference between issuing two vector operations per cycle and one.

Deciding between the remainder loop and the mask

The remainder loop is cheaper when the trip count is known and large, so the remainder is a small fraction of the work, and when the loop body is short enough that a scalar version is simple. It costs code size and an extra branch, and on a machine with a small instruction cache, the second copy of the loop body can evict the first.

The mask is cheaper when the trip count is unknown, when the loop body is long enough that duplicating it is expensive, and when the loop already needs masks for control flow. In those cases the mask infrastructure is present anyway, and using it for the tail costs nothing extra.

A scalable vector ISA’s length control combines both: no duplicated body, no mask construction, and a tail handled by hardware. That is the strongest argument for including a length register in the ISA, and it is the one feature that fixed-width designs emulate with a mask and a computed bit pattern rather than with a dedicated mechanism.