Fixed-Slice Partitioning of Shared Engines Inside One Device

A card can be shared by time-slicing, in which case tenants take turns and each one’s state must be saved and restored at every switch, or by space-slicing, in which case each tenant owns a fixed set of resources for as long as it holds them. Space-slicing inside a single PCI function is the third design point in this space: it gives isolation as strong as SR-IOV without the cost of replicating the function, and it gives direct access as fast as passthrough without a host in the data path. What it cannot give is independent control of the function itself, because there is still only one function.

The mechanism is a partition of the on-chip resources that carry identity: the execution units, the L2 cache slices, the memory controllers and the device memory behind them, and the translation and fault handling that goes with each. Partitioning is done once, at configuration time, by a privileged operation on the physical function. After that the instances are independent for every purpose that matters to a tenant, but they are not independent devices.

Why the memory system decides the partition count

The natural unit of partitioning is the memory slice, not the compute slice, because the memory system is what cannot be subdivided after the fact. A GPU’s device memory is not one pool behind one controller; it is several pools, each behind its own controller, each with its own portion of the last-level cache, and each with its own path to the crossbar. A memory access from a streaming multiprocessor goes through an address interleave that spreads consecutive addresses across the controllers, so any single controller sees a fraction of every large access stream.

That interleave is the obstacle. If two tenants share a controller, they share its bandwidth, its row buffers, and its queues, and no amount of address-space separation prevents one tenant’s access pattern from destroying the other’s row locality. The only way to make the isolation real is to give each tenant whole controllers, which means the number of partitions is bounded by the number of controllers. A card with eight memory slices can be split at most eight ways, and the split must be in whole slices.

The cache follows. Each L2 slice is attached to a memory controller, so a partition that owns a controller owns the corresponding cache slice. This is why partitioning a GPU produces instances whose cache capacity is proportional to their memory capacity, rather than instances that share one large cache with a smaller effective capacity each. The proportionality matters: a workload tuned for a full card’s cache-to-bandwidth ratio behaves the same way on a proportional slice, which is what makes the partition usable for capacity planning.

The compute side is more flexible. Execution units can be assigned in groups, and the assignment need not be equal, so a partition can be heavy on compute and light on memory or the reverse. But the compute assignment is constrained by the memory assignment, because an instance whose execution units outnumber what its memory slice can feed will simply be memory-bound, and the operator has paid for execution units that cannot be used.

What an instance inherits and what it does not

An instance receives a set of execution units, a set of memory slices with their caches, its own translation context and fault path, and its own set of submission queues with their own doorbells. From inside, the instance behaves like a smaller GPU: kernels launch, memory is allocated within the slice, and faults are reported to the instance’s own driver. Crucially, a fault in one instance does not reset another, because the fault handling is per-instance and the memory of the other instance is not in the faulting instance’s address space at all.

What an instance does not receive is its own configuration space, its own reset domain at the PCI level, or its own set of copy engines and video engines unless the partitioning explicitly covers those. Copy engines are frequently shared or assigned only to some instances, which means an instance without one cannot perform device-to-device copies at full rate and must route them through compute. Video engines are similar: a decode block is a single unit, and giving it to one instance removes it from the others.

The absence of an independent PCI reset is the sharpest limitation. Resetting a function resets every instance on the card, so a tenant that wedges its instance cannot be recovered without disturbing its neighbours. Implementations mitigate this by making the instance-level reset path thorough enough that the PCI reset is rarely needed, but the fallback still exists and still affects everyone. A virtual function design does not have this problem, because each virtual function has its own function-level reset; that is one of the things the address-space cost of SR-IOV buys.

Addressing: one device, several address spaces

Because the instances share a physical function, they share the function’s BAR. The register window is partitioned by offset: each instance’s doorbells and control registers live in a disjoint range, and the hardware checks that a write from a given instance targets its own range. Device memory is partitioned by address: the physical device address space is divided into contiguous regions, one per instance, and the instance’s memory management unit refuses to translate an address outside its region.

That refusal is the isolation property and it must be enforced in hardware on every access, not by convention in the driver. The reason is the same as in the passthrough case: the tenant’s driver writes page tables, and a page table entry naming another instance’s memory is a valid-looking entry. Only a range check in the translation hardware, performed per instance, catches it. Where the check is a comparison against a base and limit register pair, the check is cheap; where it requires a separate table per instance, the hardware must have that table.

The host’s view is the union of the instances plus the physical function’s own state. That is what makes a partitioned card awkward for a hypervisor: the guest’s driver sees an instance, but the device the hypervisor manages is the whole card, and the operations the hypervisor wants (reset, power management, firmware update) apply to the whole card. Passing an instance to a guest therefore requires the guest’s driver to be aware that it is operating an instance, which in practice means a driver mode rather than the ordinary driver.

Configuration is a one-way door while instances exist

The configuration operation takes a set of requested instance profiles, each with a compute count and a memory count, checks that the requested slices are disjoint and that the total fits, and programs the hardware’s partition table. It is only permitted when no instance has work in flight and no instance is assigned. Changing the partition therefore requires tearing down every tenant on the card.

This makes partitioning a capacity-planning decision rather than an operational one. An operator who partitions a card into seven equal instances to serve seven small jobs cannot later give one job half the card, because the half-card shape was not among the configured profiles, and reconfiguring means evicting all seven. The mitigation is to choose profiles that cover the expected mix, and to keep some cards unpartitioned for workloads that need the whole device. The cost of a wrong choice is measured in idle slices: a tenant whose job needs less than its instance leaves the difference unused, and no other tenant can borrow it.

Where fixed slicing sits among the alternatives

Compared with time-slicing, fixed slicing removes the context switch entirely, which matters because a GPU context switch is expensive in proportion to the amount of state the hardware must save. Compared with SR-IOV, it removes the address-space cost of replicated BARs and the requirement that the hardware replicate the identity-carrying state; it also allows finer granularity than the eight-function limit that the Routing ID imposes, since instances are not PCI functions. Compared with the mediated device model, it removes the host from the data path entirely, at the cost of the dynamic allocation the mediated model provides.

The residual weakness is the shared fate of the physical function. Everything that is not partitioned, from the PCI reset to the firmware to the shared copy engines, is a channel through which one tenant’s behaviour can reach another. Fixed slicing is therefore a strong isolation boundary for memory and execution, and a weak one for the card as a whole. That distinction determines what it can be used for: it is adequate for separating mutually distrusting tenants’ data, and inadequate for letting one tenant perform privileged operations on the device.

The other residual weakness is granularity. Because the partition count is bounded by the number of memory slices, the smallest instance is a fixed fraction of the card, and there is no way to make it smaller without oversubscribing through a scheduler, which reintroduces the switching cost that slicing was meant to avoid.