Device Memory Oversubscription and Page Migration

Device memory is the resource that most often limits how many tenants a GPU can host. A card with 80 GiB can hold four 20 GiB tenants, or ten 8 GiB tenants, or one tenant that wants 64 GiB. Without a way to use host memory as backing store, the number of tenants is exactly the quotient, and the operator either over-provisions or refuses work. With a way to move pages between device and host memory on demand, the quotient becomes a policy rather than a hard limit.

The mechanism is the same idea as CPU paging, transplanted into a device whose page faults are raised by hardware that was not designed to wait. That transplant is where all the difficulty lies: a CPU fault is a trap into an operating system that can block the faulting thread, while a GPU fault arrives from a thread among tens of thousands, and the device has no mechanism to park one thread while the rest continue.

What has to be true for a page to move

Moving a page from device memory to host memory requires three things to happen in order, and each has a failure mode if it happens out of order.

First, the device’s own translation must be invalidated for the page. A GPU MMU that still holds a translation for the page will let a thread write to it after it has been migrated, corrupting whatever the page now holds. Invalidation must therefore be acknowledged by the device, not merely issued, and the acknowledgement must cover any device-side caches of translations as well as the tables themselves.

Second, any in-flight writes to the page must be completed and made visible. A GPU’s writes are posted, and there is no ordering guarantee between a write issued by one thread and a subsequent copy of the page performed by a different engine. The migration path must therefore establish a barrier, which in practice means either waiting for the writing contexts to quiesce or using a fence that the hardware can acknowledge.

Third, the copy itself must be performed, and the destination must be a page the host has pinned and that the device can no longer reach.

The reverse direction has the same three steps with an additional requirement: before the page can be used by the device, the device’s translation must be installed and the page’s contents must be visible to the device, which on a platform without hardware coherence requires an explicit cache maintenance operation on the host side. On a coherent platform the operation is unnecessary, which is one of the reasons coherent interconnects change the design of these systems.

Faulting versus prefetching

There are two ways to decide which page to move. The demand-paged way is to let the device fault on an unmapped page and move exactly that page. The prefetching way is to move pages in bulk based on a prediction of what the tenant will touch next.

Demand paging gives the minimum traffic but pays a fault for every page. A fault on a GPU is expensive in a way a CPU fault is not: the faulting access is typically a warp-wide access, so one fault may represent 32 threads’ worth of demand for the same page, and the fault must be delivered to the host, resolved, and the page installed before the warp can be rescheduled. Devices that can only handle a small number of concurrent faults will serialise, and a kernel with a scattered access pattern generates exactly the pattern that defeats them.

Prefetching amortises the fault cost by moving pages before they are needed, at the risk of moving pages that are not. The prediction is usually based on the access pattern of the previous kernel or on an explicit hint from the runtime. The hint interface is what makes this practical: a runtime that knows a buffer will be read sequentially can declare it as such, and the migration engine can then transfer it in large blocks rather than page by page.

The middle ground is to move in larger granularity than a page. If the device’s page size is 64 KiB or 2 MiB, a fault brings in the whole region, which reduces the fault count by the ratio of the sizes at the cost of over-fetching. The right granularity depends on the spatial locality of the tenant’s access pattern, which is exactly what the hardware cannot know.

Eviction policy under multi-tenancy

Single-tenant oversubscription has a simple objective: keep the working set resident. Multi-tenant oversubscription has a harder one: decide which tenant’s pages to evict, knowing that the evicted tenant will fault them back in and that the faulting will consume the bandwidth of the tenant that is currently running.

A least-recently-used policy at the page level treats all tenants identically and therefore allows a tenant with a large working set to evict a tenant with a small one. The small tenant then faults on nearly every access, and its throughput collapses while its neighbour is unaffected. This is the memory analogue of the cache-pollution problem in time-sliced sharing, and it has the same class of fix: partition the resource, or account for it explicitly.

Per-tenant quotas are the direct fix. Each tenant is granted a maximum resident set, and evictions are charged against that tenant’s quota rather than against the global pool. A tenant that exceeds its quota evicts its own pages, and its neighbours are unaffected. The cost is that the device may be under-utilised when some tenants are below quota and others are above, since the surplus cannot be lent without breaking the guarantee.

A middle policy allows lending with a reclaim priority: a tenant below quota may use free capacity, but the capacity is reclaimable, and the reclaim targets the borrower first. This is the same structure as a memory balloon in a hypervisor, and it has the same weakness, which is that the borrower’s performance depends on when the lender wants its memory back.

Migration and the cost of moving a guest

A guest that holds device memory can be migrated only if the device memory contents can be moved with it. For a passthrough device this is impossible, because the device’s state is not observable. For a mediated device the driver owns the state and can serialise it, which makes migration possible but not free: the serialised state includes every page the tenant has resident, and the transfer must happen while the tenant is stopped or while its dirty pages are tracked.

Dirty page tracking is the standard technique and it is harder on a device than on a CPU. A GPU writes through its own translation, so a page’s dirty bit must be maintained by the device’s MMU, and the tracking granularity is the device’s page size. A tenant that writes a single byte per page dirties the whole page, so the dirty set after a kernel can be the entire working set. Iterative pre-copy works only if the dirty set shrinks between passes, and a workload that rewrites its whole buffer every kernel never converges, leaving a final stop-and-copy of the full resident set.

The alternative is to accept that a device-holding guest cannot be migrated live and to restrict migration to guests whose device memory is small enough to transfer in a bounded stop. That is a policy decision rather than a technical limit.

What oversubscription cannot fix

Oversubscription changes the capacity limit; it does not change the bandwidth limit. A tenant whose working set does not fit in device memory will spend part of every kernel waiting for pages, and the fraction is bounded below by the ratio of the miss traffic to the total traffic. If a tenant sweeps a working set several times the size of the device memory it was granted, most of its accesses miss, and the resulting throughput is set by the interconnect between host and device rather than by the device’s own memory system.

That interconnect is the binding constraint. A link that carries a fraction of the device’s internal bandwidth turns an oversubscribed tenant into a host-bandwidth-bound tenant, and the ratio between the two is the factor by which oversubscription degrades performance. Designs that place host memory and device memory on a coherent, high-bandwidth fabric narrow that ratio, which is why coherent interconnects change what oversubscription can be used for.

The second thing oversubscription cannot fix is latency. A fault is a round trip through the host, and a tenant whose access pattern is fine-grained and unpredictable will fault constantly regardless of how much memory it is granted. Oversubscription is a capacity technique, and treating it as a latency technique produces configurations that are correct and unusably slow.