Capability Negotiation Across Architectures

An application that targets more than one accelerator must decide, before it runs any kernel, which device to use, which compiled variant of each kernel to load, and which code paths are unavailable. This decision is made on a machine the developer does not control, against a device whose property set is only discoverable by asking. Getting it wrong produces the two characteristic failures: a program that crashes on launch, and a program that runs correctly at a fraction of the available performance because it selected the conservative path on capable hardware.

The device is a bag of queried properties

Every API in this family exposes an enumeration of device properties, and they fall into four kinds that need to be treated differently.

Capability flags are booleans or bitmasks that gate code paths: whether 64-bit floating point is supported natively, whether the device supports atomics in global and local memory respectively, whether it can map host memory, whether it supports peer access, which extension strings are present. These are hard gates: code that uses a feature not present fails, so the gate must be respected.

Limits are numeric maxima: maximum work-group size, maximum constant-buffer size, size of local memory, number of registers per thread, maximum number of resident groups per multiprocessor. These are constraints on the launch configuration and on kernel resources, and are the values most often hardcoded by mistake.

Performance characteristics describe what the machine does well: the number of multiprocessors, the clock rate, the width of the memory bus, the size of the caches. These do not gate correctness; they select between candidate configurations that produce identical results.

Identity is the vendor, the device name, the driver version, and the architecture revision. Identity is not a capability. It is the key used to look up prior knowledge, and treating it as a replacement for querying is the main source of per-device special cases accumulating in a dispatch table.

The distinction matters because the first two can be read directly from the enumerations and the third cannot. The multiprocessor count times the number of groups each can hold gives a plausible number of blocks to launch, but how many groups fit depends on the kernel’s register and local-memory usage, which no device property query reports. The launch configuration itself is discussed in Launch Geometry and the Arithmetic of Thread Indexing.

Selection happens at three different times

A working dispatch layer makes three decisions at three points.

At configuration time, which backend libraries to build and which kernel variants to compile. This is a build system decision, and it determines what exists to select from at run time.

At initialisation, which device to use and which variants of each kernel are viable for it. This is where the property enumeration is consumed, and where the choice is recorded so it is made once rather than per launch.

At launch, nothing. If a selection decision is being made at launch time, it is being made once per kernel invocation, and the branch and the property lookup cost is paid on the critical path.

The three-time split is the design constraint behind the cache discussed below, and it is also why the interface of a good dispatch layer is a function that returns a fully bound launch configuration rather than a set of predicates the caller evaluates.

Keying a kernel cache

Runtime compilation, described in Driver-Level Runtime Compilation and Module Loading, is only usable if the compiled artefact is cached. The cache key must include everything that changes the compilation result:

struct cache_key {
    uint32_t driver_version;   // driver's compiler differs between releases
    uint32_t device_arch;      // major*100 + minor, from a property query
    uint32_t device_index;     // per-device binaries are not interchangeable
    uint32_t build_options;    // a hash of the -D and optimisation flags applied
    uint32_t source_hash;      // a content hash of the kernel source
};

Two entries of this structure that differ in any field may legitimately produce different binaries, and a cache that omits one of them will eventually load a binary built under different conditions. The two commonly omitted fields are the driver version, because it is the driver’s compiler that consumes PTX or SPIR-V, and the build options, which is where a -D that changed a tile size silently invalidates the binary.

A precompiled image shipped for one architecture fits the same scheme. Its presence in the container is part of what the loader selects on, and recording which architecture it was built for keeps it distinguishable from an artefact produced on the target machine.

Where feature detection diverges from the specification

Queries return what the implementation reports, which is not always what the specification promises, and the gap has three usual shapes.

The first is a reported maximum that cannot be achieved by any realistic kernel. A maximum work-group size of 1024 counts threads, but a kernel that uses a large amount of local memory cannot actually be launched with 1024 threads, because the local allocation per group exceeds the per-group budget. The effective limit is the minimum of the reported limit and whatever the kernel’s resource usage permits. APIs that report an occupancy figure per kernel, given a candidate block size, exist precisely because the two limits are separate.

The second is an extension string present but the entry point absent, or present through a loader that resolves it to a null pointer. Extension availability should be confirmed by resolving the entry point and checking it, not by scanning the string.

The third is a capability reported inconsistently across related queries: a device that reports support for a precision in one query and fails to execute the corresponding instruction. This is rare and cannot be defended against in general, which is why conformance testing on the target hardware, rather than trusting the first enumeration, is part of shipping a dispatch layer.

The conservative default and its cost

Every dispatch layer has a fallback path: the kernel variant with the fewest assumptions, which works on the oldest supported device. The fallback is what makes the library correct, and it is also what makes it slow if the gates are too coarse.

A gate that requires one flag for a compound feature forces a device that supports most of the feature onto the fallback. The standard error is a single predicate covering the whole fast path, with everything else routed to the generic version; the correct structure gates each variant on the specific operations it needs, so a device with native double-precision but no fast shared-memory atomics still gets the variant that uses the former and the fallback only for the latter.

The reverse error is a gate that admits a device that cannot run the variant because the check was copied from a similar device. This is where an identity-based table diverges from capability-based logic: a table entry for a known-good device is fine as a performance hint, but never as a correctness gate.

Reporting what was selected

A dispatch layer whose decisions are invisible cannot be diagnosed. The minimum instrumentation is the name of the backend, the selected device, the kernel variant, and the fallback reason, emitted once at initialisation. Testing a configuration on hardware the developer does not possess reduces to reading those four fields from a log, and their absence turns every portability report into a guessing exercise.

The information should also be available through a query, so a packaging decision, such as whether to ship the SPIR-V path at all or whether the PTX fallback is ever exercised, can be answered from a deployment rather than from a support thread. Software that reports its selected path can be deployed onto new hardware and adjusted; software that cannot has to be rebuilt.