Driver-Level Runtime Compilation and Module Loading

Below the vendor’s C-level API there is a second API: the driver interface, which exposes contexts, modules, functions, and the compilation of intermediate code into loadable images. The runtime API is largely a wrapper over it, and the wrapper hides the stage that determines startup latency, binary portability, and a class of load-time failures that the runtime API reports with a generic error.

Two APIs, one implementation

The driver API and the runtime API are layered, not parallel. The driver API has an explicit context object; the runtime API creates one implicitly on first use, per host thread, and manages it. The runtime API’s symbol names resolve at link time against a stub that loads the driver at run time; the driver API loads the driver and resolves every entry point through function pointers, which is what allows a program to run on a machine with an older driver by checking the entry point’s presence before calling it.

The split has a practical consequence for versioning. A driver API call added in a later release is available only if the installed driver returns a version at or above it, and the version negotiation is explicit: cuDriverGetVersion reports the driver’s supported API version, and the application compares it to the constant it was compiled against. The runtime API abstracts this away by refusing to start, which is unambiguous but leaves the process unable to report which version was required.

Module loading is the part of the driver API that has no clean runtime equivalent. cuModuleLoad takes a file path containing a fat binary; cuModuleLoadData takes a pointer to the image already in memory; cuModuleLoadDataEx adds a list of options that reach the JIT compiler. From the loaded module, cuModuleGetFunction resolves a kernel by name.

The load walk

A module load resolves the container to executable code in a fixed order, and the order is the whole reason a fat binary is structured the way it is.

First, the container is inspected for a cubin the current device can execute: an exact architecture match, or an earlier minor revision of the same major architecture. A match is a direct copy into the device’s instruction memory, with no compilation.

Second, absent a match, the container’s PTX images are examined. The loader selects the highest PTX version whose ISA version the driver’s JIT compiler can assemble. Lower PTX versions remain usable, so an old image still loads; a newer image than the driver knows is rejected.

Third, the JIT compiles the selected PTX for the running device. The result is cached by the driver, keyed on the image content and the target architecture, in a location shared across processes on the same machine. A second process loading the same module therefore skips the compilation; only the first pays for it.

Fourth, the compiled or copied image is relocated against the module’s address space, its global and constant variables are allocated and initialised from the image, and the functions become resolvable by name.

CUmodule mod;
CUfunction fn;
CUresult r = cuModuleLoad(&mod, "kernels.cubin");
if (r != CUDA_SUCCESS) {
    const char *msg; cuGetErrorString(r, &msg);   // name the stage that failed
    fprintf(stderr, "module load: %s\n", msg);
    return 1;
}
// Resolve an entry point by name; the module keeps its code loaded.
r = cuModuleGetFunction(&fn, mod, "reduce_partial");

Each of the four stages fails in a distinct way, and the error code identifies the stage even when the message does not. A container with no image for the device fails at the first or second stage; an image whose ISA version exceeds what the driver knows fails at the second; a JIT failure fails at the third and reports through the module load call, with the compiler’s own diagnostic written to an error log buffer if one was supplied in the options list; a relocation failure at the fourth generally means a missing symbol or a device with insufficient resources for the module’s static allocations.

JIT options are part of the build

cuModuleLoadDataEx accepts a list of options that change what the JIT emits, and they parallel the offline compiler’s flags: the target architecture the code is generated for, the optimisation level, the maximum number of registers per thread, and the size and location of a buffer to receive the compiler’s diagnostic output. The options apply to the whole module, so the register cap chosen for one kernel constrains every kernel in it.

The register count option is the interesting one. Capping registers per thread forces the compiler to spill to local memory, which is a deliberate trade: fewer registers raise occupancy because more blocks fit on a multiprocessor, and the spills add memory traffic. Where the trade is worth making depends on the kernel, and the fact that it can be made at load time rather than at build time is what lets one PTX image serve several register-pressure regimes.

Setting options per module, rather than per function, is a coarse granularity: every kernel in the module is compiled with the same register cap. Splitting a module into several smaller ones by kernel is the workaround, at the cost of more load operations and more per-module global state.

Why the runtime API hides this, and when it hurts

The runtime API performs the same walk at the first kernel launch, using the fat binary embedded in the host executable. The first-launch latency therefore includes the JIT for a PTX-only build, and it appears at an arbitrary point in the program rather than during initialisation.

For a process that runs for hours the cost is irrelevant. For a short-lived process, whether a command-line tool, a serverless invocation, or a test that runs one kernel, the JIT dominates the wall time, and the driver’s persistent cache is the only mitigation. The cache is not guaranteed to be enabled or writable in every environment; a container without the cache directory present, or with it mounted read-only, compiles on every invocation.

Moving the load earlier is the standard remedy: perform an explicit module load, or an explicit runtime initialisation that triggers the same work, before the latency-sensitive phase. This converts an unpredictable first-launch stall into a known startup cost, which is a different problem and a much easier one.

Cache invalidation rules

The driver’s cache key is not documented as a stable contract, and the safe assumption is that it covers the image bytes and the target architecture. It does not necessarily cover a change to a JIT option passed with the load, so two loads of the same PTX with different register caps may share or not share an entry depending on the implementation.

An application-managed cache should make its own assumptions explicit. Key on the content hash of the image, the device architecture, the driver version, and the full option string, as sketched in Capability Negotiation Across Architectures. Store the result next to those fields so a mismatch is visible rather than inferred, and treat a load failure as a cache miss and rebuild rather than as a fatal error.

The artefact a cache stores is the driver’s output format, not necessarily a cubin. Saving it and reloading it through the module load path is supported where the API allows a binary image to be loaded directly, but the format is opaque: it is valid for the driver that produced it and for the device it was produced for, and a mismatch produces the same load-stage failure as an incompatible cubin. The trade between shipping a portable intermediate and caching a machine-specific artefact is the same one described in PTX, SASS, and What a Fat Binary Contains, applied one level down.