Adapter Residency When One Replica Serves Many Fine-Tunes

A fine-tune per customer, per task, or per language multiplies the number of models a fleet must serve without multiplying the traffic each one receives. Hosting each as a full replica gives every tenant a dedicated copy of the base weights, which for a 7-billion-parameter model is 14 GB in 16-bit precision whether the tenant sends one request per minute or a thousand per second. Low-rank adaptation offers a way out: freeze the base model and store only a small correction per fine-tune.

The correction is a pair of matrices per adapted layer. For a weight of shape d by k, the adapter holds a d-by-r matrix and an r-by-k matrix with r much smaller than either dimension, typically in the range of 8 to 64. The adapter’s parameter count is r(d + k) instead of dk, which for d = k = 4,096 and r = 16 is 131,072 parameters rather than 16.7 million, a factor of 128 smaller. In 16-bit precision that is 256 KiB per adapted layer rather than 32 MiB. A model with a few dozen adapted projections therefore carries adapters in the tens of megabytes, and hundreds of fine-tunes fit in the memory that one full copy of the base model would occupy.

What the forward pass has to do differently

The adapter changes the computation in a way that constrains batching. For an input x, the layer computes the base projection Wx and adds the scaled low-rank term B(Ax), where A is r by k and B is d by r. The base projection is shared by every request in the batch and is the large matrix multiplication. The low-rank term is per-adapter and is two small matrix multiplications with a rank-r bottleneck in between.

Two implementation shapes exist, and they differ in where the cost lands.

The first applies the base projection to the whole batch, then applies each adapter only to the rows belonging to requests that use it, and adds the results. This keeps the large matmul dense and confines the adapter work to a gather-scatter over row indices. Its cost is proportional to the number of distinct adapters present times the batch size, but the constant is tiny because r is small. The added kernels are small, so they are latency-bound, and their launch overhead becomes visible when the batch contains many adapters with few rows each.

The second folds the adapter into the weights before the forward pass, producing an effective matrix W + BA per adapter. That restores a single dense matmul per request but requires a materialized d-by-k matrix per adapter, which defeats the memory saving: one folded adapter is as large as the base weights. Folding is viable for a single adapter serving a large batch and not for many adapters sharing one.

def adapted_linear(x, base_w, adapter, rows, scale):
    # base projection is dense over the whole batch
    out = x @ base_w.T
    if adapter is not None:
        sub = x[rows]
        # rank-r bottleneck: two small matmuls, not one large one
        low = (sub @ adapter.a.T) @ adapter.b.T
        out[rows] += scale * low
    return out

The scaling factor is a fixed property of how the adapter was trained, and applying it incorrectly produces output that is subtly wrong rather than obviously broken. It belongs with the adapter weights, not with the request.

Residency: which adapters stay loaded

An adapter is small, but not free, and the number that can be resident is bounded by memory. A replica serving a thousand fine-tunes cannot hold a thousand adapters in accelerator memory alongside the base weights and the key-value cache, so adapters must be paged in and out like any other working set.

The access pattern is what makes this tractable. Traffic to fine-tunes is usually skewed: a few adapters receive most requests and a long tail receives almost none. A cache that keeps the hot set resident and loads the tail on demand captures most of the benefit. The cost of a miss is a transfer of the adapter’s tens of megabytes over the host interconnect, which is small compared with loading a full model but large compared with a decode step, so a miss inside a generation is expensive: the adapter must remain resident for the whole sequence, not just for the first token.

That last point is the difference from prefix caching. A prefix cache is read-only and shareable, so eviction decisions are about future hit rate. An adapter is in use for the duration of every token of a sequence, so evicting one that a resident sequence still needs either stalls that sequence or forces its removal. Residency therefore has to be negotiated at admission, and a scheduler that admits a request for a cold adapter must first ensure the adapter can be brought in and kept.

There is a second, cheaper form of the same problem: adapter weights can be applied from host memory, streamed into the kernel as they are used. That removes residency constraints entirely at the cost of reading the adapter on every layer of every token, which for a rank-16 adapter across 32 layers is a few megabytes per token. Whether that is cheaper than holding the adapter resident depends on the adapter’s reuse, and the crossover is exactly the working-set question above.

Batch composition under many adapters

An iteration-level batching loop fills its position budget from whatever sequences are ready. With adapters, the composition of the batch determines the cost of the iteration: a batch of forty sequences drawn from forty different adapters issues forty small adapter kernels per layer, while a batch of forty sequences from one adapter issues one. The base matmul costs the same either way.

This creates a scheduling lever that does not exist in single-model serving: the scheduler can prefer to admit requests that share an adapter with sequences already running. The benefit is reduced kernel launches and better reuse of adapter weights that are already resident; the cost is latency for requests that would have to wait for a co-resident group to form.

Grouping by adapter is a form of batching by affinity, and it trades the same way as the prefix affinity discussed in Prefix-Cache-Aware Routing: the gain is real but bounded by how much of the per-iteration cost the shared resource represents, and the loss is queueing delay for whatever is held back. Because adapter kernels are small relative to the base matmul, the gain from grouping is modest at moderate adapter counts and becomes significant only when a batch would otherwise contain dozens of distinct adapters.

A middle path avoids the choice. The scheduler admits by load as usual, then sorts the rows of the batch by adapter before the forward pass, so that each adapter’s rows are contiguous. Contiguity reduces gather cost and improves memory access patterns without delaying any request. It does not reduce the number of kernels, but it makes each of them larger, which is often the more valuable of the two effects.

Routing consequences

Adapter identity changes the meaning of a replica. Two replicas with the same base model but different resident adapters are not interchangeable, and a router that treats them as such will send requests to replicas that must load an adapter before serving them. The load score must therefore include a term for whether the adapter is resident, and the choice between replicas becomes a comparison of load against expected adapter-load cost.

Replication of popular adapters mitigates this: an adapter that receives enough traffic to matter is placed on several replicas, so the router has alternatives. The placement decision is a small-scale version of the packing problem in Replica Packing and Topology Constraints, with adapters as the items and replicas as the bins, except that the items are movable and their value depends on traffic.

Cache complicates the picture further. The key-value cache is specific to the base model and, for adapters that modify attention projections, to the adapter as well. Sequences from different adapters cannot share cached keys and values when the adapter changes the projections that produced them, so a replica serving many adapters fragments its cache by adapter. That fragmentation reduces the effective batch size, and it is the reason high adapter counts and long contexts are difficult to serve together.

Finally, the fairness question reappears at a finer granularity. A tenant with a hot adapter gets cheap iterations; a tenant with a cold one pays a load on every admission. A scheduler that charges only for tokens produced hides that asymmetry and lets cold adapters consume disproportionate capacity. Charging admission cost to the requesting tenant, in the manner of the virtual-time accounting in Deadline-Aware Batching and Tenant Fairness, restores the balance without penalizing the adapter itself.