Replica Packing and Topology Constraints

A replica of a small model is one process on one device, and placing it is an ordinary scheduling problem. A replica of a model that does not fit on one device is a collective: several processes, each holding a shard of the weights, each participating in an all-reduce on every layer, all advancing in lockstep. The scheduler cannot place the members independently, because the cost of the collective depends on where the members sit.

The shape of a sharded replica

Tensor parallelism splits each weight matrix along one dimension. In the standard arrangement for a transformer, attention and feed-forward projections are split by column and row so that each rank computes a partial result and one all-reduce per layer combines them. For a model of P parameters across k ranks, each rank holds roughly P/k parameters plus a small amount of replicated state such as normalization weights and embeddings in some configurations.

The communication volume per layer is what matters for placement. The all-reduce exchanges activations of size batch times hidden width per token; for a hidden width of 4,096 and a batch of one position, the message is 8 KiB in 16-bit precision, and it is exchanged twice per layer in the standard split. Across 32 layers and 80 ranks that is a fixed pattern of small messages, repeated every iteration. Small messages are latency-bound rather than bandwidth-bound, so the interconnect’s latency and its ability to overlap communication with computation determine whether adding ranks helps.

The scaling consequence is well known and asymmetric. Adding ranks reduces the weight read per rank, which is what decode is limited by, so decode scales close to linearly until the all-reduce latency becomes comparable to the per-layer compute. Prefill, being compute-bound, scales less well because the arithmetic per rank also shrinks while the all-reduce count stays fixed. A sharded replica therefore has a shape: a number of ranks beyond which decode stops improving, and a different, smaller number beyond which prefill stops improving.

Memory arithmetic sets the floor on rank count. A 70-billion-parameter model in 16-bit precision needs 140 GB of weights, which does not fit on a single 80 GB device. Two ranks give 70 GB of weights each, leaving 10 GB for cache and workspace, which is not enough for a meaningful batch. Four ranks give 35 GB each and leave room. The rank count is thus chosen by the joint constraint of fitting weights and leaving cache space, not by throughput alone.

Why the placement unit is a gang

Because the ranks must communicate every layer, they must be mutually reachable with low latency and high bandwidth. Within a node, that means the NVLink domain or its equivalent; across nodes, it means an interconnect whose latency is small relative to a layer’s compute time. Placing one rank on a fast island and another on a slow one makes every iteration wait for the slowest link, and the whole replica runs at the speed of its worst pair.

The scheduler therefore treats the replica as a gang: all k ranks are placed together or not at all. This is a constraint on the packing problem, not merely a preference. A node with three free devices cannot host a four-rank replica even though it has capacity, and a cluster with fragmented free devices can refuse to schedule a replica while reporting abundant free memory in aggregate.

Two placement patterns follow. The first confines a replica to one node, which is possible when the rank count is at most the device count per node and the model fits in the node’s combined memory. It has the simplest communication properties and the coarsest packing granularity: a node is either hosting a replica or available for one. The second spans nodes, which is necessary for large models and requires the interconnect to be provisioned for it; the gang then occupies a slice of the fabric that must be allocated as a unit.

Gang scheduling interacts badly with bin packing. A bin-packing scheduler that fills nodes to the brim with small replicas leaves no node with enough free devices for a large one, and the large one waits while the cluster looks busy. The standard remedy is to reserve capacity for the largest gang, or to use a scheduler that understands the gang size as a first-class resource rather than as a set of independent device requests.

def can_place(node, ranks, devices_per_node):
    # a gang either fits inside one node's free devices or spans nodes
    if ranks <= devices_per_node:
        return node.free_devices >= ranks
    # spanning requires the interconnect to be provisioned for the group
    return node.free_devices == devices_per_node and node.fabric_slice_available

Failure and upgrade domains

A gang is also a failure domain. If one rank’s device fails, the replica cannot continue: the collective cannot complete with a missing participant, and the remaining ranks cannot take over its shard because they do not hold those weights. Recovery means restarting the whole gang, reloading all shards, and rebuilding any cache that was resident. From the router’s perspective the replica disappears entirely, which is a coarser loss of capacity than a single-device failure in a replica that fits on one device.

The same coarseness applies to upgrades. A change to the model weights, the parallelism configuration, or the runtime version must be applied to all ranks together, since a version mismatch across ranks corrupts the collective. Rolling upgrades therefore proceed gang by gang, with the pool sized to absorb the loss of one gang at a time. That requirement is why serving fleets often keep spare capacity equal to the largest replica rather than to a percentage of the fleet, and it is the reason capacity planning for sharded models differs from capacity planning for stateless services.

Cache residency makes the loss expensive in a second way. A replica holding thousands of sequences has a large key-value cache whose contents cannot be reconstructed; the sequences must be restarted or replayed. Whether that is acceptable depends on the workload, and it argues for routing long-running sequences away from replicas that are about to be drained. Draining a gang deliberately, by stopping admission and waiting for resident sequences to finish, converts a failure into a delay bounded by the longest output length.

Packing against the interconnect

The fabric itself is a resource with a shape. A cluster of nodes connected by a fat-tree or a torus has groups of nodes that communicate more cheaply than others, and a gang placed across a cheap group performs better than the same gang spread across expensive links. Schedulers that ignore topology see a flat pool and place gangs wherever devices are free, which produces replicas whose throughput depends on an accident of placement.

Two approaches mitigate this. Topology-aware scheduling treats the fabric as a hierarchy and requires a gang to fit within one subtree, so that all communication stays inside a cheap region. This is the same idea as rack awareness in storage placement, applied to an interconnect with much lower latency. The alternative is to accept arbitrary placement and to make the collective robust to it, which usually means larger messages, more overlap, and a lower rank count, giving up some decode throughput for placement freedom.

There is a further interaction with phase separation. When prefill and decode run on separate pools, they have different optimal shapes: prefill prefers few large devices, decode prefers many devices with fast links. A single sharded configuration cannot be optimal for both, so a cluster that hosts both pools must either partition devices between them or accept that one pool is misconfigured. The protocol consequences of that split are in Prefill and Decode Disaggregation, and the capacity consequences are in Autoscaling Signals for Token Servers.

The packing decision also constrains what the router can assume. A router that spreads requests across many replicas assumes replicas are interchangeable, and gangs of different sizes are not: a four-rank replica and an eight-rank replica have different throughput and different latency under the same batch. Treating them as equal in a least-outstanding comparison underweights the larger one, which is a load-scoring problem that begins in the placement layer and surfaces in Least-Outstanding Routing.