Scheduling Batch Training Against Shared Accelerator Capacity

A training job is not a long-running service with a large memory footprint. It is a set of workers that must run concurrently, communicate in lockstep for the duration, and produce nothing until the last of them finishes. That shape breaks the assumptions of a scheduler designed for request-serving processes, and the adaptations required are specific: all-or-nothing placement, a definition of fairness that survives the fact that a job’s resource request changes at the last moment, and a preemption policy that does not destroy the work of an interrupted job.

All-or-nothing placement

Data-parallel training runs $N$ workers that synchronize at every step, typically through an all-reduce over gradients. If $k$ of the $N$ workers are placed and the remaining $N - k$ wait in a queue, the placed workers are consuming memory and idling in a collective that cannot complete. Under a first-come-first-served scheduler that places what it can, this produces a specific pathology: the cluster has enough free capacity for the entire job only at moments when nothing else is placed, and in between, partial placements block the capacity they hold.

Gang scheduling makes the placement atomic: the scheduler reserves capacity for every worker before starting any of them, and if the full set cannot be placed, none is started. The scheduler must therefore consider the whole job as a unit when choosing what to run, which changes the optimization problem from per-task placement to per-group placement. Two costs follow. Fragmentation is worse: a job needing $N$ units may not fit even when the total free capacity exceeds $N$, and the scheduler must be able to build a placement decision over a large candidate set rather than a single task.

The composition of the gang depends on the parallelism strategy. Data parallelism asks for $N$ identical workers. Model parallelism asks for workers pinned to a specific topology, because a tensor shard on one node talks to shards on specific other nodes, and a placement that crosses a slow link degrades the whole job rather than one worker. Pipeline parallelism has a third shape, with stages that must be ordered and co-located with their neighbours. A scheduler that understands only “$N$ identical units” can place the first and will place the other two badly, in a way that shows up as reduced throughput rather than as failure.

Quotas, and what a fair share means

The base unit of allocation is an accelerator: a device with a fixed memory size and a fixed throughput. Allocating fractional devices is possible on hardware that supports partitioning, but the scheduling problem is essentially the same with units of half and quarter.

Fair sharing across teams is usually implemented as a quota: each team has a share of total capacity, and jobs are admitted when the team is under its share. The naive interpretation, a hard ceiling on concurrent consumption, leads to idle capacity when a team’s jobs finish and no others are queued, and to starvation when a single large job cannot fit under any team’s ceiling alone. A work-conserving variant permits a team to exceed its share when others are idle, with the excess reclaimable when demand returns. This is the property Borg calls reclaimable capacity, and it is the reason a cluster’s reported utilization can approach capacity while every team’s quota is far below it.

When several resource types are involved, accelerators, host memory, CPU for data loading, network for distributed storage, the fairness definition has to be multidimensional. Dominant resource fairness compares allocations by each user’s dominant share, the maximum over resource types of the user’s fractional allocation, and allocates the next unit to whoever has the smallest dominant share. The property that makes it the standard is that it is strategy-proof: a user cannot obtain more by misreporting the resource mix the job needs. The alternative, equalizing raw counts of accelerators, ignores memory and network and produces unfair allocations in practice.

Preemption that respects checkpointing

Preemption is what makes reclaimable capacity work: when a team that owns the capacity returns, jobs running on borrowed capacity must release it. For a training job, releasing capacity means stopping a process that may have hours of progress in memory. The scheduler’s policy and the job’s checkpoint discipline are therefore a single design.

Checkpoint frequency sets the expected lost work. If a job checkpoints every $T$ minutes and is preempted at a uniformly random time, the expected work lost is roughly $T/2$ minutes plus the restart and data-loading time, which for a large job can exceed the interval itself. Frequent checkpoints reduce the loss and add overhead proportional to the size of the model state and the bandwidth to storage. A model with a billion parameters in 32-bit precision occupies about 4 GB per replica of weights alone, and optimizer state for a common Adam-style update multiplies that by two or three, so a checkpoint is a multi-gigabyte write and cannot be made arbitrarily frequent.

The scheduler’s contribution is to give the job warning. Preemption with a grace period, a signal that the job must stop within some minutes, and a job that uses the interval to write a final checkpoint, converts an unpredictable restart into a bounded pause. Immediate termination, which is appropriate for stateless services where a replica can be replaced, is destructive here. The grace period interacts with the placement decision: a scheduler that preempts a job and cannot place the replacement until the job has actually released its devices has to sequence the two, or it produces a window in which the capacity is idle.

An alternative policy avoids preemption for a class of jobs entirely. If the cluster maintains a reserved pool sized for the organization’s committed demand and a separate opportunistic pool, jobs that must not be interrupted run in the reserved pool and preemptible work runs elsewhere. This costs capacity in the form of the reserve, and it is the price of a workload mix where some jobs are interactive and some are batch.

Data loading as a scheduling constraint

The accelerator is rarely the only resource a training job needs, and the scheduler that models only device count produces jobs that run slower than their placement suggests. Each worker needs host CPU to decode and augment input data, host memory for buffers, and network bandwidth to fetch shards. A job placed eleven workers on a node with twelve cores and one device each has enough cores only if the decoding fits in one core per worker, which for image augmentation it does not.

The observable symptom is a job that is placed successfully and reports low device utilization. From the scheduler’s perspective the job is running; from the job’s perspective it is waiting for the next batch. The fix is to declare the host resources per worker and let the scheduler bind them, which reduces the packing density and, in exchange, makes the job’s throughput predictable. Systems that skip this step often compensate by over-requesting host resources defensively, which is a manual version of the same declaration with less accuracy.

Storage locality has the same character. If each worker streams its shard from remote storage, the aggregate read rate can saturate the network, and the mitigation is to cache shards on local disks ahead of time and schedule the job where the cache exists. That converts placement into an affinity constraint, which conflicts with the packing objective: the nodes with the cached shard may be the nodes that are full.

When a job should not be scheduled at all

The scheduler is the right place to enforce some policies and the wrong place for others. Admission control that rejects a job whose declared resource request exceeds any plausible node, a single worker asking for more device memory than exists, saves everyone time. A queue that holds jobs indefinitely produces the illusion of capacity while users wait; a queue with a deadline that fails the job explicitly lets the user resubmit with a smaller shape.

What the scheduler cannot do is decide whether a job is worth running. A cluster that admits every job spends capacity on trials that would have been abandoned after ten minutes had anyone looked at the first evaluation. Places where that judgement lives vary, but the structural point is the same as in the rest of this cluster: the resource whose shortage matters most is often not the accelerator but the attention needed to decide what to train, which is the concern behind model registries and promotion gates and the retraining triggers described in closing the retraining loop.