Interruptible Capacity: Eviction Notices, Checkpointing, and the Cost of Being Preempted

A preemptible or spot instance is capacity sold at a discount because the provider retains the right to take it back. The discount is not a marketing gesture; it prices the option the provider keeps. Understanding what that option is worth requires knowing how the reclaim is signalled, how much time the signal gives, and what the workload must do with that time to avoid losing more than the discount saved.

The option the provider keeps

The provider’s problem is that demand for a given instance type in a given zone is not constant, and neither is the supply. Capacity exists as a set of physical hosts, and hosts are committed to on-demand and reserved customers first because those commitments carry an availability obligation. Whatever remains is sold as interruptible capacity, and the provider reclaims it when a higher-priority commitment needs it.

The reclaim is not random. It is triggered by a demand event, which means it is correlated across instances: when a large customer launches a fleet in a zone, many spot instances in that zone become candidates at once. This correlation is the property that makes naive diversification strategies fail. Ten spot instances spread across ten instance types in the same zone are not ten independent bets if the demand that triggers reclamation is zone-wide rather than type-specific.

The price mechanism differs by provider. Some price interruptible capacity by a market that clears continuously, so the price rises as demand for the spare capacity rises and an instance is reclaimed when its bid falls below the clearing price. Others use a fixed discount against the on-demand price, with reclamation driven by capacity needs rather than by price. The practical difference is whether the customer can influence the reclamation probability by bidding higher. Under a fixed-discount model, no bid exists; the only control is instance size and type.

The notice and its length

The reclaim arrives as a signal, and the signal is delivered through a channel the guest can observe without polling an external API. On some platforms it is an ACPI event or a metadata service flag that flips; on others it is a signal delivered to a system process that the guest image must be running to receive. The notice period is short, measured in tens of seconds to a couple of minutes depending on the provider, and it is not a guarantee of graceful termination. The provider begins the shutdown at the end of the period regardless of what the guest has done.

Two properties of the notice are commonly misread. First, the notice is best-effort: an instance can be terminated without one, particularly when the reclaim is due to a hardware failure rather than a capacity decision. A design that depends on receiving the notice has a hole in it. Second, the notice is delivered once. There is no escalation and no second warning, and the guest’s clock starts when the notice is delivered, not when the guest’s handler notices it.

The usable time is therefore the notice period minus the guest’s detection latency minus whatever the shutdown sequence itself consumes. A handler that takes two seconds to notice a flag polled every five seconds has already spent a meaningful fraction of the budget. This is why the reliable detection mechanism is an interrupt or an event, not a poll of a metadata endpoint.

# Illustrative drain sequence: only shell builtins and coreutils.
shutdown() {
    touch /var/run/draining                    # health check starts failing
    sleep 5                                    # let the balancer observe it
    kill -TERM "$(cat /run/app.pid)"            # stop accepting new work
    timeout 25 tail --pid="$(cat /run/app.pid)" -f /dev/null
}

The arithmetic in the script is the whole design. The drain delay must be long enough for the load balancer’s health check to notice and stop routing, the in-flight wait must be bounded, and the sum must fit inside the notice with margin. If the load balancer’s health check interval and unhealthy threshold sum to more than the notice period, the instance will be terminated while still receiving traffic, and the requests it was serving fail.

What must be checkpointed

The decision about what to persist is a decision about what the workload can afford to recompute. Three categories behave differently.

Stateless request handling needs no checkpoint at all, only a drain. The work in flight at the moment of the notice is either completed within the drain window or retried by the client. The retry is safe if the operation is idempotent, which is a property of the API rather than of the compute model.

Batch processing needs a checkpoint at a granularity that bounds the recomputation cost. If a job processes items in units that take a minute each and checkpoints after each unit, the expected loss is half a unit. If it checkpoints every hour, the expected loss is half an hour of work, and the effective throughput of the fleet is reduced by the reclamation rate times the loss. The optimum granularity balances checkpoint overhead against expected recomputation, and it shifts as the reclamation rate rises: cheap capacity with frequent interruption justifies frequent checkpoints.

Stateful services need a different treatment, because the state is not recomputable. A database on interruptible capacity must have its replication factor and its quorum placement arranged so that the loss of any single instance does not lose committed data. This usually means the interruptible instances are replicas rather than primaries, or that the replication protocol’s commit path does not depend on them. Running a quorum member on capacity that can vanish is a availability decision that has to be made explicitly, not inherited from the discount.

Correlated loss and the fleet

The most expensive failure mode is not one instance disappearing. It is a large fraction of a fleet disappearing within the same short window, which happens when the reclamation trigger is shared. A fleet sized to survive the loss of one instance survives the loss of one instance; it does not survive the loss of forty percent of itself.

The defences are structural. Diversify across zones as well as instance types, because a zone-wide demand event is the common trigger. Maintain a baseline of on-demand capacity sized to the peak load, and treat interruptible capacity as burst above that baseline, so that losing all of it degrades throughput rather than availability. Set the fleet’s target size against the worst plausible simultaneous loss rather than the average. Where the interruptible capacity lands is a scheduling outcome rather than a purchasing one: the placement decisions determine whether the stable baseline and the interruptible burst end up sharing failure domains.

There is also a control-plane consideration. Replacing many instances at once generates a burst of launches, and the provider may rate-limit launches per account per region. A fleet that loses a third of its instances and then cannot launch replacements quickly is down for longer than the notice period suggested. The replacement rate is a separate quota from the running capacity, and it has to be checked against the size of the fleet.

When the discount is not worth it

The break-even calculation is straightforward in principle and frequently skipped. The discount saves a fraction of the on-demand price per instance-hour. The cost is the expected value of the work lost to interruption plus the engineering cost of making the workload interruptible plus the cost of the baseline capacity held to absorb correlated loss.

A workload that is already stateless and horizontally scaled has an interruption cost near zero, and the discount is close to free money. A workload that requires a long, expensive initialization has an interruption cost that scales with how often it is interrupted, and a high reclamation rate can erase the discount entirely because the fleet spends its time re-initializing. A workload with a single instance and no redundancy has an interruption cost equal to its full availability, and no discount compensates for that.

The honest way to evaluate it is to compute the effective cost per unit of completed work rather than the price per instance-hour. The instance-hour price is the headline; the completed-work cost includes the recomputation and the idle capacity. For a workload with cheap checkpoints and a low reclamation rate, the two are close and the discount is real. For one with expensive initialization and correlated reclamation, they diverge sharply and the headline is misleading.

The interaction with the autoscaler is where most of the operational difficulty lives. A scaler that treats interruptible instances as equivalent to stable ones will size the fleet to the target and let reclamation reduce it below the target, then react by launching replacements into a capacity pool that may not have any. The scaler needs to know the difference, both to avoid counting interruptible capacity as guaranteed and to avoid a launch storm when a correlated reclamation occurs.