Steal Time and vCPU Skew: What Happens When a Guest Gets Fewer Cores Than It Asked For

A virtual machine with eight vCPUs does not own eight processors. It owns eight threads that the host scheduler places on physical CPUs according to the host’s policy and the host’s load. When the host cannot run all of them at once, some vCPUs do not run, and the guest has no direct way to observe it. Instructions do not fail, memory does not move, and the guest’s own scheduler continues to treat its runqueue as if it were backed by real cores.

This is the fundamental asymmetry of CPU overcommit. Memory overcommit at least produces a fault when it goes wrong; CPU overcommit produces nothing but slowness, and slowness with the wrong shape. A workload that loses 30 percent of its CPU but keeps everything else behaves predictably. A workload whose lock holders are descheduled while dozens of waiters spin can lose far more than the fraction of time that was taken.

Making the invisible visible

The hypervisor knows when it preempts a vCPU, because the preemption is its own scheduler decision. It can therefore report the event. The mechanism is a paravirtualized MSR that the guest writes early in boot with the guest-physical address of a small per-vCPU structure in guest memory. The host then maps that structure and keeps it up to date.

The structure holds an accumulator of nanoseconds during which the vCPU was runnable but not running, plus a version counter and a status byte. The version counter exists because the host updates the structure while the guest may be reading it, so both sides use the standard seqlock pattern: the guest reads the version, reads the accumulator, reads the version again, and retries if the two version values differ or are odd.

On each entry to the guest, the host accounts for the time the vCPU spent waiting in the host’s scheduler since its last exit and adds it to the accumulator. The guest’s timekeeping code, when it reads the clock, folds that accumulated value into the appropriate category. That is why the per-CPU lines in the guest’s /proc/stat carry a steal column distinct from idle: the guest knows the difference between having nothing to do and having work and not being allowed to do it.

# Fields 2..9 are user nice system idle iowait irq softirq steal, in clock
# ticks since boot. Fields 10 and 11 (guest, guest_nice) are already included
# in user and nice, so summing them too would double-count.
awk '/^cpu[0-9]+ / {
  total = 0
  for (i = 2; i <= 9; i++) total += $i
  printf "%s steal=%.2f%%\n", $1, 100 * $9 / total
}' /proc/stat

The exactness of this accounting depends on the host’s clock source. If the host’s notion of elapsed time is derived from the physical time-stamp counter and the counter does not run at a constant rate, the conversion needs a multiplier that the host supplies. A hypervisor that reports steal without also reporting how to convert it leaves the guest with a number it cannot use, which is why the paravirtual clock interface publishes both.

Why spinning multiplies the loss

The direct effect of steal is that the guest receives less CPU than its share. The indirect effect is much larger and comes from synchronization.

Consider a guest kernel taking a spinlock. In the traditional design, a waiter repeatedly tests a memory location until the holder clears it, and no other work can proceed meanwhile. The design assumes the holder is currently running on a processor and will release the lock within a few hundred cycles. Under CPU overcommit that assumption is false: the holder may be descheduled between taking the lock and releasing it.

The waiters now spin against a lock that cannot be released until the host reschedules the holder. Every spin iteration consumes the waiters’ CPU share, which is itself scarce, and produces no progress. The guest’s throughput collapses even though the grand total of CPU time the guest receives is only moderately reduced. This is the classic lock-holder preemption problem, and its cost scales with the number of waiters, not with the length of the stolen interval.

Hardware provides a partial answer. A pause instruction in a spin loop is already a hint that the loop is waiting, and the processor can be configured to exit to the hypervisor when a loop containing it, bounded by a gap of instructions, exceeds a repetition count. The hypervisor receives the exit, concludes that the vCPU is spinning on a lock, and yields the physical CPU in the hope that the lock holder gets scheduled. Both the gap threshold and the repetition count are hypervisor configuration parameters, and the count is usually adjusted dynamically: a vCPU that keeps exiting for this reason gets a larger allowance so that genuine short spins stop costing exits, while a vCPU that never triggers gets a smaller one.

The software answer is stronger. Paravirtualized spinlocks replace the memory test with a test of a byte in the shared per-vCPU structure, which the host sets to indicate that the vCPU is not currently running. A waiter that sees the holder is descheduled stops spinning and yields its own vCPU instead, which lets the host schedule the holder sooner and removes the wasted spin. The mechanism requires the guest’s lock implementation and the host’s structure to agree on the byte’s meaning, which is why it is a negotiated feature rather than an unconditional one.

TSC synchronization and the meaning of time

Guest timekeeping is a second-order performance problem with the same root cause. The x86 time-stamp counter increments at a constant rate, and the guest reads it to derive wall-clock time. Under virtualization the counter’s value and the guest’s notion of elapsed time diverge: the guest’s wall clock must advance while the vCPU is descheduled, but the counter keeps running regardless.

The paravirtual clock solves this by giving the guest a shared structure containing the current wall time, a multiplier and shift to convert counter ticks to nanoseconds, and an offset that accounts for the vCPU’s position. The guest reads the counter and the structure and computes. The structure is per-vCPU because the offset differs between vCPUs.

The trap is that guest code often assumes the counter is synchronized across processors, which holds on most bare-metal systems and need not hold in a virtual machine whose vCPUs are running on physical processors with different counter offsets. A guest that timestamps an event on one vCPU and measures elapsed time on another can obtain a negative duration. The hypervisor can compensate by supplying an offset per vCPU, which the guest adds, but only if the guest’s timekeeping code knows to do so. Guests that read the raw counter and assume synchronization will show occasional anomalous intervals that no amount of CPU overcommit tuning removes.

Skew between vCPUs of the same guest

A guest with multiple vCPUs expects them to make progress at comparable rates. When the host runs some of them and not others, the guest’s view of time diverges between its own processors. The consequences are concrete: a thread holding a lock on one vCPU blocks its peers; an RCU grace period, which requires all vCPUs to pass through a quiescent state, takes as long as the slowest vCPU needs; a barrier or a distributed counter stalls; and a TLB shootdown cannot complete until every target vCPU has been scheduled and acknowledged the interrupt.

The mitigation at the host level is to prefer placing the vCPUs of one guest together in time as well as in space, with a tolerance for skew. Strict gang scheduling, where all vCPUs of a guest run together or not at all, wastes CPU when the guest is not using all its processors and is rarely implemented in that form. The practical approach is affinity and a scheduler that, when it must preempt one vCPU of a guest, prefers to preempt another vCPU of the same guest rather than leaving the guest half-scheduled. This makes the guest’s remote work more balanced at the cost of making its total CPU time lower, which is usually the better trade for a guest running a tightly coupled workload.

The user-visible consequence is that two guests with identical assigned CPU counts can perform very differently depending on how many other guests are competing for the same physical cores and whether the host is willing to skew a guest’s own vCPUs.

The cost of an idle vCPU

A vCPU with no work executes a halt instruction. The processor exits, and the host has a choice. It can block the vCPU thread, which frees the physical CPU for another guest but means that waking the vCPU later requires a context switch and a scheduler wake-up whose latency is measured in microseconds. Or it can spin in host context for a bounded time, watching for an interrupt, so that a vCPU that halts briefly and wakes quickly resumes without ever leaving the physical CPU.

The second option is a bet that the idle period is shorter than the polling window. Hypervisors implement it with an adaptive window: it grows while polling succeeds in avoiding a block and shrinks when polling wastes time. The trade is directly between latency and CPU waste, and it is the reason a host full of guests that are each lightly loaded can consume more CPU than the sum of their useful work.

Polling interacts with overcommit in a way that is easy to get backwards. If the host is heavily overcommitted, spinning on behalf of an idle vCPU takes a CPU away from a vCPU that has work. If the host is lightly loaded, spinning costs nothing because the cores would be idle anyway and saves a wake-up. The optimal window therefore depends on host load, and a fixed value will be wrong in one of the two regimes.

Symptoms and their misreading

Guest-visible symptoms of CPU overcommit are easy to attribute to the workload. Context-switch rates measured inside the guest rise, because the guest’s own scheduler is compensating for processors that are not running. Runqueue latency measured inside the guest rises. Applications show long-tailed response times that do not correlate with their own CPU consumption.

The measurement that separates the cases is steal time against the guest’s own busy time. A guest whose vCPUs are descheduled heavily shows a steal fraction that tracks its busy fraction, which is the signature of contention for physical cores. A guest whose steal is low but whose internal runqueue latency is high has a problem inside itself. The two look identical in an application-level latency graph and completely different in the accounting, which is why the paravirtualized counter exists: without it, the guest cannot distinguish the two at all, and every investigation starts from the wrong hypothesis. What the guest’s own performance counters report during the same interval makes the distinction sharper still, since a counter that stops while the hypervisor runs leaves out exactly the time the steal counter is measuring.