Virtualized Performance Counters and the Time They Do Not Count

A physical processor has a small set of performance counters, each of which can be programmed to count one event, and a control register that determines which events are counted and in which privilege modes. The counters are a hardware resource with a fixed population, and virtualization must decide who gets them. Giving them to the host loses the guest’s ability to profile itself. Giving them to the guest means the host must save and restore counter state on every context switch and must somehow keep the guest from seeing the host’s activity through them.

The result is a resource that is virtualized differently from memory, CPUs, or devices, because the cost of virtualization is borne in the measurement path rather than in the data path. The instrument changes what it measures.

Programming the counters on behalf of another context

The plainest model is assignment: the hypervisor programs a subset of the physical counters to count for the guest, and on every transition to or from that guest it reads the counts, adds them to a per-vCPU accumulator, and reprograms the counters for whichever context is about to run.

Reading and writing counter MSRs are privileged operations that trap, so a naive implementation would make every vCPU switch several exits long. Hardware reduces this: the control structure has fields that save and load a list of general-purpose registers, and the same facility is used to save and load a set of performance monitoring registers as part of the exit and entry sequences. The list of MSRs that can be handled this way is bounded, so the hypervisor must choose which registers to include. A guest that uses more counter registers than the list accommodates gets the remainder managed in software, with an exit per access.

The alternative model is to count selectively in software. The hypervisor can leave the guest’s events programmed but not enabled, enable them at entry, and disable them at exit by clearing the appropriate bits in the global counter control register. Hardware support for this is the same facility used to make the counters behave, and the effect is worth stating precisely because it determines what the guest’s numbers mean.

The cycles that happen between two guest instructions

Consider a guest instruction that causes a VM exit. The guest’s own cycle counter is running while the instruction executes, and by the time the instruction is intercepted the counter has recorded the cycles the processor spent decoding and executing its way to the exit. Then the exit happens, the host runs a handler whose execution may take an order of magnitude longer than the emulated instruction, and the guest resumes.

If the counters are disabled for the duration of the host’s execution, the guest’s cycle count for that operation contains only the pre-exit portion. If they are left running, they contain the exit time as well, but they also contain whatever other activity the physical processor performed while the counters were running, which may include work belonging to another context that the host scheduled in between.

Neither choice is wrong, and they answer different questions. Counters disabled during host execution measure the guest’s own instructions and are the right instrument for optimizing guest code. Counters left running measure the guest’s experience including the virtualization tax and are the right instrument for capacity planning. The default in most configurations is the first, which produces the systematic omission: a guest under heavy exit load reports a cycle count that excludes the largest cost it is paying.

The gap is measurable without any special instrumentation. Take a workload that performs a fixed amount of guest work per operation, measure the guest’s wall-clock duration per operation and its retired-instruction and cycle counts, then repeat it on a system with a lighter or heavier exit load. The instruction count will be nearly identical and the cycles per instruction, as reported from inside the guest, will barely move, while the wall time changes by a factor that the guest’s counters do not show. The difference is the exit cost plus the time the vCPU spent descheduled, which is precisely the steal accounting that the guest keeps separately.

#include <linux/perf_event.h>
#include <string.h>
#include <unistd.h>
#include <sys/syscall.h>

/* Count cycles in guest code only: no hypervisor samples, no samples
   taken while the processor was in host context. */
static int open_guest_cycles(void)
{
    struct perf_event_attr attr;
    memset(&attr, 0, sizeof(attr));
    attr.type = PERF_TYPE_HARDWARE;
    attr.size = sizeof(attr);
    attr.config = PERF_COUNT_HW_CPU_CYCLES;
    attr.disabled = 1;
    attr.exclude_kernel = 1;
    attr.exclude_hv = 1;          /* skip samples attributed to the hypervisor */
    return syscall(__NR_perf_event_open, &attr, 0, -1, -1, 0);
}

The exclude_hv bit is the one to notice. It suppresses counting while the processor is executing at the hypervisor’s privilege level, which is what makes an in-guest profile resemble a bare-metal profile of the same code. It also removes the information that would have shown where the time went.

Multiplexing and the counts it produces

A processor has more events of interest than it has counters. The standard technique is to assign each event a fraction of the time, switch the counter’s event selection at intervals, and scale the recorded count up by the reciprocal of the fraction. The scaled number is an estimate of what the count would have been had the event been counted continuously.

For a workload whose behaviour is uniform in time, the estimate is good. For a workload with phases, such as one that alternates between compute and I/O, the estimate is a weighted average of two different rates, and it can be wrong by a large factor in either direction. The error is not reported with the number, so a profile of a phased workload that shows an event at forty percent of a theoretical peak may be showing a real forty percent or a fifty percent phase weighted down by a ten percent phase, and the scaled counts alone cannot distinguish them.

Virtualization adds a second layer of multiplexing when the host also wants counters. The guest’s counters are scheduled alongside the guest’s vCPUs, so an event that is not programmed for the entire time the vCPU is running yields a count that must be scaled by the vCPU’s run time, not by wall time. A guest profiler that computes a rate per second from a multiplexed count and divides by wall time will understate the rate by the ratio of run time to wall time, which is the same ratio the steal counter reports.

Attribution when the interesting work is in the host

The hardest case in virtualized profiling is a guest whose time is consumed by mechanisms the guest cannot see. A packet that requires an emulated device register access consumes host cycles for the decode, the device model call, and the entry, and none of that appears in any guest counter.

The instrument for that case runs on the host and attributes samples back to guest addresses. A sampling profiler running on the host can, when a sample lands in the hypervisor while it is handling an exit for a particular vCPU, read the vCPU’s saved instruction pointer and the guest-physical address of the code and report the sample against the guest function that caused the exit. This requires the host to read the guest’s symbol tables and to translate guest addresses, and it produces a profile in which the time is charged to the guest operation rather than to the hypervisor function, which is the attribution that answers the question.

Exit count per reason is the complementary measurement. The host can record, per vCPU, why each exit occurred and how long it took, which turns the question from “where did the time go inside the guest” into “which guest operations generate exits”. That is the measurement that connects the mechanisms in this cluster: whether an exit is an MMIO access, an EPT violation, an interrupt that posted interrupts did not capture, or a CPUID. The reason histogram is the actionable one, and it is invisible from inside the guest.

Sampling precision and its virtualization

Hardware-assisted sampling reduces the interval between the event that triggers a sample and the instruction at which the sample is recorded. Without it, a counter overflow interrupt arrives some number of instructions after the instruction that caused the overflow, an error called skid, and on an out-of-order processor busy with long-latency memory operations the skid can exceed hundreds of instructions. Attribution is then systematically biased toward instructions that retire slowly.

The hardware fix records sample data in a buffer in memory without an interrupt, using a mechanism that captures the precise instruction address and some register state. Under virtualization, the buffer is guest memory that the host must arrange for the hardware to write, and the sample records contain instruction pointers and data addresses that the host must translate before they mean anything. A second-level mapping is involved, which means the sample-writing path pays the same translation cost as any other access, and a misconfiguration causes samples to be dropped rather than misattributed, which is at least a failure that announces itself.

Sampling on one vCPU while another vCPU of the same guest is running raises a further issue of interpretation. A sample that lands in the hypervisor on behalf of a vCPU is attributable to that vCPU, but a sample that lands in the host kernel for unrelated reasons is not. Profilers that do not distinguish the two produce a profile of the host that is presented as a profile of the guest.

What to trust

The reporting layers are consistent once their scopes are kept separate. Guest counters that exclude hypervisor execution describe guest instructions and predict how the guest would perform on bare metal with the same data. Guest wall time describes the guest’s experience and includes everything. Steal time describes the part of that experience attributable to not being scheduled. Host-side exit accounting describes the part attributable to the hypervisor’s work on the guest’s behalf.

No single one of these numbers is a profile of the guest, and the common error is to read one of them as if it were. A guest that reports a cycle count per operation falling while its wall time per operation rises is not contradictory; it is the signature of virtualization overhead growing in the gap between the two, and only the host-side instruments can say which mechanism put it there.