Instance Shape Selection and the vCPU-to-Memory Ratio

An instance type is a bundle: a number of virtual CPUs, a quantity of memory, a network allowance, and sometimes local storage, sold as one unit at one price. The bundle cannot be split. A workload that needs four cores and thirty-two gigabytes cannot buy four cores and thirty-two gigabytes; it buys whichever bundle contains at least that much of both, and pays for everything in the bundle it does not use. The ratio of memory to CPU across the available families therefore determines cost as directly as the workload’s own demand does.

The bundle is a rectangle, and the workload is a point

Families are commonly offered at memory-to-vCPU ratios of roughly two, four, and eight gigabytes per vCPU, with variants for storage-heavy and compute-heavy shapes. The exact numbers differ by provider and generation, but the structure is stable: a small number of ratios, each available at a range of sizes.

A workload’s requirement is a point in the plane of (vCPU, memory). To place that point in a bundle, both coordinates must be covered. If the point is (4, 32), a family at 8 GiB per vCPU needs four vCPUs and fits exactly. A family at 2 GiB per vCPU needs sixteen vCPUs to reach 32 GiB, so it pays for twelve vCPUs it cannot use. A family at 16 GiB per vCPU needs only two vCPUs but provides 32 GiB, so it pays for memory at a premium rate. The mismatch is not a small effect: the wrong family can cost two to four times the right one for the same work.

The shape of a fleet’s demand matters more than any single workload. A fleet of many small services that each need a gigabyte and half a core fits a low-ratio family; a fleet of caches and in-memory indexes fits a high-ratio one. Mixed demand on a single family strands capacity in both directions at once: some instances are memory-bound and idle on CPU, others the reverse, and the fleet’s total spend reflects the sum of the wasted halves.

Sizing on separate percentiles overstates the requirement

The usual rightsizing procedure measures CPU and memory over a window and picks a size that covers a high percentile of each. That procedure is wrong in a specific and predictable direction.

If CPU at the 95th percentile is four cores and memory at the 95th percentile is 32 GiB, the pair (4, 32) is not the 95th percentile of the joint requirement. The two maxima usually occur at different times: the CPU peak comes from a burst of computation, the memory peak from a large working set, and the instant when both are at their maxima may never occur. Sizing to the sum of separate percentiles therefore provisions capacity that no observed moment required.

The joint measure is a percentile over the vector, taken by sorting observations of the pair. Equivalently, compute the utilization of a candidate size directly: for each sample, the fraction of the candidate’s CPU and memory that is used, take the maximum of the two, and choose the percentile of that series. This yields a size at which the workload exceeded the bundle’s capacity in at most five percent of samples, which is the property actually wanted.

def fits(candidate, samples, pct=0.95):
    """candidate: (vcpu, gib). samples: [(cpu_used, gib_used)]. True if usage stays inside."""
    worst = [max(c / candidate[0], m / candidate[1]) for c, m in samples]
    worst.sort()
    return worst[min(len(worst) - 1, int(pct * len(worst)))] <= 1.0

The gap between the two methods grows with the correlation between the dimensions. For a workload whose CPU and memory rise together, the separate percentiles nearly coincide with the joint one. For a workload whose peaks are independent, the separate method over-provisions by the amount implied by the product of the tail probabilities, and the over-provisioning is paid every hour.

The choice of percentile is itself a cost decision. Sizing at the 99th percentile instead of the 95th buys tolerance for the rarest peaks at the price of capacity that is idle for almost all of the month. The correct percentile is the one whose exceedance the workload can absorb: a stateless service behind a queue can tolerate being undersized for a minute, and a service with a strict latency objective cannot.

Burstable instances move the cost into a credit balance

Some families offer a baseline performance level with the ability to burst above it, governed by a credit mechanism. Credits accrue while the instance runs below its baseline and are consumed while it runs above. The billing implication is that the price is set for the baseline, so a workload that stays near the baseline is cheap and a workload that bursts continuously is not.

Sustained bursting has two possible outcomes, and which one applies is a configuration choice. In one mode the instance is throttled to its baseline when credits are exhausted, which converts a cost saving into a latency penalty with no billing signal. In the other mode the instance may continue to burst and is billed for the surplus credits, which converts the saving into a variable charge that appears as an unbudgeted line. Neither outcome is visible in a utilization chart, because the utilization during throttling is measured against the baseline rather than against the burst.

The economics are straightforward once the credit balance is treated as a second currency. The average burst above baseline over a month must be less than the average accrual, or the instance is mispriced for the workload. A workload with a daily peak and a quiet night accumulates credits at night and spends them during the peak, which is exactly the pattern the mechanism is designed for. A workload with a flat, sustained load above baseline never accrues, and the family is simply the wrong purchase.

Placement is a second packing problem

Once instances exist, they must be placed on physical hosts, and the provider does that packing. The buyer’s control is indirect: the choice of family, the size, and the use of placement mechanisms that constrain which instances share a host. These constraints have prices. Spreading instances across hosts for availability costs more because it consumes host slots that could otherwise be densely packed; dedicating hosts to a single tenant costs the full host regardless of utilization.

Inside a cluster the buyer controls, the packing is explicit. The scheduler places pods by declared requests, so the cost of a fleet is determined by the sum of requests rather than by the sum of usage. A request that is set at the 99th percentile of one dimension and left generous in the other produces a fleet with low utilization and a high bill, and the mechanism that would fix it, adjusting requests to match measured usage, is the same joint-percentile computation applied per pod rather than per instance.

Fragmenting the fleet across many small instance sizes makes packing worse. A host has a fixed capacity, and the residual after placing large instances may be too small for the next instance even though the total free capacity is sufficient. Larger instances reduce the number of fragments; smaller ones increase placement flexibility at the cost of more fragments. The trade is between the granularity of the fit and the efficiency of the packing, and it has no universal answer, which is why the observed utilization of a fleet is a property of its instance-size distribution as much as of its workload.

The shape decision is not reversible for free

Changing an instance family is not a resize of a running process. It requires stopping the instance, or replacing it, and the replacement carries the cost of the transition: the new instance starts cold, its caches are empty, and its connections are re-established. A fleet of many instances changed at once produces a load spike on the remaining capacity, which is why the change is usually done in batches.

For stateful systems the constraint is stronger. A database’s memory allocation is sized to its working set, and changing the instance size means a restart with a recovery period proportional to the data it must read. The cost of the transition is real and is paid once; the cost of the wrong shape is paid every hour. The asymmetry means a shape that is slightly too large is usually preferable to a migration that must be repeated, and that the correct target for a sizing exercise is the shape that will remain correct for the next several months rather than the shape that fits this week’s measurements.

Where the workload’s ratio is genuinely unusual, the alternative to a mismatched bundle is to split the workload: run the CPU-heavy part on a compute-optimized family and the memory-heavy part on a memory-optimized one, connected by a network path whose cost is described in egress boundaries and billable data transfer. Splitting recovers the ratio but adds a hop, and the hop is only free if both parts stay within a zone.