Virtualization · Platform Engineering

Service Level Objectives for a Platform, Not for a Service

A platform SLO measures whether tenants can get work done, not whether a component is up. How to define the request journey, choose indicators that tenants feel, and handle the shared-dependency problem.

A service’s availability is measured by whether its own requests succeed. A platform’s availability is measured by whether its tenants can accomplish what they came to do, and those are not the same quantity. The API server can be answering every request while deployments fail, because the failure is in the admission webhook, the image registry, the scheduler’s capacity, or the pipeline that renders the template. An SLO defined on the API server’s error rate would report that platform as healthy.

The unit of measurement is a tenant journey

The first decision is what counts as a request. For a platform, the request is not a single API call but a journey with several steps, and the indicator must be attached to the journey’s end.

A representative journey for a deployment is: the tenant submits a change, the pipeline renders the pinned template version, admission accepts the rendered objects, the scheduler places them, the images pull, the containers start, and readiness reports true. Each step has its own failure mode, and the tenant experiences the sum.

An indicator on the journey’s completion is therefore a ratio of journeys that reached readiness within a time bound, over journeys attempted. The time bound matters as much as the ratio: a deployment that completes in four hours has not served the tenant even though it succeeded. The bound is derived from what the platform is for, which is turning a change into a running service quickly enough to matter, and it should be published as part of the objective rather than left implicit in a dashboard.

The alternative indicator, measuring each step separately, produces a set of numbers that cannot be combined into a tenant-facing statement. It is useful for diagnosis and useless for a promise.

Why the denominator has to be counted carefully

A ratio needs a denominator, and for a platform the denominator is the set of legitimate attempts. Three exclusions are common and only two are defensible.

Attempts that fail validation because the tenant’s input violates the declared schema are not platform failures. The schema is the contract; rejecting an invalid request is the contract working. Counting these as failures makes the platform’s error budget depend on tenant mistakes and produces an SLO that improves when tenants become more careful, which is not a property of the platform.

Attempts that fail because the tenant’s own workload crashes are also not platform failures, provided the crash is attributable to the workload. Attribution requires the platform to distinguish a container that exits non-zero from a container that never starts, which is a distinction the runtime reports and the pipeline must preserve.

Attempts that fail because of a dependency the platform does not control are the hard case. An image registry outage or an identity provider outage causes real tenant pain and is not caused by the platform’s code. Excluding it produces an SLO that looks good during an outage everyone noticed; including it produces an SLO the platform team cannot influence. The resolution that survives scrutiny is to count it, report it separately as an attributed external failure, and keep the objective’s arithmetic inclusive while the reporting separates the causes. An objective that excludes the failures tenants experience is a measurement of the wrong thing.

The shared-dependency problem

Platform SLOs inherit the topology they run on. If every tenant shares one control plane, then the control plane’s availability bounds every tenant’s journey, and the platform’s objective cannot exceed it. This is arithmetic, not pessimism: if the control plane is unavailable for a fraction $u$ of the time and no journey can complete during that time, the platform’s achievable completion ratio is at most $1 - u$, whatever the platform’s own software does.

Two consequences follow. The first is that the objective must be set below the shared dependency’s demonstrated availability, or it is a promise the topology cannot keep. The second is that improving the objective beyond that bound requires changing the topology, which is exactly the trade described in cluster segmentation and blast radius: separation buys independent failure, and its cost is the operational surface and the reserved capacity.

The same arithmetic applies to dependencies inside the journey. If a deployment requires an image pull from a registry with its own availability, the journey’s success rate is bounded by the registry’s. The platform can reduce the coupling by caching images at the node level, which converts a registry outage into a failure that affects only new images rather than all deployments, and that change is worth more than any amount of work on the platform’s own components.

Error budgets as a release gate

An error budget is the amount of failure the objective permits. With a 99.5 percent objective over a 30-day window, the budget is 0.5 percent of attempts, which is roughly 3.6 hours of full unavailability equivalent, or a proportionally larger number of partially failed journeys.

The budget’s use is to gate change. When the budget is exhausted, the platform stops taking risky changes and spends the remaining time on reliability. When the budget is intact, the platform can take risks it would otherwise defer.

For a platform, the gate has a second dimension that services do not have: the platform’s changes affect every tenant’s journey at once. A template change that breaks rendering, an admission rule that rejects a common pattern, a default that makes workloads unschedulable, each of these can consume the budget in a single rollout. The gate should therefore apply to changes in proportion to their reach, and the reach of a template change is the number of tenants pinned to the affected version. A change that affects one tenant’s parameters can be released freely; a change that affects every tenant at the current version cannot.

The mechanism that makes this operational is a canary in the fleet rather than in the code: apply the change to a small set of tenants, measure the journey completion ratio for those tenants, and proceed only if it holds. The measurement has to be the tenant-facing one, because a change that keeps the API server’s error rate flat while breaking rendering passes every component-level check.

What the platform should not promise

There is a class of objective that looks attractive and is unmeasurable: the platform should be “easy to use”, or should “reduce time to production”. These are outcomes with many contributors, and attaching a number to them produces a metric that moves for reasons unrelated to the platform.

The measurable version is narrower and more useful: the fraction of journeys that complete without a support request. That number counts the cases where the tenant could not proceed and needed a human, and it is attributable to the platform in a way that satisfaction surveys are not. It also has a direct operational reading, because every support request is a journey the platform failed to make self-service, and the reasons attached to those requests are the backlog.

A second unmeasurable objective is a promise about the tenant’s own reliability. The platform can promise that a workload, once running, stays running subject to declared resources; it cannot promise that the workload does not crash. Keeping the boundary at the platform’s own actions is what makes the objective honest.

Reporting that survives an incident

The reporting format matters because the objective is consulted during disagreement. When a tenant says the platform was down and the platform says it was up, the resolution is a shared definition and a shared measurement, and neither exists unless both were published before the incident.

Three elements make the report usable. The journey definition, stated in terms of steps a tenant recognizes. The measurement window and the exclusions, stated as rules rather than as judgement calls made after the fact. And the per-tenant breakdown, because an aggregate ratio can hide a tenant whose journeys failed systematically while the aggregate stayed inside the objective. A platform whose overall ratio is inside the objective but which failed one tenant’s every attempt for a day has a problem the aggregate conceals.

The per-tenant view is also what makes the objective actionable at the boundary between platform and tenant: it shows whether a failure is shared or specific, which is the first question in every incident review. The lifecycle question of what happens when a path’s objective is no longer met and the path is retired rather than repaired is covered in sunsetting a golden path.

References

  1. Site Reliability Engineering: How Google Runs Production Systems — Beyer et al., O'Reilly, 2016
  2. Implementing Service Level Objectives — Alex Hidalgo, O'Reilly, 2020
  3. The Site Reliability Workbook — Beyer et al., O'Reilly, 2018