DevOps · Platform Automation

Delegating Provisioning Without Delegating the Account

How a self-service pipeline obtains cloud credentials through workload identity federation, what the trust policy must pin, and how the blast radius stays bounded.

Self-service provisioning has an unavoidable asymmetry: the person requesting a database usually should not be able to create one directly, and the pipeline that creates it must be able to. The pipeline therefore holds authority that the requester does not, and every request it processes is a request to exercise that authority on someone else’s behalf. This is the confused deputy problem in its original form: a program with permission is asked to use that permission by a party that lacks it, and the program has to decide whether the request is legitimate.

The design question is not whether to delegate. It is what the delegate can prove about the request, and how much damage a wrong decision causes.

Why stored credentials fail this specifically

The traditional arrangement places a long-lived access key in the CI system’s secret store. It works, and it fails in three ways that compound.

The key is not tied to a workload. Anything that can read the secret can use the key, from any network, for as long as it exists. A leak in a build log, a fork pull request that echoes environment variables, a compromised dependency in a build step — all of them yield a credential that outlives the incident.

The key’s permissions are the pipeline’s permissions, for every request the pipeline handles. A pipeline that provisions databases, buckets, and queues for any team needs all of those permissions, so a bug in one provisioning path exposes all of them. Narrowing per path means multiple keys, which multiplies the number of long-lived secrets.

Revocation is coarse and slow. Rotating the key breaks every job until the new value propagates, so rotation happens on a schedule that is longer than anyone would like, and an incident response that needs to revoke one key ends up revoking the pipeline.

Federation replaces the secret with a signed assertion

Workload identity federation inverts the trust. The CI platform issues a short-lived token — a JWT, per RFC 7519 — that asserts facts about the job: which repository, which branch or environment, which workflow, which audience. The cloud provider is configured to trust that issuer’s signing keys and to accept tokens that satisfy a policy over those claims. In exchange it returns temporary credentials with an expiry measured in minutes to an hour.

Nothing long-lived is stored. The token exists for the duration of the job, is issued only to a job the CI platform actually started, and cannot be replayed after it expires. Revocation becomes a matter of removing the trust relationship or changing the claim the trust policy requires, both of which take effect for the next token rather than for the next rotation window.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.example" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.example:aud": "sts.amazonaws.com",
        "token.actions.example:sub": "repo:platform/infra:environment:production"
      }
    }
  }]
}

The structure is the same across providers even where the claim names differ: identify the issuer, identify the audience, and constrain the subject. Kubernetes does the equivalent internally with bound service account tokens, which are audience-scoped, time-scoped, and projected into the pod rather than stored as a Secret, so a stolen token is worth less than a stolen Secret and expires without anyone rotating anything.

The subject claim is the entire security boundary

A trust policy that accepts any token from the issuer is a trust policy that accepts any job in the CI system, including one started by a fork pull request. The subject claim is what distinguishes them, and it has to be pinned to a value the attacker cannot influence.

The dangerous patterns are stable across implementations. A wildcard in the subject — matching every repository in an organisation, or every branch in a repository — means a contributor who can push a branch can obtain the production role. A subject that includes a user-controlled name, such as a workflow file path or an environment name that any repository admin can create, moves the decision to whoever controls that name. And a subject pinned to a branch rather than to a protected environment means the protection rules on the environment are not part of the trust decision.

The corresponding safe pattern is narrow on both axes: one repository, one protected context, and an audience that names the intended service. Where the CI platform distinguishes pull requests from branch pushes in the subject, that distinction is load-bearing and must not be erased by a wildcard. Where it does not distinguish them, the trust policy cannot tell them apart and the platform has to enforce the separation somewhere else.

There is a second-order problem: the trust policy is itself configuration, often written by the same pipeline it constrains. A change to the trust policy is a privilege escalation that looks like a routine diff. It belongs under review by a different group than the one that benefits from it, which is a policy question rather than a technical one — but the technical part is that the policy document should live in a repository whose approvals are stricter than the pipeline’s.

Narrowing what the credential can do

Federation answers “may this job assume this role”. It does not answer “may this role do this thing”. Three mechanisms narrow the second question.

Session policies. A role can be assumed with an inline policy that further restricts the resulting session. The restriction can only intersect, never extend, so a session policy is a safe way to hand a pipeline a role that is broader than any single job needs while keeping each job inside its lane.

Permission boundaries. A boundary attached to a role caps the maximum permissions the role can have, independently of its identity policies. The useful property is that the boundary is written once by the platform and the per-team policies can be written by the teams: a team that grants itself too much still cannot exceed the boundary.

Resource-level constraints enforced outside the credential. A role that may create buckets can be prevented from creating one whose name falls outside the team’s prefix by a policy that evaluates the request, not by the credential. This is the admission-time pattern described in what an admission webhook can decide, and it has the advantage of being visible to the requester as a rejection rather than as an opaque authorization failure.

The choice between these is a choice about who can grant what. Putting the constraint in the credential means the platform team writes every team’s policy. Putting it in a boundary plus an admission rule means teams write their own policies inside a fixed frame. The second scales; the first is easier to reason about when the number of teams is small.

Lifetimes, replay, and the metadata endpoint

Short-lived credentials reduce the value of a leak without eliminating it. A token valid for an hour is valid for an hour regardless of who holds it, so the useful controls are the ones that bound what it can do and make its use visible.

Replay is the concrete risk. A token extracted from a build log, or from a process’s environment, can be used from anywhere until it expires, because the token asserts facts about the job, not about the network location of the caller. Some providers allow a source-IP condition on the trust policy; that helps only where the CI egress range is stable, which is often not the case for hosted runners.

The metadata service is the other classic path. A workload that can reach the instance metadata endpoint can obtain the instance’s credentials, which is why server-side request forgery against that address is a recurring finding in web applications. Modern metadata services require a session-establishing request before returning credentials, which raises the cost of a naive forgery but does not remove the risk: an attacker who can make arbitrary requests from the workload can also make the session request.

Audit: the credential names the job, not the person

Every action taken with a federated credential is logged against the assumed identity — the role, and the session name if one was set. That is enough to attribute the action to a pipeline run and not enough to attribute it to a human, unless the pipeline passes the requester’s identity through.

The mechanism for that is a session name or session tag derived from the request, so the audit record carries both the machine identity and the human one. Without it, an incident review can establish that the provisioning pipeline created a publicly readable bucket and cannot establish who asked for it. RFC 8693 describes the general shape of this delegation, where a token carries both the subject on whose behalf the action occurs and the actor performing it; the cloud-specific session attributes are the practical equivalent.

The residual limitation is that the pipeline is trusted to report the requester honestly. If the pipeline derives the identity from a request parameter, a caller can claim to be someone else. Deriving it from the authenticated identity of the request — the token the requester presented to the self-service API — is the only version that holds, and it makes the self-service front end part of the security boundary rather than a convenience layer over it. The same reasoning applies to how the provisioned resources are later modified: once the credential is gone, the guarantee that only the owning team can change the resource has to come from the resource’s own policy, which is what makes quota accounting and admission rules the continuing enforcement rather than the initial creation.

References

  1. RFC 8693: OAuth 2.0 Token Exchange — IETF
  2. RFC 7519: JSON Web Token (JWT) — IETF
  3. Kubernetes Documentation: Managing Service Accounts — Kubernetes project