The Instance Credential Chain and What the Metadata Service Exposes

A process on a virtual machine needs credentials, and the machine has no secure place to keep a secret. The disk is an image shared by every instance launched from it, the environment is readable by every process on the host, and any file written at boot is visible to anyone who can read the filesystem. The design that resolves this removes the secret entirely: the machine asks a local endpoint for credentials, and the endpoint decides whether to answer based on where the request came from.

A link-local endpoint with no authentication

The instance metadata service listens on a fixed link-local address, 169.254.169.254, on port 80, and answers HTTP GET requests from the instance itself. RFC 3927 defines the link-local range and the rules for claiming an address in it; the service uses that range because a link-local address is not routable off the local segment, so the endpoint is by construction unreachable from outside the host’s own network.

A request to the credentials path returns a JSON document containing a key identifier, a secret access key, a session token, and an expiry. The response carries no authentication of the caller. Any process on the instance that can open a TCP connection to that address and port receives credentials for the instance’s role. There is no per-process identity, no file permission to check, and no user context consulted.

This is the design decision that everything else follows from. Credentials are scoped to the instance, not to a process. The unit of least privilege is the machine, and a machine running one workload has a coherent identity while a machine running five has an identity shared by all five.

Why a shared endpoint was acceptable

The alternative designs each cost more than they saved. Baking a secret into the image means the secret is in every copy of the image, including the ones handed to developers. Fetching a secret at boot from a configuration service requires the instance to authenticate to that service, which is the same problem one level up. Encrypting the secret at rest on the instance requires a key, and the key has the same problem.

The metadata service breaks the recursion by making the network position the credential. The instance is inside a network boundary that the provider controls, and the address is only reachable from inside. Anyone who can reach the endpoint is, by construction, already on the machine. From the provider’s position that is a reasonable definition of “the instance is asking”.

It is reasonable, and it is wrong in one specific case that turned out to be common: an application on the instance that can be induced to make an HTTP request to an arbitrary URL. A server-side request forgery vulnerability lets an attacker who can influence a URL cause the application to fetch http://169.254.169.254/latest/meta-data/iam/security-credentials/<role>, and the response is a working credential. The application was the deputy, and the network position it lent was the instance’s.

The address being link-local does not help here, because the request originates from the instance and is therefore exactly the kind of request the service is designed to answer.

The request-side defence

The mitigation is to require proof that the caller is a process on the instance rather than a request that the instance was tricked into making. The mechanism is a two-step exchange: a PUT to a token endpoint with a specified hop limit returns a session token, and subsequent GET requests must carry that token in a header.

The PUT requirement is the load-bearing part. A cross-protocol attack that induces an HTTP GET cannot produce a PUT, because the attacker’s vector is a URL that the victim fetches. Requiring PUT and requiring a header on the follow-up GET removes the class of attack in which the victim’s own request is replayed, though it does not remove an attack that can issue arbitrary requests with arbitrary methods and headers.

The token has a lifetime and the response advertises it. The hop limit is a separate control: it bounds how many network hops a token-carrying request may traverse, so a token cannot be proxied out of the instance’s own network namespace into a container network that the instance does not control. The default hop limit is 1, which means the token works only from the instance itself and not from a container behind an additional bridge.

The second control is to disable the service version that has no token requirement at all, at the instance or account level. Setting the metadata options to require the tokenized version means a request without a token is refused rather than served, and the refusal is logged. This is a per-instance setting that can be enforced at launch time by a policy condition on the launch call, which is the pattern described in attribute conditions and tag-based access: the launch is denied unless the metadata options in the request require the tokenized version.

Containers and the hop limit

A container on the instance has its own network namespace. A request from inside the container to the link-local address traverses the container bridge, which is one hop, and then reaches the host, which is a second. With a hop limit of 1 the token request is refused, and the container cannot obtain credentials.

The usual response is to raise the hop limit to 2, which makes the endpoint reachable from containers. That restores convenience and reintroduces the problem: any container on the host can now obtain the instance’s credentials, regardless of which workload it belongs to. The blast radius of a compromised container becomes the instance role, and the instance role is usually broader than any single container needs.

The alternative is to not use the instance role for container credentials at all, and instead inject credentials per workload through a mechanism that scopes them to the pod or task. Those mechanisms exist and they work by giving each workload its own endpoint and its own role, which is the correct shape: the unit of privilege matches the unit of failure. The instance role then covers only the host-level agent, which needs very little.

This is the same trade as the shared-endpoint decision at the top, moved down a level. Sharing credentials within a failure domain is cheap and expands the blast radius to the whole domain. Splitting credentials per workload costs an extra endpoint and an extra role per workload, and it makes revocation meaningful, because revoking one workload’s credentials does not affect its neighbours.

What else the endpoint exposes

The credentials path is the part that matters for authorization, but the same endpoint serves instance identity documents, user data supplied at launch, and network configuration. The identity document is a signed JSON blob containing the instance identifier, the account, the region, the image identifier, and the availability zone, and it is signed by the provider so a third party can verify that a request came from a genuine instance without contacting the provider.

User data is the dangerous one in practice, because it is whatever the launch process put there. Scripts that bootstrap an instance frequently include configuration with secrets in it, and user data is readable from the metadata endpoint by any process on the instance and by anything that can reach the endpoint. The service does not encrypt it. A secret placed in user data is a secret placed on a world-readable endpoint within the instance, and the only reason it is not world-readable outside is the network boundary that the token exchange is meant to reinforce.

The identity document is also the basis for a different pattern: an instance can present its signed identity document to a token service that trusts the provider’s signing key, and receive credentials for a role without the token service ever seeing the instance’s network position. That is federation, and it is the same shape as the workload identity exchange described in OIDC federation and claim validation, with the provider acting as the issuer.

Latency, caching, and expiry in the credential chain

The metadata endpoint answers from local state, so a credential fetch is a local HTTP round trip rather than a network call to a regional service. That is why libraries fetch credentials lazily and cache them: the cost of a miss is small but not zero, and the cost of a fetch per request would be paid on every call.

Expiry handling is where the chain breaks in practice. The response includes an absolute expiry, and the standard client behaviour is to refresh before it passes, with some jitter. A process that caches credentials in a file for other processes to read reintroduces the problem the endpoint was designed to remove: the file is a secret on disk, it outlives the process that wrote it, and it is readable by anything with filesystem access. The endpoint exists so that no such file needs to exist, and writing one gives back the exposure that was avoided.

The final constraint is that the credential is bound to the instance’s role, and the role’s policies are looked up at request time. Changing the role’s policies changes what an already-fetched credential can do, immediately, because the credential never contained the permissions. That property, and the difference between narrowing a session and revoking one, is set out in role assumption and session tokens.

The chain as a whole is a sequence of narrowing decisions: which role is attached to the instance, which policies are attached to the role, which session policy caps the session, and which conditions the request satisfies. Each step is a place where the effective permission can only shrink. None of them is a place where it can grow, which is why the design is safe by construction and why the failures that occur are failures of scope rather than of escalation.