Cloud Infrastructure · Cloud Networking
Private Endpoint Services and the Reverse-Direction Connection
How a service in one VPC becomes reachable from another through an address that lives in the consumer subnet, why the connection is accepted by the provider, and what the endpoint does not hide.
Peering and transit both require the consumer’s network to have a route to the provider’s addresses, which means the consumer must know those addresses and must accept the provider’s prefix into its route table. Private endpoint services invert that: the consumer gets an address inside its own address space, and traffic to that address is delivered to the provider without the consumer ever routing to the provider’s prefix. The mechanism is a one-way, provider-authorised connection built on a load balancer on the provider side and an interface in the consumer’s subnet.
The address the consumer owns
A consumer creates an endpoint for a service it has been told about, and the endpoint creates a network interface in a chosen subnet with an address from that subnet’s range. Nothing is created in the provider’s network on behalf of the consumer except a record of the association.
This is the inversion that makes the design useful. The consumer’s route table needs no entry for the provider’s prefix, because the endpoint’s address is inside a prefix the consumer already routes locally. The consumer’s address space is unaffected: the endpoint consumes one address from a subnet, and the provider’s prefixes never appear. Two consumers whose own address spaces overlap can both use the same service, because neither one’s traffic is addressed to the other’s space. Peering cannot do this; the whole point of the endpoint is that the two address spaces never have to be reconciled.
The interface is a real interface with an address, and it is subject to the consumer’s own filtering. A security group attached to the endpoint’s interface governs what may reach it, and it is evaluated exactly as it would be for any other interface. That is worth stating because it is a common misconception that the endpoint is a transparent pipe: it is a destination, and the consumer’s own policy applies to it before anything reaches the provider.
What sits on the provider side
The provider publishes a service by attaching a network load balancer to it. The balancer’s listeners define the ports the service accepts; the balancer’s targets are the actual endpoints of the service, usually in the provider’s own subnets. The published service references the balancer, and the balancer is what makes the service addressable by the endpoint machinery.
The load balancer is not incidental. It is the component that terminates the consumer’s connection and opens a separate one to a target, which is the same two-connection model described in load balancer listeners and target selection. The consequence for the endpoint design is that the provider’s service must be reachable through a balancer, and a service that cannot sit behind one, whether because it needs the client’s source address preserved at the transport level or because it uses a protocol the balancer does not support, cannot be published this way.
The balancer’s own address is never visible to the consumer. The endpoint’s address is the only address the consumer sees, and the translation from endpoint address to balancer happens in the provider’s fabric.
Acceptance is a two-party decision
Creating an endpoint is not sufficient. The provider must accept the connection request, and until it does, the endpoint exists in a pending state and does not pass traffic. The acceptance step is the security boundary: a provider that publishes a service does not thereby expose it to every consumer that learns the service name.
Two modes exist. In one, the provider reviews each request and accepts or rejects it individually, which gives the provider a list of exactly which consumers are connected. In the other, the provider pre-authorises a set of principals, and requests from those principals are accepted automatically. The automatic mode trades the review step for an identity check, and its security depends entirely on the identity being unforgeable, which is why it is tied to the provider’s account-level identity system rather than to a name or an address.
The acceptance model has an operational consequence that surfaces during decommissioning. A consumer that deletes its endpoint does not notify the provider in any way the provider can rely on; the provider discovers the change by observing that the endpoint is no longer connected. A provider that maintains a list of authorised consumers and removes entries only when asked will accumulate entries for endpoints that no longer exist, and each entry is a standing authorisation that would accept a new endpoint claiming the same identity.
Naming and the DNS override
An endpoint is reachable by its address, but services are usually addressed by name, and the name resolves to the provider’s public or private address, not to the endpoint’s. The endpoint therefore has to change what the name resolves to, for the consumer only.
The mechanism is a private hosted zone associated with the consumer’s network, containing a record for the service’s name that points at the endpoint’s address. Resolution inside the consumer’s network returns the endpoint address; resolution anywhere else returns whatever the public record says. This is the same split-horizon pattern used for internal names generally, and it depends on the resolver being the one the consumer controls.
The override has a failure mode worth naming. A consumer whose instances use a resolver that is not the consumer’s own, either because the instance was configured with an external resolver or because the resolver is reached across a path that bypasses the private zone, will resolve the public address and connect to it, bypassing the endpoint entirely. The connection may even succeed, if the service is also publicly reachable, which means the traffic silently takes a path the design intended to eliminate. Because the failure is a successful connection, it is not noticed until someone inspects the path.
# Inside the consumer network the name must resolve to the endpoint address.
dig +short api.internal.example.com
# A public address here means the private zone is not in the resolution path.
The check is worth performing from an instance rather than from a workstation, because the resolver configuration that matters is the one the instance uses. An endpoint that is correctly created and accepted can still be unused by every client that resolves the name the wrong way.
What the endpoint does not do
Several properties are commonly assumed and are not provided.
It does not make the provider’s network reachable. The endpoint delivers traffic to the published service’s listeners and to nothing else. A consumer cannot use it to reach a database in the provider’s VPC, an administrative interface on a target, or any address other than the ones the balancer exposes. The isolation is per-service, not per-network.
It does not provide name resolution for the provider’s internal names. Only the names the provider publishes, mapped through the consumer’s private zone, resolve to the endpoint. Everything else in the provider’s namespace is invisible.
It does not carry traffic in the other direction. The connection is initiated by the consumer; the provider cannot open a connection to the consumer through the endpoint. A service that needs to call back into the consumer, such as a webhook delivery or a push notification, needs a separate mechanism, and the endpoint will not serve as one. This asymmetry is the same one a NAT gateway has, and it exists for the same reason: there is no mapping that would tell the provider’s fabric where to deliver an unsolicited connection.
It does not remove the need for the consumer’s own egress control. Traffic to the endpoint stays inside the consumer’s network until it reaches the endpoint interface, so it does not traverse an internet gateway or a NAT gateway, which is often the point. But it does traverse the consumer’s subnets and is subject to their ACLs, and a subnet ACL that permits the endpoint’s port only from a specific range will block clients outside that range with no indication that the endpoint is the cause.
Cost and the direction of the charge
The billing model follows the direction of the connection rather than the volume of configuration. In the offerings that publish a price for this mechanism, the consumer is charged for the endpoint interface and for data processed through it, while the provider is charged for the load balancer and, in most cases, for the data the balancer handles. A service with many consumers therefore has a cost that grows with the number of endpoints rather than with the number of connections, since each consumer creates at least one interface per zone it uses.
The interface-per-zone requirement is the architectural detail that matters most for availability. An endpoint created in one zone fails when that zone fails, and the consumer’s clients in other zones have no path to the service. Creating an endpoint in each zone the consumer uses costs one address and one interface per zone, and it is what makes the service’s availability independent of any single zone. A design that creates one endpoint and expects zone redundancy has confused the endpoint with the service: the service may well be multi-zone, but the consumer’s path to it is not.
Where it fits among the other mechanisms
The three connectivity mechanisms answer different questions. Peering joins two networks and lets each address the other, and why VPC peering is not transitive and what a transit gateway adds covers the many-to-many case. A transit gateway joins many and holds the routing policy in one place. A private endpoint joins a consumer to one service without either network knowing the other’s addresses.
The choice is therefore about what must be addressable. If the consumer needs to reach many services and hosts in the provider’s network, the networks must be joined. If the consumer needs one service, and especially if the two address spaces overlap or the provider is a different organisation, the endpoint is the only mechanism that works at all. The overlap case is the decisive one: peering and transit both fail on identical prefixes, and the endpoint does not care, because the only address the consumer uses is one it allocated itself.
References
- RFC 1918: Address Allocation for Private Internets — IETF
- RFC 1035: Domain Names — Implementation and Specification — IETF
- AWS PrivateLink Concepts — Amazon Web Services