Concurrency Per Environment and the Unit a Function Platform Scales

A function platform does not scale by request. It scales by execution environment, and the number of requests an environment can hold at once is a property of the function’s configuration rather than of the platform’s capacity. This single fact reorganizes every capacity calculation: the scaling unit is a sandbox with a fixed memory and CPU allotment, the trigger for adding one is a concurrency threshold, and the queue that forms when the platform will not add more sits outside the function where the function cannot see it.

The scaling unit is the environment

When an invocation arrives, the platform must place it in an environment. If every environment is at its concurrency limit, the platform either creates a new one or the invocation waits. The decision is made by a controller that watches a metric — typically the number of in-flight requests per environment, or the number of queued invocations — and adjusts the environment count toward a target.

The important property is that the controller’s unit of adjustment is coarse. One environment brings with it a fixed memory allocation, a fixed share of vCPU, and a fixed cold-start cost if it was not warm. The controller cannot add a quarter of an environment. So the granularity of scaling is set by the environment’s size, and a function configured with a large memory allotment scales in large steps, which is why memory size is effectively a performance parameter on these platforms: more memory usually means proportionally more CPU, and the environment count needed to absorb a given load falls.

With a per-environment concurrency of one — the classic model — the required environment count equals the peak number of simultaneous in-flight requests. A function serving 500 requests per second at 200 milliseconds each has, by Little’s Law, 100 requests in flight on average, so it needs about 100 environments at steady state and more to absorb variance. Doubling the per-environment concurrency to two halves that number and halves the aggregate memory reserved, but it also means each environment is now running two handler invocations that compete for the same vCPU share.

That competition is the crux. The platform’s promise is that an environment has a certain amount of CPU; it does not promise that two concurrent invocations each get half of it at the same instant. Under concurrency, the runtime’s event loop interleaves the invocations, and a handler that performs a synchronous CPU-bound computation will block its sibling. A handler that awaits I/O will not. This is why the useful concurrency setting depends on the workload’s I/O fraction rather than on any platform default.

The arithmetic of concurrency and latency

For an I/O-bound handler, concurrency is nearly free. If a handler spends most of its wall time waiting on a downstream service, an environment with concurrency eight keeps eight requests in flight while consuming little CPU, and the platform needs one eighth as many environments. Aggregate memory drops by a factor of eight, cold-start frequency drops by a factor of eight, and per-request latency is essentially unchanged because the bottleneck was never the local CPU.

For a CPU-bound handler, concurrency is a queueing delay. If a handler saturates its share of vCPU for its entire duration, running two of them concurrently in one environment doubles each one’s latency. The platform’s concurrency setting has not created capacity; it has merely moved the queue from the platform’s scheduler into the runtime’s event loop, where the function cannot measure it and where the platform’s own metrics show healthy utilization.

The boundary is set by the fraction of time the handler is not runnable. A handler that is 90 percent waiting can support concurrency near ten before the CPU share becomes the binding constraint; a handler that is 10 percent waiting can support roughly one. The estimate is a ratio, not a measurement, and it is the only calculation needed to choose a setting that is not obviously wrong.

# Required environments, N, for a target throughput at concurrency c.
#
#   arrival rate          lambda  (requests/second)
#   mean handler duration L       (seconds, wall time)
#   in-flight requests    W = lambda * L          (Little's Law)
#   environments          N = ceil(W / c)
#
# CPU-bound handler: c > 1 multiplies latency by ~c, so keep c = 1.
# I/O-bound handler: c > 1 reduces N proportionally, latency roughly flat.

There is a second-order effect in the I/O-bound case. Concurrent invocations in one environment share the runtime’s connection pool, its heap, and its file descriptors. A downstream service that rate-limits per connection sees the aggregate from one environment as a single client, so concurrency multiplies the per-client request rate. The downstream service’s own limits then become the constraint, and the function’s concurrency setting has to be chosen against them rather than against its own CPU.

Where the queue lives

Every function platform has a maximum concurrency, and every platform has a queue in front of it. The two are configured separately and they behave very differently.

The maximum concurrency is a ceiling on environments, either account-wide or per function. A reserved concurrency setting carves out a guaranteed slice of the account’s ceiling for one function, which protects that function from being starved by others but also caps it: invocations beyond the reservation do not run, and what happens to them depends on the invocation source. A synchronous invocation returns an error to the caller. An asynchronous invocation is retried, and the retries accumulate.

The queue is where asynchronous invocations wait. It is not the function’s queue and the function cannot inspect it from inside the handler. Its depth is a platform metric, and it is the metric that matters for asynchronous workloads, because a queue that grows without bound means the function’s effective throughput is below its arrival rate and the backlog will eventually exceed the source’s retention. When the platform’s queue is full, the source drops or rejects events, and the failure surfaces at the source rather than at the function.

The distinction between synchronous and asynchronous failure modes is the one that catches people. A synchronous caller sees throttling as an immediate error with a specific status, which is a clear signal to apply backpressure upstream. An asynchronous producer sees success at the moment of enqueue and learns about the problem later, or never, if the retry policy exhausts and the events go to a dead-letter destination that nobody is watching. This is the same asymmetry that makes cold starts tolerable for background work and unacceptable for user-facing requests: the queue absorbs the delay when nothing is waiting on a deadline.

Concurrency as a bulkhead

Reserved concurrency is often described as a scaling control, but its more important role is as a bulkhead. A function that calls a shared downstream service can be capped so that its aggregate request rate against that service is bounded no matter how many invocations arrive. The cap is expressed in environments, and the effective request rate is the environment count times the per-environment concurrency divided by the handler duration.

This is the only mechanism on most platforms for enforcing a hard limit on a function’s blast radius. Without it, a retry storm in one function can saturate a downstream dependency and take out every other function that depends on it. With it, the storm is contained: the excess invocations fail or queue, the downstream service stays within its capacity, and the other functions continue.

The cost is that the cap is static. It does not know that the downstream service is healthy today and degraded tomorrow. Choosing it requires knowing the downstream service’s per-client limit and dividing by the maximum per-environment concurrency, and then leaving headroom for the fact that other clients also exist.

The interaction with cold starts and cost

Concurrency, cold starts, and billing are coupled through the environment as the common unit. Raising concurrency reduces the environment count for a given load, which reduces the number of cold starts and the total reserved memory. It also reduces the granularity of billing on platforms that bill per environment-second with a minimum duration: fewer, longer-lived environments bill less than many short-lived ones for the same work.

But raising concurrency past the point where CPU becomes the constraint converts a throughput saving into a latency regression that no metric attributes correctly. The platform reports the environment as fully utilized, because it was. The handler’s own timing shows the slowdown, but the handler cannot see that its sibling invocation caused it. The symptom is a function whose p99 latency rises after a concurrency increase that was intended to save money, with no corresponding rise in any platform-provided error rate.

The safe procedure is to raise concurrency only for handlers that are demonstrably I/O-bound, to verify with the handler’s own internal timing rather than the platform’s utilization metric, and to treat the per-environment CPU share as the budget that all concurrent invocations draw from. The cold-start cost is amortized by the same change, which is why the two settings are usually adjusted together and why the result is sometimes better than either change alone would suggest.