Queue Time, Service Time, and What a Timeout Measures

A service reports a median handler duration of four milliseconds. Its callers report timeouts at a five percent rate against a one-second limit. Both numbers are correct, and the gap between them is entirely made of waiting that the handler’s own timer does not cover.

Server-side instrumentation conventionally starts when the request reaches application code and stops when the response is written. Everything before the first line of the handler and everything between the last line and the client’s receipt is excluded. That excluded region is where a timeout usually fires.

The four waits in one round trip

A single request to a typical service passes through at least four distinct waits before any work is done on its behalf.

Connection establishment is the first. A new TCP connection costs one round trip for the handshake, and TLS adds one or two more depending on the protocol version and whether the session is resumed. On a link with fifty milliseconds of round-trip time, that is fifty to a hundred and fifty milliseconds spent before a single request byte is sent. Connection reuse removes it entirely for subsequent requests, which is why pooling is less an optimisation than a correction.

The accept queue is the second. The kernel completes the handshake and places the connection on the listening socket’s accept queue, where it waits until the application calls accept(). The queue has a bound, somaxconn or the backlog passed to listen(), and when it is full the kernel’s behaviour depends on the protocol: for TCP it may drop the SYN and let the client’s own retransmission retry, which converts an overloaded server into a client that sees a slow handshake rather than an error.

The worker queue is the third, and on most application servers it is the largest. A connection is accepted but no thread or goroutine is free to read from it, or the request has been read and is waiting for a slot in a bounded worker pool. This wait is a function of arrival rate against concurrency, and it is the term that grows without bound as utilization approaches one.

Service time is the fourth: the actual execution. It is the only one that the handler’s timer measures.

The client’s timeout covers all four plus the return path. That asymmetry is the whole explanation for the discrepancy in the opening paragraph: the handler is fast, and the sum is not.

The queueing curve is not linear

For a single-server queue with random arrivals and random service times, the mean time in system is 1/(μ - λ) where λ is the arrival rate and μ the service rate. The mean time waiting, excluding service, is ρ/(μ - λ), or equivalently (ρ/μ)/(1 - ρ) with ρ = λ/μ.

The useful form of that expression is the behaviour of 1/(1 - ρ) as ρ approaches one. At 50 percent utilization the factor is 2. At 80 percent it is 5. At 90 percent it is 10. At 95 percent it is 20. At 99 percent it is 100. Nothing about the server changed across those rows; only the ratio of demand to capacity did.

This is why “the server is only at seventy percent CPU” is not an argument that latency should be fine. The mean wait at seventy percent utilization is already about 2.3 times the service time, and the sensitivity near the top of the curve is severe: a two percent increase in arrival rate at 95 percent utilization raises the mean wait by roughly a factor of 1.6.

Real systems are not single-server M/M/1 queues, and the constant differs, but the shape does not. The 1/(1-ρ) term appears in every queueing approximation in use, and it is the reason capacity planning targets a utilization ceiling rather than an average.

Measuring the split

The split is observable, and it is worth instrumenting because the aggregate number hides which term is responsible.

The gap between the client’s send time and the server’s first handler timestamp is the pre-handler wait. On the server, the gap between connection acceptance and the first read is the accept-to-read wait, and the gap between the request being complete and the handler starting is the worker wait. Summing those against the handler duration gives a decomposition that attributes the delay.

Linux exposes the pieces. /proc/net/tcp and ss -lnt report Recv-Q, which for a listening socket is the count of completed handshakes waiting to be accepted. A Recv-Q that stays near the backlog value is a server that is not calling accept() fast enough, which usually means the application is saturated rather than the kernel. On the application side, a monotonic clock read at accept, at first byte, and at handler entry gives the internal split directly.

var last time.Time
for {
	conn, err := ln.Accept() // takes the next entry from the accept queue
	if err != nil {
		return err
	}
	if !last.IsZero() {
		// turnaround: everything the loop did between two accepts
		observeLoopTurnaround(time.Since(last))
	}
	last = time.Now()

	go func(c net.Conn) {
		start := time.Now()
		var b [1]byte
		if _, err := c.Read(b[:]); err != nil { // blocks until the client sends
			c.Close()
			return
		}
		observeFirstByte(time.Since(start))
		serve(c)
	}(conn)
}

A long turnaround means the loop was occupied between accepts, so completed handshakes sat in the queue while the application did something else; the queue depth itself is visible as Recv-Q in ss -lnt. A large first-byte time means the client connected but did not send, which on a busy server is more often flow control than a slow client: the client’s request bytes are queued behind a full receive buffer, and the read blocks on data that has already arrived at the host.

Why a server’s own p99 looks healthy

Three mechanisms produce the illusion.

Sampling starts inside the handler, so queue time is invisible to it. This is the dominant effect and it is structural rather than a mistake in the instrumentation.

Cancelled requests are often excluded. If the handler is cancelled when the client disconnects, and the instrumentation only records completed requests, then every request that timed out at the client disappears from the server’s latency histogram. The server’s recorded distribution is conditioned on success, which is precisely the subset that was fast.

The concurrency model distributes the wait. With one thread per connection and no admission control, requests that wait do so in the kernel or in the scheduler, where they are not measured. With an event loop, the wait appears as latency inside a handler that was supposed to be non-blocking, and it is attributed to whichever handler was unlucky enough to be running when the loop stalled.

The correction is to measure at the boundary the client experiences. A server that records the time from connection acceptance to response completion, including all queueing, produces a distribution that matches what callers see, and the difference between that and the handler distribution is the queueing cost, available as a metric rather than an inference.

The timeout as a filter on the distribution

A timeout at value T does not just fail requests slower than T; it changes the distribution that the server observes. Requests that would have completed at T + ε are cancelled and never recorded, so the measured p99 converges to something below T regardless of the true distribution. Tightening the timeout improves the measured latency and worsens the user experience, and the two move together.

This interacts with the queueing curve in a way that is easy to miss. When queue time pushes requests past the timeout, they are cancelled, which frees capacity, which reduces queue time, which lets the remaining requests through. The system can oscillate around the point where queue time equals the timeout, with throughput determined by the cancellation rate rather than by capacity. Requests that are cancelled after partial processing still consume the work already done, so the oscillation costs more than it appears to.

Setting the timeout therefore has two separate jobs: bounding the client’s wait, and defining the false-failure rate against the healthy distribution. Those are not the same value. A timeout at the p99.9 of the healthy distribution fails one healthy request in a thousand by design; a timeout at twice the p99.9 fails almost none but allows genuinely stuck requests to occupy capacity for that entire duration. The right compromise usually sits somewhere above the measured tail and is revised when the tail moves.

Where the decomposition stops being useful

The model assumes requests are independent and service times are drawn from a stationary distribution. Neither holds under the conditions that matter most.

Retries break independence: a failed request returns as a new arrival, and the arrival process becomes a function of the service process. The queueing curve then applies to a load that grows as the queue grows, and the stable region shrinks. Retry Budgets and the Load Multiplication They Prevent covers the resulting dynamics.

Long-running requests break stationarity. A request that holds a worker for thirty seconds while a hundred short requests queue behind it produces a latency distribution with a mode at thirty seconds that no mean-based model predicts. Separating those requests by class, or by concurrency pool, is the remedy, and the isolation mechanics are in Bulkhead Isolation and the Design of Partial Degradation.

Multi-stage fan-out breaks the single-queue assumption entirely. A request that waits in one queue and then in three parallel ones has a latency determined by the maximum of the three, and the maximum of several draws from a distribution is much further into the tail than any single draw. That arithmetic is the subject of Hedged Requests and the Arithmetic of Tail Latency.

The practical conclusion is narrower than the theory. A latency number without an attribution to queue time and service time is not actionable, because the two have opposite remedies: queue time is fixed by reducing arrival rate or raising concurrency, and service time is fixed by making the work cheaper. Choosing between those without the split is guessing.