Bounded Queues and Backpressure as Throughput Control

A queue with no limit is not a safety margin. It is a decision to accept unbounded delay, on the argument that accepting work is always better than refusing it. The decision holds only while arrivals are below the service rate, which is exactly the case where the queue is unnecessary. Above that rate, the queue grows without bound, memory fills, and eventually the process is killed or stops making progress entirely. The delay peak is not the failure; the failure is that by the time anything notices, the system is holding minutes of work that no client still wants.

A bound is the alternative. It says that once the waiting work reaches a certain size, further arrivals are refused immediately and cheaply. That converts a slow, memory-consuming degradation into an immediate, measurable rejection, and it moves the responsibility for the excess upstream to where it can be handled.

Why an unbounded queue fails late and total

The arithmetic of a queue is that length grows at the excess rate. If a service completes 9,000 items per second and receives 10,000, the backlog grows at 1,000 per second. A queue that started empty reaches 60,000 items after a minute. If each queued item holds a request context of a few kilobytes, that is hundreds of megabytes of memory that the process did not have before, and it was acquired at exactly the moment the process is least able to afford allocation overhead.

Two properties make this worse than the numbers suggest. First, the clients that arrive during the overload are the ones that will wait longest and are most likely to have timed out before their response is produced. Work is being done for clients that are no longer listening. Second, the memory growth and the queue scanning cost add load to the system that is already overloaded, so the service rate falls as the backlog grows. A queue that was already unable to keep up becomes less able to.

The recovery is worse than the entry. When the arrival rate falls back below the service rate, the service continues to process the backlog, and the response times remain terrible until the backlog drains. The system appears broken for as long as the backlog takes to clear, which for a large queue is minutes, long after the cause has passed.

A bound converts overload into a visible signal

With a bound of $B$ items, the resident count can never exceed $B$, so the maximum delay is bounded by $B$ divided by the service rate. A queue of 1,000 items at 10,000 items per second of service capacity has a maximum wait of 100 milliseconds. Memory is bounded by the per-item size times $B$, known at configuration time.

When the queue is full, the arrival must be refused, and the refusal is the point of the design. It happens in microseconds, before any work is attributed to the request, and it returns an error that the client can act on. The service keeps its capacity available for the arrivals it accepted, so the response time for accepted work stays near the unloaded value.

A bound also makes the overload observable in a way the unbounded queue is not. Queue depth reaching its maximum is a direct measurement of saturation, it appears in the metric system as a ratio against a constant rather than as an unbounded number, and it fires before the process is out of memory.

The equivalence to network flow control is exact. A pipe holds a fixed kernel buffer; a writer that fills it blocks, and the blocking propagates to whatever is generating the data. TCP advertises a receive window so that the sender knows how much may be in flight, and it reduces its congestion window on loss. In each case the receiver declines to accept more than it can handle, and the sender slows down as a consequence rather than as an act of cooperation.

Blocking versus rejecting

The refusal can take two forms, and they have different consequences for the caller.

Blocking makes the producer wait until space is available. This is what a full pipe does to a writer, and it is what a bounded blocking queue does to a thread. Throughput upstream is constrained to the rate downstream can absorb, which is correct behaviour for a pipeline within one process: the stages stay in lockstep and no data is lost. The hazard is deadlock when the producer and the consumer share a thread pool, because the producer occupying a worker while blocked on a queue can prevent the consumer from ever running. Blocking is safe when producer and consumer execute on separate workers and unsafe when they do not, and the distinction is invisible at the queue’s call site.

Rejecting returns an error to the producer, which must decide whether to retry, drop, or propagate the error to its own caller. This is the correct form at a service boundary, because the caller has an alternative: it can serve degraded content, queue the request durably in a system sized for storage rather than memory, or tell its user that the system is busy. The refusal also terminates the work, so the resource cost of an overloaded request is the cost of refusing it, which is close to nothing.

The choice between the two is dictated by whether the caller can do anything useful in response. A caller inside the process that could retry in a moment should block; a client that must produce a response to a user should be told immediately.

// A bounded worker pool: arrivals beyond capacity are refused, not queued.
func (s *Server) Submit(job Job) error {
    select {
    case s.jobs <- job:
        return nil
    case <-s.shutdown:
        return ErrShuttingDown
    default:
        atomic.AddUint64(&s.rejected, 1)
        return ErrOverloaded // caller decides: retry, degrade, or fail
    }
}

The default arm is what makes this a bound. Without it the send blocks, which is the blocking policy; with it the submit is non-blocking and the caller keeps control.

Which item to drop

Once refusal is possible, the question becomes what to refuse. The simplest policy rejects new arrivals once full, which protects the resident work and biases service toward older requests. This is right when requests are homogeneous and arrival order approximates value.

It is wrong when the queue holds items whose value decays. A batch of sensor readings, a metrics update, or a progress report is worthless once superseded. For those, rejecting the newest wastes the queue space on stale work, and dropping the oldest is better: the resident set stays current and the drop is not a failure but a reload of information that was superseded anyway.

A third policy, applied in many network stacks and in some streaming systems, drops at random from the queue rather than at an end. The purpose is not to keep the most valuable items but to avoid a synchronized drop pattern where several producers all reduce their rate at once and then all increase it together. Random dropping spreads the signal, and its use in network congestion control is where the technique was developed.

Policy on full	Preserves	Suits
Reject new arrivals	Oldest work, arrival order	Homogeneous requests with equal value
Drop oldest resident	Currency of data	State updates, telemetry, progress
Drop random	Producer synchrony	Many independent producers sharing a link
Coalesce	Per-key latest value	Keyed state where only the newest matters

Coalescing deserves mention as a separate category because it is not a drop policy at all. If the queue is keyed and only the latest value per key matters, then a full queue can overwrite an existing entry for the same key rather than refusing the new one. The queue never grows beyond the number of distinct keys, which is a natural bound, and the work performed is proportional to distinct keys rather than to arrivals. Systems that mix coalescing with a bound get both properties: a bound that is a statement about keys, and a refusal that occurs only when the key space exceeds it.

Where the bound belongs

A bound on one component of a pipeline does not bound the pipeline, because the refusal has to be handled by something upstream that is itself willing to stop. If a service has a bounded queue but its load balancer retries refused requests immediately and indefinitely, the bound has done nothing except multiply the traffic. Backpressure is only effective when each stage propagates its refusal to the stage before it, and the propagation terminates only where a human or a client is prepared to receive the answer.

The retry interaction is the most common way a bound is defeated. A client that retries on refusal turns one refused request into a sequence of attempts, each of which occupies a connection and a slot in the receiver’s accept path. The amplified load arrives at a service that is by construction already at its limit. The mitigations are all forms of making retries cheap for the service and expensive for the client: refusing at the connection level so retries cost a handshake, returning a retry-after indication, and bounding the client’s retry budget rather than the server’s.

Local bounds also do not compose across replicas. Ten replicas each with a queue of a thousand accept ten thousand queued items in total, none of which is a global limit. Load balancing is what determines whether one replica receives most of them, and an unbalanced distribution saturates one queue while others are idle. The bound must be stated together with the balancing policy to be meaningful.

The interaction between the bound and the queueing delay of a saturated stage is the subject of Throughput, Latency, and Why Concurrency Is Not Parallelism, where the latency curve explains why a bound of even a few hundred items is already too long near saturation. When the queue in question distributes work rather than buffering it, the equivalent design question is the granularity and stealing policy in Work Stealing and Task Granularity.