Per-Host Request Rate Limits and the Cost of a Fixed Delay

Politeness has no standard unit. The protocol that defines what a crawler may request says nothing about how often it may ask, and the directives that sites use to express a rate preference are conventions outside that protocol. What remains is a design decision made by each crawler, and the decision has a property that makes it easy to get wrong: the quantity a server experiences is not the quantity the crawler configured.

What the server actually counts

A crawler setting of “one request per second per host” sounds like a rate. It is a rate only if at most one request to that host is outstanding at a time. With four workers each holding their own timer for the same host, the four timers start together and the server sees four requests arrive within the same instant, then silence until the next interval. The mean rate over a long window is unchanged; the instantaneous rate in that instant is four times the configured value.

The distinction matters because server-side protection usually acts on the instantaneous rate, not the mean. Rate limiters in reverse proxies and application servers typically bucket by short windows, tens of seconds or less, and a burst of four requests inside one second looks like a burst regardless of the quiet period that follows. A crawler that believes it is polite because its average matches the target can therefore trigger 429 responses while conforming to its own arithmetic.

The usable rule is that politeness applies to request starts, not request completions, and that concurrency to a single host must be one unless the crawler has evidence the server can absorb more. That second clause exists because latency makes strict serialization expensive: if a host takes two seconds to respond, a strict serial crawler fetches at most one page every two seconds, which is half the configured rate and a hard ceiling on what the crawl can achieve against that host.

There is a third quantity, slots, that sits between the two: the number of requests the crawler is willing to have in flight globally, summed across all hosts. A global limit on concurrency protects the crawler’s own resources (sockets, file descriptors, memory for response bodies) while per-host delay protects the servers. Confusing the two limits produces a crawler that is gentle to each host and merciless to its own network stack, or the reverse.

Why a fixed delay is the default defensible choice

A constant interval between successive request starts to one host has three properties that recommend it. It is trivially auditable: an operator can state the policy in one number. It bounds the request rate independent of response latency, so a slow host is not punished with a higher apparent rate. And it degrades predictably: the worst case for the server is a known number of requests per hour, computable without knowing anything about the crawler’s internal scheduling.

The cost is throughput. With a one-second interval, a single crawler reaches at most 3,600 pages per hour per host, and no amount of parallelism inside the crawler changes that without violating the interval. For a site with a million pages, the lower bound on crawl duration is about 278 hours, or eleven and a half days, before any allowance for failures, re-fetches, or discovery of new URLs. That arithmetic is why large crawlers work on many hosts at once rather than on one site deeply.

The interval itself is a judgement call with no correct answer. Published discussion of the original exclusion protocol suggested a figure on the order of tens of seconds as a minimum gap for an automated client, and that order of magnitude survives in the Crawl-delay convention, where sites commonly write values between one and thirty seconds. A value of zero means no imposed delay and is not a licence for unbounded concurrency.

The delay can be measured in two ways, and the choice changes the effective rate. Measuring from the start of one request to the start of the next caps the rate at 1/interval regardless of how long the server takes, because the period is the larger of the interval and the response latency. Measuring from the end of a response to the start of the next makes the period latency plus interval, so the rate is always lower and falls further on slow hosts. The second is the safer of the two for the server and costs throughput in proportion to latency; the first keeps the crawl rate predictable. Stating which one a crawler implements is more useful than the number itself, because the same number means different loads under the two readings.

Reading the rate the server asks for

Three signals carry a server’s own view of acceptable load, and they differ in authority.

Retry-After in a 429 or 503 response is the most specific and the most authoritative: it is a direct instruction, expressed either as a number of seconds or as an HTTP-date, and it applies to the resource or the host that issued it. Treating it as advisory is a mistake that reliably produces escalation. A crawler receiving Retry-After: 120 should exclude the host from scheduling for two minutes, and should not resume at the previous rate afterwards without evidence that the condition has cleared.

The Crawl-delay convention is weaker in every respect: it appears in robots.txt, it is not standardized, it is expressed in seconds, and its interaction with a specific crawler’s concurrency is undefined. Honoring it is a courtesy. Merging it into the effective interval — taking the larger of the configured delay and the declared one — is the interpretation that respects intent without letting a site’s file throttle a crawler below its own floor for other hosts.

Response latency and error patterns are the third signal and the only one that requires inference. A host whose median response time climbs while the crawler’s rate is unchanged is telling the crawler something, and a rising share of 5xx responses says the same thing more loudly. This is the input to adaptive rate control, which is a distinct mechanism with its own stability problems; the point here is that a fixed delay has no way to use the signal, and therefore treats a degrading host exactly like a healthy one until the responses turn into explicit rejections.

The three signals can disagree. A site may declare a delay of sixty seconds while answering in ten milliseconds, and a crawler may be configured for one second while receiving 429s at that rate. The resolution that avoids harm in both directions is to take the most restrictive applicable constraint at each moment and to let the inferred signal push the rate down faster than it allows it to recover.

Concurrency, head-of-line blocking, and the queue that keeps its promise

The per-host queue described in the frontier design enforces the interval by carrying a timestamp. The queue’s next request may start only after that timestamp. A one-second delay against a host that answers in fifty milliseconds leaves the connection idle about 95 percent of the time by design, and the frontier can use the idle capacity by assigning other hosts to other connections.

The failure mode is head-of-line blocking inside the queue. If the first URL in a host’s queue points at a resource that takes thirty seconds to respond or times out, every subsequent URL for that host waits behind it, and the host’s effective rate collapses to well below the configured value. Serial per-host politeness makes the slowest resource in the queue the gate for all the others. Bounding per-request timeouts — connecting, sending, and reading the body — is what keeps one pathological URL from stalling a host’s entire scheduling.

A second failure is the retry loop. A crawler that retries a failed request immediately, and counts each retry as a fresh request, can issue several requests inside one interval to a host that is already failing. Retries must consume the same politeness budget as first attempts, which means the retry goes to the back of the same host queue with the same timestamp advance. A retry that bypasses the queue is how a crawler turns a transient server error into a sustained overload.

Politeness also has an address-level subtlety that per-host bookkeeping hides. Hosts behind shared infrastructure share the machine that receives the requests even though they have distinct names. Two authorities on the same server, or a site and its CDN origin, can be contacted at twice the intended rate through two independent host queues. Nothing in DNS or in robots.txt reveals this. The only general mitigation is to keep the global request rate bounded and the per-host rate conservative enough that a small multiplier stays within what a shared server tolerates.

Where the fixed delay is the wrong model

A constant interval assumes the server’s capacity is constant. Real servers have diurnal load, deploy windows, and cache warmth cycles, and a crawler that could shift its deep crawls into quiet hours would be strictly better for the site and no worse for itself. Fixed delay cannot express that; it can only be made smaller or larger globally.

The inverse assumption — that a site’s capacity scales with its size — is also false. A small site on modest hosting and a large site on a CDN may both answer quickly at one second intervals, but only one of them is at risk from a partially misconfigured crawler. Since the crawler cannot measure capacity directly, and since the cost of being too aggressive is a site outage while the cost of being too gentle is a longer crawl, the asymmetry argues for a conservative fixed floor plus a mechanism that only lowers the rate in response to evidence. Constructing that mechanism requires estimating how fast a site changes and how much of it the crawler can afford to revisit, which is the subject of the freshness models.