Conditional GET and the Revalidation Economy

A crawler revisiting a page it already holds faces a bandwidth problem with an obvious shape: most of what it fetches will be unchanged. The mechanism for avoiding the transfer is older than the modern web and lives in the HTTP specification itself. The crawler sends a request that carries evidence of what it already has, the server compares that evidence against the current representation, and if they match the server answers with a status line and no body.

The saving is real but narrower than it first appears. A 304 response saves the payload; it does not save the round trip, the request, or the server’s work in deciding. For a small page, the headers and the latency dominate, and conditional GET is worth far less than the byte count suggests. Understanding where the saving actually lands is what determines whether a crawl should revalidate aggressively or simply refetch on a schedule.

Two validators, two different claims

Last-Modified is a timestamp: the server’s statement about when the representation last changed. Its resolution is one second, which is the granularity of the HTTP-date format, so two changes inside the same second are indistinguishable. It also requires the server to know the modification time accurately, and a representation generated on the fly has no natural one — implementations often substitute the request time, which makes the validator useless.

ETag is an opaque string chosen by the server. Its form is unconstrained apart from quoting, and its meaning is whatever the server decides. The two common strategies are a hash of the content, which makes the tag change exactly when the bytes change, and a version counter from the database, which makes it change when the record is written even if the rendered bytes are identical.

The distinction between strong and weak validators is a claim about byte equality. A strong validator, written "abc123", asserts that two representations with the same tag are byte-for-byte identical. A weak validator, written W/"abc123", asserts only semantic equivalence: the representation may differ in ways that do not matter for the resource’s meaning, such as whitespace, ordering of equivalent fields, or a cache-busting comment.

The crawler cares about the difference in one specific way. If the goal is to detect any change at all, a strong validator is the better signal and a weak one will hide byte-level differences. If the goal is to avoid storing near-duplicates, the weak validator is the better signal and a strong one will report change for cosmetic differences. Choosing between them is choosing what counts as a change, and there is no validator that answers both questions.

A response may carry both headers. When it does, the ETag takes precedence for If-None-Match, and the date is a fallback for clients that only understand the older mechanism. Servers that emit both should keep them consistent; a server that updates the ETag on every request while leaving Last-Modified frozen produces a validator pair that reports change to one client and stability to another.

The request and the answer

The conditional form of a GET puts the validator in a request header. If-None-Match carries one or more entity tags, comma separated, and the server compares them against the current representation’s tag using the weak comparison function — a match on the opaque string, ignoring the W/ prefix, because the question asked is “is this the same representation” and not “is this the same bytes”. If-Modified-Since carries a date, and the server answers 304 if the representation has not been modified since that instant.

The response to a match is 304 Not Modified, which carries the headers that would have been sent with a 200 but no body. The crawler must treat the cached headers as replaced: a 304 can carry a new ETag, a new Cache-Control, or a new Last-Modified, and ignoring those updates leaves the crawler holding stale metadata about a resource whose content is unchanged. The body is not sent, and the crawler’s stored copy is still current.

The response to a mismatch is the full 200 with a new body, and the crawler replaces its stored copy along with both validators.

import httpx

def revalidate(client, url, stored):
    headers = {}
    if stored.etag:
        headers["If-None-Match"] = stored.etag
    elif stored.last_modified:
        headers["If-Modified-Since"] = stored.last_modified
    r = client.get(url, headers=headers)
    if r.status_code == 304:
        stored.headers.update(r.headers)   # validators may have rotated
        stored.last_fetch = r.headers["Date"]
        return stored
    r.raise_for_status()
    stored.body = r.content
    stored.etag = r.headers.get("ETag")
    stored.last_modified = r.headers.get("Last-Modified")
    stored.last_fetch = r.headers["Date"]
    return stored

Two details in that sketch carry most of the correctness. The elif matters because a request carrying both conditionals is evaluated by the server against both, and the interaction rules are subtle enough that sending one is easier to reason about. The header merge matters because a 304 is a full response for metadata purposes even though it has no body.

What a 304 actually costs

The byte saving is the payload size, which for an HTML page is typically in the range of tens of kilobytes and for a large asset can be megabytes. Against that, the request still travels, the server still resolves the route, reads whatever it needs to compute the validator, and returns headers that are often a few hundred bytes.

For a crawl of a site whose pages average twenty kilobytes and whose headers average four hundred bytes, a 304 saves roughly ninety-eight percent of the response bytes. That ratio is the honest way to state the benefit, and it is why revalidation is worth doing for content pages. It is also why the benefit is much smaller for a site serving tiny responses, where headers are a meaningful fraction of the transfer, and why the benefit does not extend to request-side costs at all: the crawler still spends one politeness slot per revalidation.

The cost that is easy to miss is the server’s. Computing an ETag from content requires rendering or hashing the representation, which on a dynamic site may be the expensive part of the request. A crawler that revalidates every URL daily against a site that renders each page from a database pays the rendering cost every day and gets a 304 back. From the site’s perspective, a conditional request is not free; it is a full request whose response happens to be short. The politeness budget consumed is identical to an unconditional fetch, which is the argument for scheduling revalidations by expected change rate rather than by convenience.

When the validator is not a freshness signal

A 304 says the representation is unchanged since the validator was issued. It says nothing about how long the crawler should wait before asking again. Those are different questions and the headers answer them separately: Cache-Control and Expires describe freshness lifetime, which is a hint about how long the representation may be reused without revalidation. A page with max-age=86400 is telling caches not to bother asking for a day; a page with no-cache is telling them to revalidate before every reuse.

Crawlers commonly ignore freshness lifetime and revalidate on their own schedule, because the crawl’s purpose is to observe change, not to serve a user quickly. That is defensible, but it means the crawler is not acting as a cache and should not claim the cache’s exemptions. A no-store directive, by contrast, is a prohibition on retaining the representation at all, and honoring it means the crawl cannot index that resource — a case where the site’s cache policy directly constrains what the index can contain.

Three failure modes recur:

• Validator churn. A server that emits a new ETag on every request, whether from a per-request timestamp or a random value, makes every revalidation a full fetch. The crawler sees a 200 every time and concludes the page changed every time, which corrupts any change-rate estimate built from those observations.
• Last-Modified in the future or absent. A date ahead of the response’s Date header is not a valid validator, and a server that omits Last-Modified for generated content leaves If-Modified-Since unusable. Both cases push the crawler back to ETag or to unconditional fetching.
• Clock skew and second granularity. A Last-Modified equal to the current second cannot distinguish a change made within that second, and a client whose clock is ahead produces an If-Modified-Since in the future, which some servers interpret as “always modified”. Using the server’s Date header as the reference rather than the local clock removes the second problem.

There is a deeper issue with Last-Modified as a change signal: a page can be modified without its content changing, and content can change without the timestamp moving, if the modification path does not touch it. A CMS that rewrites a template updates the timestamp on every page while changing none of their text. A change-detection scheme built on that header will schedule refetches of pages that never change and will never converge.

Revalidation inside a scheduling policy

The economics point to a specific structure. Every URL the crawler holds has a last-fetch time, a validator, and an estimated change rate. The scheduler’s job is to allocate the next fetch slot to the URL with the highest expected value per unit of politeness budget. Revalidation enters that calculation as the cheaper of two actions: it produces a definite answer about change for the cost of one request, whereas an unconditional fetch produces the same answer plus bytes the crawler may not need.

That framing makes conditional GET a scheduling primitive rather than a bandwidth trick. A page with a high change rate should be refetched unconditionally, because the probability of a 304 is low and the body will be needed anyway. A page with a low change rate should be revalidated, because the probability of a 304 is high and the body is usually wasted. A page whose validator churns should be scheduled by estimated change rate alone, since revalidation tells the crawler nothing.

The estimates that drive those choices come from observed history — how often the resource actually changed when it was fetched — and building them from noisy observations is the next problem. A single observation that a page changed after a week does not establish a weekly change rate, and treating it as one produces a scheduler that oscillates between extremes. The estimation problem is where freshness scheduling becomes statistics rather than bookkeeping.