Matching Paths in robots.txt: Longest Rule, Not First Rule

The file format looks like a configuration language and behaves like a precedence-ordered pattern matcher, and the difference matters. A crawler reading Disallow: /private followed by Allow: /private/public is not reading a sequence of instructions applied in order; it is collecting rules and then asking, for each URL, which single rule is the most specific. Implementations that treat the file as first-match-wins will fetch pages the site meant to protect, or refuse pages the site meant to expose.

One file per authority, at one fixed path

The protocol defines exactly one location: the path /robots.txt on the host that serves the page. Scheme and port are part of the identity. For a URL fetched over HTTPS on the default port, the file is https://host/robots.txt. A site served on a non-default port has its own file at that port; the bytes served on port 80 and port 8080 are separate documents governing separate sets of resources, even when the same machine answers both.

The path is case-sensitive and has no query string. A server that responds to /Robots.txt or to /robots.txt?v=2 is not implementing the protocol; those responses are irrelevant to it.

Because the file is per authority rather than per hostname, a crawler with a queue of URLs from ten thousand authorities must fetch up to ten thousand separate files before it can decide anything. That cost is why the result is cached and why the cache lifetime matters: there is no standard freshness signal beyond ordinary HTTP caching headers, and implementations choose their own re-fetch interval, commonly in the range of hours to a day. The frontier design that holds those URLs is described in the URL frontier, and the fetch of the file itself consumes a slot in the same per-host queue as any other request. A crawler that revalidates the file with a conditional request pays a round trip per authority per interval; one that re-fetches unconditionally pays for the bytes as well. The mechanics of that choice are the same as for any other resource, which is why the economics of revalidation applies to robots.txt as much as to content pages.

The protocol also permits the file to be reached through redirects, and requires crawlers to follow at least five consecutive ones, including redirects that cross from one authority to another. Following a redirect means the rules that apply come from wherever the chain ends, not from the file at the original path — a cross-authority redirect therefore lets one site’s rules govern another’s URLs, which is a real deployment hazard for shared hosting and for sites that moved domains.

Parsing into groups, and the records that mean nothing

The file is line-oriented. Blank lines and lines beginning with # are ignored, with the # starting a comment that runs to the end of the line. Each remaining line is a field: value pair, where the field name is matched case-insensitively.

A user-agent line starts a new group. Every allow or disallow line after it belongs to that group until the next user-agent line appears. Consecutive user-agent lines with no rules between them share the group that follows, which is how a site writes a group that applies to several crawlers at once:

User-agent: ExampleBot
User-agent: OtherBot
Disallow: /tmp/
Allow: /tmp/public/

Rules that appear before any user-agent line have no owner and are discarded. A group with no rules at all permits everything, and a disallow with an empty value is a no-op rather than a prohibition — the natural reading, “disallow nothing”, matches the specification’s intent, and implementations that treat an empty value as “block the entire site” produce the opposite of the file’s meaning.

Field names the protocol does not define are ignored. That rule is what keeps the format extensible: a site can write Crawl-delay: 10 or Sitemap: https://example.com/sitemap.xml into the same file, and a crawler that does not implement those extensions is required to skip them rather than fail to parse. Neither directive is part of the standardized protocol, and both are widely honored by convention rather than by specification. A crawler that ignores Crawl-delay is conforming; one that honors it is being generous to the server.

Group selection follows a similar logic of specificity. The crawler looks for groups whose user-agent line names it, and the comparison is case-insensitive. If several groups match, their rules are merged into one group before matching begins. If no group names the crawler, the group whose user-agent is * applies. If neither exists, everything is permitted. The order of groups in the file carries no precedence — a group for a specific crawler always beats the wildcard group, regardless of position, because group selection is a separate step from rule selection and happens first.

The winning rule is the longest path, not the earliest line

Within the selected group, each rule is a path pattern. A pattern without wildcards matches any URL path that begins with it, so /private matches /private, /private/, /private/a/b, and also /privateers, which surprises people who expect path-segment semantics. Adding a trailing slash changes the match set to exclude the extension-like case, which is why well-written files use /private/.

Two metacharacters extend the pattern. * matches any sequence of characters, including the empty sequence, and $ at the end of the pattern anchors the match at the end of the path. So /search$ matches /search but not /search/results, and /*.pdf$ matches any path ending in .pdf.

Selection among all matching rules is by length: the rule whose pattern has the most octets wins. Only if two matching patterns have equal length does the tie-break come into play, and the tie is resolved in favor of the less restrictive outcome, meaning an allow beats a disallow of the same specificity. Order in the file is irrelevant to both steps.

import re

def compile_pattern(pattern):
    """Translate a robots.txt pattern: '*' is any run, trailing '$' anchors."""
    anchored = pattern.endswith("$")
    body = pattern[:-1] if anchored else pattern
    regex = ".*".join(re.escape(part) for part in body.split("*"))
    return re.compile(regex + ("$" if anchored else ""), re.S)

def select_rule(rules, path):
    """rules: list of (allow: bool, pattern: str), already group-filtered."""
    best = None                     # (octet length, allow) of the winner
    for allow, pattern in rules:
        if compile_pattern(pattern).match(path):
            score = (len(pattern.encode("utf-8")), allow)
            if best is None or score > best:
                best = score
    return None if best is None else best[1]

The len(...) on the encoded bytes rather than on characters is not pedantry. Comparison happens on octets, and paths containing non-ASCII characters are percent-encoded before the comparison in the specification, so a pattern with two multi-byte characters can outrank a longer-looking ASCII pattern or vice versa. Implementations that count JavaScript string length will disagree with implementations that count octets on exactly the URLs where the answer matters.

The practical consequence of longest-match is that a site can carve exceptions out of a blanket prohibition by writing longer allow patterns, and it can broaden a prohibition by writing longer disallow patterns inside an allow region. Neither trick depends on line order, so a file edited by hand over years still resolves identically after rules are moved around, provided their lengths do not change.

Rules apply to paths, with an authority-shaped caveat

A disallow line constrains the path portion of the URL. Nothing in the pattern syntax refers to the host, so Disallow: /admin/ applies to https://example.com/admin/ and would apply equally to any other authority whose robots file contained the same line — but that other authority has its own file, so in practice the rule is scoped by which file it came from. A crawler that consults the file for authority A must not apply A’s rules to a URL on authority B, which sounds obvious until a redirect or a shared CDN enters the picture.

Query strings are part of the path being matched. Disallow: /search therefore also blocks /search?q=anything, because prefix matching does not stop at ?. A site that wants to block only parameterized search URLs must write the ? explicitly, and a site that wants to allow a specific query while blocking others depends on the longest-match rule to make the exception specific enough.

Rules also apply to the robots.txt file itself by the ordinary reading, and this produces a fixpoint problem: if the file disallows /robots.txt, a crawler follows the rules it already has, which may forbid re-reading the file. Implementations differ in how they handle that, and the specification’s guidance is thin. The safe position for a crawler is to fetch the file regardless of what the file says about itself, since the alternative is permanently freezing a policy that the site may have changed.

What the protocol requires when the file cannot be read

The response status is part of the protocol, not an operational detail. A 2xx response with a parsable body yields rules. A 4xx response, including the very common 404 for sites that never wrote a file, means the file does not exist, and the crawler may access any resource. A 5xx response, or a connection failure, means the rules are unknown, and the specification requires the crawler to assume complete disallow — treat the whole authority as off limits until the file can be read.

That asymmetry is deliberate. A site that has deliberately removed its file signals permission; a site whose server is failing has not signalled anything, and guessing “permit” would turn a transient outage into a licence to crawl a site that may have been protecting itself. The cost is that a crawler meeting a persistently broken authority stops crawling it entirely rather than proceeding cautiously, which is the intended trade: availability of the crawl is worth less than the site owner’s expressed intent.

Two more cases have assigned behaviour. If the file exceeds the size the crawler is willing to parse — the specification suggests parsing at least the first 500 kibibytes — the crawler may ignore the remainder, and a rule placed past that boundary has no effect. If the response body is not parsable as the format at all, implementations generally treat it as absent, though the specification’s language on this point is less prescriptive than for status codes.

The larger point is that robots.txt is a cooperative protocol with no enforcement mechanism. A crawler that ignores it cannot be prevented from doing so at the protocol layer; the file is a statement of preference that conforming crawlers honor. Every behaviour above — group merging, longest match, disallow on 5xx — is a convention that makes the preference machine-readable. The rate at which a conforming crawler then issues requests is a separate question from what it is permitted to request, and that separation is why the delay rules live outside this file.