Information Retrieval · Cluster
Crawling
Scheduling, politeness, and freshness.
- 01 The URL Frontier: Priority Queues Behind Per-Host Politeness How a crawler satisfies two conflicting orderings at once, using a randomized priority front end and a back end of per-host queues keyed by ready time.
- 02 Matching Paths in robots.txt: Longest Rule, Not First Rule How a robots.txt file is split into groups, why the winning rule is chosen by octet count rather than by order, and what the protocol requires when the file cannot be fetched.
- 03 Per-Host Request Rate Limits and the Cost of a Fixed Delay Why a constant delay between requests is the simplest defensible politeness policy, how concurrency multiplies the rate a server actually sees, and where the model fails.
- 04 URL Normalization as a Set-Equality Problem Which transformations are safe when deciding two URLs identify the same resource, why scheme and host case rules differ from path rules, and how over-normalization merges distinct pages.
- 05 Conditional GET and the Revalidation Economy How ETag and Last-Modified turn a refetch into a question, what 304 responses actually save, and why strong and weak validators answer different questions.
- 06 Estimating Change Rates from Interval-Censored Fetches Why a page observed to change once in ten fetches does not have a ten percent change rate, and how a Poisson model with a Gamma prior turns sparse observations into a revisit interval.
- 07 Detecting Crawler Traps by Budget per Host How infinite URL spaces, calendar generators, and session identifiers make a crawl diverge, and the per-host accounting that stops it without losing real pages.
- 08 Adaptive Per-Host Rate Control from Latency and Error Signals How a crawler can raise and lower its request rate per host using response latency and rejection codes, and why the control loop is asymmetric and needs damping.
- 09 Partitioning a Crawl Across Machines Without Duplicate Fetches Why host-based partitioning is the only split that preserves politeness, how a frontier survives reassignment, and what happens when one partition dies.
- 10 Crawl Order as Resource Allocation Under a Fixed Budget How to decide which of the ten million known URLs to fetch first, why importance and freshness compete for the same request, and what ordering rules can and cannot promise.