Tag
web-crawling
- Aug 18, 2026 URL Normalization as a Set-Equality Problem Which transformations are safe when deciding two URLs identify the same resource, why scheme and host case rules differ from path rules, and how over-normalization merges distinct pages. information retrieval
- Feb 26, 2026 Crawl Order as Resource Allocation Under a Fixed Budget How to decide which of the ten million known URLs to fetch first, why importance and freshness compete for the same request, and what ordering rules can and cannot promise. information retrieval
- Mar 26, 2025 The URL Frontier: Priority Queues Behind Per-Host Politeness How a crawler satisfies two conflicting orderings at once, using a randomized priority front end and a back end of per-host queues keyed by ready time. information retrieval
- Dec 13, 2024 Adaptive Per-Host Rate Control from Latency and Error Signals How a crawler can raise and lower its request rate per host using response latency and rejection codes, and why the control loop is asymmetric and needs damping. information retrieval
- May 8, 2024 Per-Host Request Rate Limits and the Cost of a Fixed Delay Why a constant delay between requests is the simplest defensible politeness policy, how concurrency multiplies the rate a server actually sees, and where the model fails. information retrieval
- Mar 28, 2024 Detecting Crawler Traps by Budget per Host How infinite URL spaces, calendar generators, and session identifiers make a crawl diverge, and the per-host accounting that stops it without losing real pages. information retrieval
- Jan 30, 2022 Matching Paths in robots.txt: Longest Rule, Not First Rule How a robots.txt file is split into groups, why the winning rule is chosen by octet count rather than by order, and what the protocol requires when the file cannot be fetched. information retrieval