Tag
crawl-budget
- Feb 26, 2026 Crawl Order as Resource Allocation Under a Fixed Budget How to decide which of the ten million known URLs to fetch first, why importance and freshness compete for the same request, and what ordering rules can and cannot promise. information retrieval
- Mar 28, 2024 Detecting Crawler Traps by Budget per Host How infinite URL spaces, calendar generators, and session identifiers make a crawl diverge, and the per-host accounting that stops it without losing real pages. information retrieval