Tag
crawl-frontier
- Mar 26, 2025 The URL Frontier: Priority Queues Behind Per-Host Politeness How a crawler satisfies two conflicting orderings at once, using a randomized priority front end and a back end of per-host queues keyed by ready time. information retrieval
- Aug 8, 2023 Partitioning a Crawl Across Machines Without Duplicate Fetches Why host-based partitioning is the only split that preserves politeness, how a frontier survives reassignment, and what happens when one partition dies. information retrieval