Tag
politeness
- Mar 26, 2025 The URL Frontier: Priority Queues Behind Per-Host Politeness How a crawler satisfies two conflicting orderings at once, using a randomized priority front end and a back end of per-host queues keyed by ready time. information retrieval
- Dec 13, 2024 Adaptive Per-Host Rate Control from Latency and Error Signals How a crawler can raise and lower its request rate per host using response latency and rejection codes, and why the control loop is asymmetric and needs damping. information retrieval
- May 8, 2024 Per-Host Request Rate Limits and the Cost of a Fixed Delay Why a constant delay between requests is the simplest defensible politeness policy, how concurrency multiplies the rate a server actually sees, and where the model fails. information retrieval
- Jan 30, 2022 Matching Paths in robots.txt: Longest Rule, Not First Rule How a robots.txt file is split into groups, why the winning rule is chosen by octet count rather than by order, and what the protocol requires when the file cannot be fetched. information retrieval