When the Web Pushes Back: Pacing, Recovery, and Continuity
Scrapling's crawl controls are designed for a web that changes its behavior while the crawl is running. This chapter follows a request through configured limits, robots rules, AutoThrottle, blocked-response recovery, proxy or session changes, and interruption; it also separates production continuity from development replay and live streaming. The goal is not to make every request continue, but to make each continuation, delay, or stop an observable and bounded decision.
Pacing is a feedback loop, not a magic number
A fixed download delay is easy to understand: wait the same amount of time before each request. It is also a guess. If it is too short for a slow or sensitive target, the crawl may create unnecessary pressure. If it is too long for a responsive target, the crawl wastes time without learning anything from the site's actual behavior. Fixed concurrency has a similar problem when it is chosen without regard to the domain receiving the requests.
Scrapling offers global and per-domain concurrency controls, a fixed delay, optional robots.txt compliance, and AutoThrottle. These controls answer different questions. The global limit bounds how much work the process attempts at once. The per-domain limit prevents one target from consuming all of that budget. A fixed delay supplies an operator-chosen floor. Robots directives can raise that floor when the target asks for a crawl delay or request rate. AutoThrottle then adjusts the delay for each domain in response to observed behavior.
When robots compliance is enabled, the spider fetches and caches robots.txt information for its domains, checks requests against disallow rules, and respects Crawl-delay and Request-rate directives. A disallowed request is dropped rather than sent to the session. The delay calculation takes the maximum of the configured delay and the applicable robots-derived delay, so enabling robots support does not silently make an existing pacing policy more aggressive. Robots compliance is an explicit option, however; it is off by default in the spider framework, which means the operator must choose it rather than assume that the default represents a target's rules.
AutoThrottle uses the time a request actually takes as feedback. Its target is the observed latency adjusted for the desired number of requests in flight for that domain. The new delay moves toward that target rather than jumping to every individual measurement, while a latency spike can cause an immediate increase. The result is a per-domain pace that can settle near the response behavior of the site instead of treating every domain as identical.
That feedback is based on the full fetch. It includes internal request retries and, for browser sessions, the time needed to render the page. This is an important condition. A browser page that takes several seconds to render is not equivalent to a static response that arrives quickly, even if both ultimately expose a title and a price. A throttle that used only network-connect time would understate the pressure placed on the target and the resource occupancy inside the crawler.
The throttle also distinguishes a healthy response from a response that merely arrived quickly. A rate-limit page or a challenge response may be faster than the real content. If the response is not a healthy 2xx result, or if the spider's isblocked() hook identifies it as blocked, the default block-backoff behavior prevents the fast error from accelerating the crawl.