From One Page to a Controlled Crawl

Scrapling's spider framework answers the coordination problem that appears when one Response can create many more requests. This chapter traces a Request through the scheduler, crawler engine, session manager, callback, and output, then explains how domain scope and declarative templates turn traversal into a policy that can be inspected rather than an accidental side effect of link-following. The useful result is a crawl whose identity, concurrency, and boundaries have clear owners.

A request becomes work with an identity and a destination

A one-page fetch can leave most coordination in the caller's hands. A crawl needs a system to decide what happens next. Scrapling's spider model starts with a class that has a name, one or more starting URLs, and an asynchronous parse callback. The callback receives a Response and can yield a dictionary as a scraped item or a Request as future work. That small contract gives the engine two distinct outputs to manage: information already obtained and instructions to obtain more.

The first request normally comes from the spider's starting URLs. The Scheduler receives it and assigns it a fingerprint. In the current design, the fingerprint accounts for the URL, HTTP method, body, and session identity, with options for including additional request arguments or headers. The fingerprint is not a cosmetic identifier. It defines what the engine considers the same piece of work. If a request with the same fingerprint has already been seen, the scheduler drops it unless the request explicitly bypasses filtering.

This is the first important difference between a controlled crawl and a loop that keeps following links. URLs alone do not always describe equivalent work. The same URL requested with a different method, body, or session can produce a different observation. Scrapling's fingerprint model preserves those distinctions when configured to include them, while the default deduplication prevents ordinary repeated work. A retry path may deliberately bypass deduplication because it represents a new attempt under a changed recovery policy, not a new discovery of the same page.

Identity can be examined at two levels. The URL and its canonical form determine which destination the crawl believes it has discovered; method, body, and session conditions determine what kind of observation it is asking for. A product detail URL requested through a normal HTTP session is not necessarily equivalent to the same URL requested through a browser session with different cookies. A form submission is not equivalent to a GET of the page that contains the form. If those distinctions are collapsed, the scheduler may discard work that is genuinely different. If every incidental argument is treated as unique, the crawl may repeat work that should have converged. Fingerprinting is therefore a policy about meaning, not merely a performance trick.

The Scheduler is also a priority queue. Higher-priority requests are dequeued first, so the crawl can prefer some work without changing the callback contract. The engine does not drain every discovered URL into active tasks at once. It checks the configured global concurrency limit before spawning more work, then applies an optional per-domain limiter as well.