Searching a Site Without Crawling Blindly
Crawl4AI’s site-level problem is no longer how to extract one page; it is how to decide which pages deserve the cost of browser rendering, content processing, and extraction. This chapter follows the project’s URL seeding, prefetch, filtering, and deep-crawl strategies, then compares breadth-first, depth-first, and best-first traversal. The practical judgment it supports is to choose a search policy by the kind of coverage or value required, while making every limit and omission explainable.
A page can contain hundreds of links, but a link is only a candidate. It may point to navigation, a duplicate view, a utility file, a different domain, or a page that has little to do with the question. Treating every link as an instruction to run the full browser pipeline wastes time and can make the resulting knowledge base less focused. Crawl4AI therefore separates candidate discovery from the more expensive work of turning a candidate into evidence.
Discovery is not crawling
Crawl4AI’s AsyncUrlSeeder can start with more than the links visible on a homepage. It can read sitemaps, follow sitemap indexes, query the Common Crawl index, and combine those sources. The seeder can filter URLs by pattern, test whether a URL responds, fetch only an initial portion of a document’s head, and extract metadata such as a title, description, links, and JSON-LD. When a query is supplied, it can score head contexts with BM25 and return candidates ordered by relevance.
This is valuable when the starting URL is a poor map of the domain. A documentation site may expose its useful pages through a sitemap but hide them several clicks behind a navigation tree. An archive may have thousands of historical URLs in Common Crawl even though only the newest pages appear in the live site. A site migration may require a broad inventory before any full-page extraction occurs.
The seeder also treats discovery as a bounded asynchronous operation. A producer finds URLs and places them in a bounded queue. Workers validate or inspect them concurrently, while a rate semaphore can limit the overall pace. The queue’s capacity creates backpressure: if validation cannot keep up, discovery does not continue growing an unbounded in-memory list. Per-domain caches can retain sitemap, live-check, and head results, reducing repeated network work when a discovery operation is rerun.
These features do not make the candidate set complete. Sitemaps can be missing or stale. Common Crawl reflects archived observations rather than the current site. A HEAD request or partial head fetch can fail even when a full browser request would work. BM25 over titles and metadata can prioritize a page that sounds relevant but contains little useful body content. Discovery is therefore a cheap evidence about where to look, not evidence that the page answers the question.
Prefetch provides a similar separation from the other direction. Given a page that has already been reached, Crawl4AI can short-circuit heavy processing and return the HTML plus quickly extracted internal and external links. The fast path skips Markdown generation, structured extraction, and media processing. That makes it suitable for a two-phase design: discover and rank first, then fully process a selected subset.
The trade-off is explicit. Prefetch accelerates link discovery because it does less work; it cannot provide the filtered Markdown or structured fields that the later phase would produce. If a pipeline stores prefetch output as if it were extracted knowledge, it has confused a search map with the content that map points toward.
The frontier is shaped before traversal
Deep crawling starts with a frontier: URLs that are candidates for the next crawl. Crawl4AI normalizes links before processing them, including resolving relative links and removing fragments that would otherwise create duplicate visits. The strategies validate that a candidate has an HTTP or HTTPS scheme, a network location, and a domain-like host. The start URL is treated differently from later links, because filtering the entry point could prevent a crawl from beginning at all.