When Is Enough Enough? Adaptive Crawling and the Cost of More Pages

Crawl4AI’s adaptive crawler asks a question that BFS, DFS, and best-first traversal cannot answer on their own: when has the collected material become sufficient for a particular query, so that another page is no longer worth its cost? This chapter follows the state, scores, link choices, and stopping conditions behind the project’s statistical and embedding strategies. The practical judgment it supports is cautious interpretation: use adaptive crawling to turn information need into a bounded work policy, but do not mistake its confidence score for a probability that an answer is complete or true.

A fixed page limit is simple, but it is blind to information value. Ten pages can be excessive for a narrow question and insufficient for a broad one. A traversal order can choose which page comes next, but it does not know whether the current pages repeat one another or leave a major concept uncovered. Adaptive crawling adds feedback from the growing knowledge base so that selection and stopping depend on what the crawl has learned so far.

The adaptive loop

Crawl4AI stores more than a list of visited URLs. Its CrawlState includes the query, crawled URLs, pending links, collected results, term statistics, crawl order, metrics, and—when using the embedding strategy—query and knowledge-base embeddings. That state is what allows each round to influence the next one. The crawler first reaches a starting page and collects candidate links, then repeatedly measures the current knowledge, ranks pending links, crawls a selected batch, and updates the state.

stateDiagram-v2 state "Initial page and pending links" as INITIAL state "Update knowledge state" as UPDATE state "Calculate statistical or embedding confidence" as CONF state "Rank and select links" as RANK state "Crawl selected batch" as CRAWL state "Persist state when configured" as SAVE state "Stop on threshold, limit, no links, or no gain" as STOP [*] --> INITIAL INITIAL --> UPDATE UPDATE --> CONF CONF --> STOP: stopping condition met CONF --> RANK: more expected value remains RANK --> CRAWL CRAWL --> SAVE: persistence enabled CRAWL --> UPDATE: next iteration SAVE --> UPDATE: resume state is current STOP --> [*]

The loop changes the meaning of a page limit. max_pages remains a hard resource boundary, but it is no longer the only reason to stop. The crawler can stop because its configured confidence threshold is met, because no links remain, because the expected gain is too low, or because its strategy detects a saturation or convergence condition. The page count protects the machine; the information signals shape the search.

What the statistical strategy measures

The default statistical strategy does not call an LLM or load an embedding model. It represents the query and crawled Markdown as tokens, tracks how often terms occur, and records how many documents contain them. Its coverage score asks whether query terms appear across the knowledge base and whether their frequency provides a stronger signal. A term present in several documents contributes more document coverage than a term found nowhere; a frequency boost distinguishes a single incidental appearance from repeated use.

The implementation averages the term scores and applies a square-root curve before returning a value between zero and one. This produces an internal measure of how much of the query’s vocabulary is represented in the collected pages. It is useful for exact terminology. A query about a named configuration parameter may be covered when that parameter appears in several relevant documents. A conceptual query expressed through synonyms may look under-covered even when the pages explain the idea well.

The second signal is called consistency, but its implementation needs to be read precisely. It computes pairwise Jaccard overlap between the sets of terms in collected documents.