Overview

Firecrawl Crawl treats a site as a graph whose frontier must be bounded before links can create work. A seed, path and domain rules, sitemap and depth limits, robots behavior, threat policy, and page caps govern admission. Kickoff can use sitemaps and indexed links, but every candidate is normalized, filtered, and locked before a page job enters the shared scrape pipeline. Workers extract links to expand the frontier, while redirects, denials, duplicates, failures, pagination, warnings, and webhooks preserve lifecycle meaning. Completion accounts for admitted work; it does not prove exhaustive site coverage.

Learning outcomes

Crawl follows only URLs admitted by scope, policy, normalization, and locking rules; a discovered link is not automatic work.
A successful page can expand the frontier, but its links may be denied without invalidating unrelated crawl work.
A completed Firecrawl crawl accounts for permitted jobs and can still be narrow, partial, or warning-bearing.

Keywords

FirecrawlCrawlFrontierRobotsSitemapsDeduplication

Crawl as Controlled Graph Traversal

Firecrawl’s Crawl endpoint answers a different question from a single-page Scrape: not “Can this URL become a document?” but “Which pages may enter this site-shaped job, in what order, and when is the group complete?” This chapter focuses on that frontier lifecycle. The useful judgment is to predict why a link will be followed or skipped, understand how each admitted page re-enters the ordinary scrape pipeline, and interpret a crawl’s final result without assuming that every discovered link must succeed.

A crawl begins by creating a frontier

A website is a graph of possible pages, but a production crawler cannot treat the whole graph as its job. Firecrawl starts with a seed URL and a set of boundaries: path patterns, domain and subdomain rules, sitemap policy, depth and discovery limits, robots behavior, delay, and a total page limit. Those options define the frontier—the set of URLs that may become page work—before the first discovered link is allowed to expand it.

The API controller validates the request and creates a crawl identity. It stores the origin URL, crawler options, scrape options, team information, retention behavior, and the selected queue backend as a StoredCrawl. It also creates a group for the crawl and enqueues a kickoff job. The group is important because a crawl is not one request that happens to run for a long time. It is a durable unit whose members will be page jobs with separate states, while the group carries the information needed to interpret them together.

Kickoff work prepares the first frontier. Firecrawl can obtain robots.txt for the origin, inspect sitemap locations, and query indexed links when those sources are enabled. If a sitemap is found, it can supply many candidate URLs without waiting for each page to reveal its own links. The index can provide another form of prior knowledge: URLs that Firecrawl has seen before. Neither source replaces page acquisition. They accelerate the discovery of candidates that still have to pass crawl filters and, when admitted, go through page scraping.

The repository also supports a prompt-shaped crawl request. When the caller supplies a natural-language prompt, the controller can first obtain a bounded view of the website’s structure, add a sample of URLs to the prompt, and ask an extraction component to generate crawler options. Those generated options are merged carefully with explicit user values, and the result is checked against the same path-pattern constraints. The important architectural point is that natural language can help describe the frontier, but it does not bypass the frontier’s validation rules.

After kickoff locks and enqueues the first page work, each member is processed through the same single-page pipeline described earlier. A crawl job asks the page scraper to retain raw HTML even when the caller did not request it directly, because the crawler needs the source material to discover links. The worker can therefore receive a normalized document for the caller and still use the underlying page content for the next expansion step.

Published by DeepMox

FAQ

Why does Firecrawl Crawl return fewer pages than a site appears to contain?

Firecrawl Crawl returns only pages admitted by the seed’s scope, depth, domain, file, robots, threat, duplicate, and limit rules.

How does Firecrawl prevent duplicate crawl jobs?

It normalizes candidate URLs and uses crawl-scoped visited and locking state so only one worker admits a normalized target.

How do robots.txt and threat policy differ in Firecrawl Crawl?

robots.txt expresses a site’s crawler preference, while threat policy expresses the operator’s safety decision; both can deny discovered URLs, but for different reasons.

Previous9 minFirecrawl Work Unit Model

Next10 minFirecrawl Search Enrichment

Firecrawl Crawl Frontier

Crawl as Controlled Graph Traversal

A crawl begins by creating a frontier

FAQ