Checkpoint: The Unit of Work Is Not the Endpoint

Firecrawl’s endpoint map and single-page pipeline are two views of the same subject. This checkpoint connects them so the reader can reconstruct a request from its starting material, required capabilities, acquisition path, and result state, without memorizing implementation names. The practical judgment is to recognize what remains common across Scrape, Parse, Search, Map, Crawl, and Batch Scrape—and what must change when one document becomes a result set or a growing group of page jobs.

The same architecture appears under different starting conditions

A useful way to test the model is to remove the endpoint names and describe the request in terms of work. Suppose the caller has one URL and wants readable text. The system still has to validate the URL, decide whether a reusable representation is adequate, obtain content through a compatible route, normalize it, and return a document. The same sequence applies if the caller wants a screenshot or structured JSON, but the capability contract becomes stricter. The endpoint is only the public entry point; the underlying question is what must happen before the result can be trusted as the requested representation.

Now change the starting material to a file. The discovery problem largely disappears, but acquisition does not. Firecrawl must classify the upload, choose a parser appropriate to a PDF or office document, and shape the output into a document. The resulting path is different from a web scrape because the bytes are already supplied, yet it still participates in the same broad distinction between acquisition and transformation. “Parse” names the input boundary; it does not abolish the need for validation, capability checks, and output shaping.

A known list of URLs introduces a different kind of state. Each URL can be treated as page work, but the request as a whole needs a group identity, a way to track accepted members, and a means to report results that may finish at different times. Batch Scrape therefore does not need to discover links, but it does need to coordinate many instances of the page pipeline. The caller’s knowledge of the frontier reduces one uncertainty while increasing the need for group-level accounting.

A website boundary reverses that balance. With Map, the central job is to produce a bounded set of candidate URLs. With Crawl, the central job is to let an admitted page reveal more candidates while preserving the boundary around the seed. The page pipeline is still the unit that obtains and transforms content, but discovery now feeds future work. That feedback loop makes scope, robots behavior, URL normalization, deduplication, and completion state part of the meaning of the result.

Search begins with a question rather than a known frontier. Retrieval supplies candidates ordered for relevance, and optional scraping turns selected candidates into richer page context. The shared page pipeline appears again, but the selection rule is different from Crawl’s. A crawler asks whether a link belongs inside a site-shaped boundary. Search asks whether a result answers a query well enough to be returned or enriched. Treating both as “a list of URLs” hides the reason they produce different behavior.