Inside a Scrape: The Waterfall That Makes the Web Legible
Firecrawl’s single-URL Scrape is the clearest place to see the project’s central design problem: one input may require a cache lookup, a browser, a direct fetch, a specialty parser, or several attempts before it becomes a usable document. This chapter follows that mechanism rather than cataloguing engines. The goal is to understand why Firecrawl can change acquisition paths without changing the caller’s basic expectation—a result that matches the requested format—or silently treating empty content as success.
A result is chosen, not assumed
A scrape begins with more than a URL. Firecrawl constructs a per-request object, called Meta in the source, that carries the normalized options, feature flags, abort state, possible file handoffs, cost tracking, and the state accumulated during the attempt. That object gives the rest of the pipeline a shared answer to three questions: what did the caller ask for, what conditions apply to this target, and what has already happened during recovery?
The options become feature flags before the system selects an engine. A screenshot adds a visual requirement. Actions require a browser-capable route. A PDF or document extension adds a file-parsing requirement. Location, mobile rendering, a stealth proxy, audio, video, branding, or a wait request each narrows which acquisition methods can legitimately satisfy the request. This is the first important causal link: the caller’s output and interaction choices determine the capabilities that an engine must have.
The engine inventory then expresses two different properties. Each engine declares which features it supports, and it has an internal quality value that helps order otherwise eligible candidates. The available set can include an index, browser-based Fire Engine variants, Playwright, a direct fetch client, PDF and document parsers, image processing, and specialty paths such as Wikipedia or X/Twitter when configured. These names are less important than the division of labor they represent. An index can reuse prior work; a browser can execute and render; a fetch client is lightweight; a file parser understands bytes that are not ordinary HTML.
Firecrawl can put an indexed representation first when it is eligible for the request. A suitable index entry is not merely any document with the same URL. Its age, rendering variant, mobile or location settings, wait time, screenshot capability, and file-parsing properties must match what the request needs. If the caller asks for a representation that the stored entry cannot provide, the index behaves as a miss. That decision protects meaning: a fast response without the requested capability would be a wrong response.
When live acquisition is necessary, buildFallbackList calculates which engines support enough of the requested features. It assigns a support score to the features an engine can satisfy and compares that score with a threshold derived from the request. Higher-quality eligible engines are preferred, but specialty engines remain available when the request makes them necessary. An explicit stealth request, for example, restricts the candidates to engines that can actually honor it rather than allowing a faster but incompatible engine to ignore the option.
The list is not always executed as a simple one-after-another chain. Firecrawl starts an eligible engine and waits for either a usable result, an engine error, or a reasonable point at which another engine should be started. The current attempts can be raced while a later candidate is added to the waterfall. If one candidate returns an acceptable result, the losing work is signalled to stop. If a candidate fails, the failure is classified: some failures remove the engine, some add a feature such as stealth proxy, and some trigger a new attempt with a different interpretation of the content.
A successful HTTP request is not enough. The scrape loop examines the returned status, whether the engine reported a page error, and whether there is usable output. When Markdown or another content-dependent format is needed, Firecrawl checks whether the content produces non-empty text. A bad status may still be returned as a meaningful result in the error-handling model, but an apparently successful page with no useful content is not automatically accepted. Parsed image output is a deliberate exception: an image can be a complete result even when OCR yields no text.
This check matters because web failure often looks superficially healthy. A bot wall can return status 200 with a short challenge page. A rendering service can complete while producing an empty body. A direct client can receive a login shell instead of the article it was asked to fetch. If the pipeline judged only status codes, all of these would become false successes. Firecrawl instead uses content and engine-specific signals to decide whether it should continue.
Proxy failures illustrate how the waterfall adapts. If an automatic proxy path receives a response that looks like a proxy or access problem, the scrape can add a stealth-proxy feature and reselect engines. That is different from blindly retrying the same request: the second attempt changes an input condition that may be responsible for the failure. The recovery still has bounds. The retry tracker limits attempts and feature toggles, so a stubborn target cannot consume unlimited time while the system keeps changing its mind.
The sequence shows why this is a waterfall rather than a bag of fallbacks. The loop owns the decision about whether an output is good enough and whether a failure changes the next attempt. An engine owns acquisition and reports what it found. The transformer stage owns representation and enrichment. The cache or index owns reuse only when the stored result can preserve the same contract later.
Files expose the boundary between acquisition and interpretation
PDFs and office documents make the architecture especially visible because “fetch the page” and “understand the content” are clearly different jobs. With Fire Engine available, a PDF or document URL is often routed first through a browser engine. The browser can obtain a file through its access and proxy infrastructure, then hand the file to a specialty parser. The browser is not expected to turn every byte into final Markdown, and the parser is not expected to solve every antibot problem.
The handoff is conditional. The browser response is checked for the file type and size, and the pipeline keeps the file on disk or uses a reference path when inline transfer would be too large. The specialty parser then verifies the bytes and selects an appropriate processing path. For PDFs, the source supports several possibilities: native Rust extraction for eligible text-based files, FirePDF, MinerU or RunPod processing, and a final pdf-parse fallback. OCR mode, page limits, layout complexity, file size, time remaining, and service configuration influence those choices.
That arrangement creates useful failure distinctions. A browser may have run but returned no file. A direct download may fail at the proxy. A downloaded payload may not actually be a PDF. A PDF may be valid but require OCR that a fast mode was not allowed to perform. Those are not interchangeable errors: some justify another browser handoff, some move to another parser, and some should be reported clearly rather than producing an empty document. Temporary files are cleaned up when the parser finishes, including when the path fails.
Documents follow the same principle with a smaller pipeline. A prefetched document can be converted directly from bytes. A direct download can be used when the environment has no Fire Engine or when an explicit engine pin keeps that path available. The content type and URL extension help identify the document format, after which a native conversion library produces Markdown and a safe Markdown-to-HTML conversion supplies the normalized HTML representation. An ordinary HTML page that happens to reach the document engine is allowed to decline rather than repeatedly treating the page as a malformed document.
The point is not that Firecrawl has a parser for every format. The repository shows a design that preserves the boundary between obtaining a payload and interpreting it. That boundary makes specialized processing composable: a browser can solve access, a parser can solve format, and the general transformer stack can solve the output contract. It also makes the limits visible. If the relevant external parser or Fire Engine is not configured, a feature that depends on it may be unavailable even though the API schema can describe it.
The document is built in stages
Once an engine has supplied content, Firecrawl still has not necessarily produced the object the caller requested. The transformer stack turns engine output into a document with normalized fields. It can derive cleaned HTML from raw HTML, derive Markdown from cleaned HTML, extract metadata, find links and images, run redaction or branding logic, fetch product or menu information, invoke structured extraction or summaries, and perform change tracking or question-like transformations where requested.
The order matters. Markdown is derived only when the caller asks for it or when another requested feature depends on it. The source shows that JSON extraction, summaries, questions, highlights, queries, change tracking, and PII redaction may all need Markdown as their source text. That prevents unnecessary work for a request that wants only raw HTML or another representation, while ensuring that a higher-level format does not accidentally receive no source material.
HTML-to-Markdown conversion itself has a fallback chain. If configured, Firecrawl first calls an HTTP conversion service. If that is unavailable or fails, it can use a Go shared library, and then it falls back to Turndown with GitHub-Flavored Markdown support. The output is post-processed in each successful path. This is a small example of the same system principle operating inside the larger one: the normalizer also has a preferred route, a local alternative, and a final JavaScript fallback.
Main-content extraction has its own recovery behavior. The normal path removes unwanted elements and converts the remaining HTML. If the resulting Markdown is empty while the request asked for main content, Firecrawl reruns the HTML transformation with full content and converts again. This does not guarantee that a target contains useful prose; it means an aggressive cleaning decision is not allowed to turn a page into an empty success without a second check.
Other transformers add fields only when their conditions are met. Link extraction can populate the document’s links when the caller requested them, and it can also forward links to an indexer under configured traffic conditions. Image extraction reads from normalized HTML. Structured JSON may come from native engine output or a later LLM transformer. Deterministic JSON uses a different reusable extractor path, but both are shaped into the document’s JSON field. The caller sees a consistent output location even though the production mechanism differs.
At the end, coerceFieldsToFormats enforces the public contract. Fields that were produced but not requested are removed, and missing requested fields generate warnings. The behavior is intentionally conservative. A browser may return a screenshot as an incidental artifact, but the response should not expose it as if the caller had requested it. Conversely, if a requested format has no corresponding field, the pipeline records that mismatch rather than silently presenting a misleadingly complete object.
This is why acquisition and interpretation should not be collapsed into the phrase “scrape the page.” Acquisition answers, “What content could be obtained under these access conditions?” Transformation answers, “How should that content be represented, enriched, and filtered for this request?” An engine can return HTML without having produced the final Markdown. A parser can return Markdown without having extracted links or metadata. The document emerges from the sequence.
Reuse is valid only when the capability still matches
The index closes the loop by turning successful live work into a possible future shortcut. A successful document may be written to GCS and registered through an index-insert queue. The index stores enough variant information to distinguish factors such as mobile rendering, location, ad blocking, stealth use, wait time, screenshots, and status. A later request can consult a lightweight cache or database query before invoking an engine.
But caching is deliberately selective. The source excludes zero-data-retention requests, uploaded parse content, Exchange-delivered content, action-driven requests, custom request contexts, and output modes whose capabilities are not represented in the index schema. Page-aware PDF output and typed blocks are examples of data that should not be indexed as if ordinary document Markdown were an equivalent representation. A cache write that loses a required capability would create a future correctness problem.
The index can also reject a stored entry after it has found one. A request may ask for a newer result through a minimum-age condition. A PDF cache may contain an unparsed base64 representation when the caller wants parsed PDF output, or a parsed representation when the caller wants raw bytes. A cached document may exceed a caller’s maximum-page setting, or an OCR result may have been created under an enablement condition that the current request does not have. These become index misses, allowing the live waterfall to make a fresh decision.
The trade-off is clear. Reuse can lower latency and avoid repeated access to a difficult site, but only if the system records enough about the representation to know when reuse is safe. Freshness alone is not sufficient, and URL equality alone is not sufficient. The requested capability is part of the cache key and part of the decision to fall through.
A single Firecrawl scrape is therefore a sequence of judgments: what capabilities the request declares, which acquisition methods can honor them, whether returned content is usable, whether a file needs a handoff, which transformations are necessary, and whether the resulting document is safe to reuse. The design does not remove the web’s variability. It keeps that variability from being mistaken for a stable result.
The same page-level pipeline can be called directly, embedded in Search, or repeated as part of Crawl and Batch Scrape. What changes at larger scale is not the meaning of a page result, but the state needed to coordinate many of them. That is the question the next checkpoint brings into view.