Search Gets Useful When Retrieval Meets Fresh Content

For Firecrawl, Search is not just a wrapper around a search provider. It is a composition that starts with a question, retrieves candidate sources, applies policy before those candidates are counted or fetched, and optionally turns selected results into richer page context. This chapter focuses on the boundary between retrieval and acquisition. The practical judgment is to know when a search-only response is enough, when scraping the results is worth the extra work, and when Map or Crawl is a better fit because site structure matters more than query relevance.

Retrieve candidates without pretending they are documents

A search result and a scraped document answer different questions. A result usually gives the caller a URL, title, description or snippet, position, and source category. It is useful for deciding what deserves attention. A document is the output of acquiring and transforming a target page: it can contain normalized Markdown, HTML, metadata, links, images, a screenshot, or structured fields. Search can connect these two forms, but it should not silently treat the first as the second.

Firecrawl’s Search execution starts by building a query from the caller’s text, categories, and domain constraints. The v2 schema supports source types such as web, images, and news, and categories such as GitHub, research, PDF, and developer search in the checked-in revision. Limits, language, country, location, and include or exclude domains further shape the candidate set. These inputs are not merely filters applied to a finished response. They influence which upstream search path is called and what the returned results are allowed to represent.

Provider choice is itself a fallback mechanism. When Fire Engine Search is configured, Firecrawl sends the query and search options there. If that path is not configured, the implementation can try SearXNG and use it when it returns web results. If that does not provide a usable response, it falls back to DuckDuckGo. A final empty response is possible. The interface remains stable while the acquisition source changes, but the repository does not establish that the providers have equivalent ranking, coverage, latency, or failure behavior.

That qualification matters because provider fallback is not the same as result interchangeability. A search engine may interpret language, location, freshness filters, safe-search options, and domain constraints differently. A fallback can preserve the shape of a response while changing the distribution of candidates. Firecrawl’s code shows a practical availability strategy: prefer a configured first-party path, then use configured alternatives, and do not make the caller learn each provider’s API. It does not prove that a query produces the same answer across those paths.

The result set is also a policy boundary. When threat protection is active, the Search pipeline checks result URLs before it slices them to the requested limit, counts them, scrapes them, or returns them. Blocked URLs are removed rather than returned with a warning that leaves the caller to make the safety decision later. This ordering has two consequences. First, a denied result cannot consume enrichment work. Second, the number and positions the caller sees describe the post-policy set, not the raw provider response.

The same ordering is used for different result groups.