Choosing the Right Way to Reach a Page
Scrapling separates page access into three main modes because the same URL can expose different evidence depending on how it is requested. This chapter compares plain HTTP, dynamic browser execution, and the stealthy browser mode, then shows how sessions preserve state across requests; the practical outcome is a cost-aware choice of transport rather than a reflex to use the most powerful tool.
Start with the smallest transport that can produce the evidence
A scraper does not fetch “a page” in one uniform sense. It fetches a response produced by a particular protocol, browser state, timing, and set of request conditions. The right access mode is the least costly one that returns the content the task actually needs.
For a page whose useful information is already present in the HTTP response, Scrapling's Fetcher is the natural starting point. It uses a lightweight request engine and returns a Response that can be parsed with the same selection model used elsewhere in Scrapling. It can send common HTTP methods, preserve cookies, use proxies, retry failed requests, and impersonate browser TLS fingerprints. When browser impersonation is enabled, the library also generates browser-like headers that match the selected browser version. These features address sites that care about the shape of a request without requiring a browser to execute the page.
That distinction matters for both speed and evidence. A plain HTTP request observes the server's response. It does not execute the JavaScript that a browser would run after receiving that response. If the server has included the product price, article body, or API data in the initial HTML, that limitation is irrelevant and browser execution adds cost without improving the result. If the page contains only a shell that later requests its content, the HTTP response may be valid but incomplete.
Scrapling's dynamic browser mode addresses that second case. DynamicFetcher launches Chromium or Google Chrome, loads the page, waits according to the selected conditions, and returns the resulting content through the same Response-shaped interface. It can wait for a selector to be attached or visible, wait for network activity to settle, run an action after navigation, and run setup code before navigation when listeners or routes must be installed early. A user can also connect to a browser through the Chrome DevTools Protocol instead of launching a local one.
A rendered page is therefore a timed observation, not simply a larger version of an HTTP response. The returned DOM depends on when the fetcher stops waiting, what JavaScript has completed, whether a selector has appeared, and whether the requested resources were allowed to load. The network_idle option can help with applications that load data in the background, while a specific wait_selector can express a more meaningful completion condition for a page whose network never becomes quiet. Conversely, disabling images, fonts, stylesheets, or other resources can speed some requests but can also keep a site from finishing correctly. The faster observation is not automatically the more accurate observation.
Dynamic browser access can also expose a route that is invisible in the initial markup. Scrapling can capture XHR or fetch responses matching a URL pattern while a browser session loads the page. Those captured responses are returned as Response objects with their own URL, status, headers, and body. This can be useful when a single-page application obtains its product data from a background endpoint, but it changes what the scraper is relying on: the evidence is now tied to browser timing and the selected capture pattern as well as to the visible DOM.
The third mode, StealthyFetcher, is a specialized browser path for sites with harder protection. The project documents additional controls for browser fingerprints, isolated JavaScript execution, WebRTC leak prevention, canvas noise, headless-mode detection, and optional Cloudflare Turnstile or interstitial solving. These controls are mechanisms intended to make a browser session less distinguishable from an ordinary browser. They are not a universal guarantee that a site will serve the desired content, and they should not be confused with authorization to access it.
The three modes are best understood as an access ladder with different costs and failure conditions:
- Use plain HTTP when the response contains the required content and the site's protection does not require browser behavior.
- Use a dynamic browser when JavaScript rendering, interaction, a wait condition, or browser-only state determines the content.
- Use the stealthy browser when a permitted task encounters protection that the ordinary browser mode cannot address and the additional resource and maintenance cost is justified.
This is not a hierarchy in which stealth is “better” than HTTP. A browser can be slower, use more memory, require browser dependencies, and introduce timing choices that complicate reproducibility. Plain HTTP can be insufficient for a client-rendered application. A stealthy browser can still fail on a site-specific challenge or return a page that looks successful but lacks the expected record. The evidence requirement should choose the mode, not the other way around.
The installation model reinforces that judgment. Scrapling's base package contains the parser, while fetchers and browser features are supplied through optional dependency groups and browser installation. A user who only needs to parse stored HTML should not need to launch a browser or install its dependencies. A user who needs browser fetching has accepted a larger operational surface, including browser version compatibility, timeouts, resource policy, and cleanup.
Redirect handling is another access decision with a security consequence. In the current HTTP fetcher documentation, redirects default to a safe mode that follows ordinary redirects while rejecting destinations in internal or private address ranges. That is useful when a URL comes from a user or another untrusted source, because a seemingly harmless target can redirect the fetcher toward a service that was never intended to be reachable. The restriction can be changed deliberately, but relaxing it changes the threat model. Convenience is not a reason to erase the boundary without understanding the destination.
The sequence shows an important discipline: escalation follows validation. A failed CSS selection does not prove that a browser is needed; it may indicate a structural change, a missing selector, or a wrong response. A 403 or challenge page does not prove that repeated stealth attempts are appropriate; it may be a policy boundary. First identify what evidence is missing, then choose the access change that could plausibly supply it.
Sessions change the cost and state of access
One-off fetches are convenient because they create and dispose of their access machinery around a single request. Repeated requests change the calculation. Opening a new connection, browser, or context for every page can discard cookies, repeat setup, and spend resources on initialization. Scrapling's session classes keep a configured access path alive so that later requests can reuse state.
An HTTP FetcherSession can share connections and a connection pool, cookies, browser impersonation, retry settings, and proxy configuration across requests. That is useful for a sequence that depends on an authenticated or stateful exchange, and it avoids rebuilding the same request environment for each URL. A session also makes the access decision explicit: the request sequence has an owner for cookies and connection state instead of relying on hidden global variables.
Browser sessions preserve a different kind of state. A dynamic or stealthy session can keep a browser context and reuse tabs. In the current release, browser tabs remain open after a request and can be reused for a later one. Each new request reapplies its tab-level settings, such as timeouts, headers, resource blocking, or wait conditions, so settings from an earlier request do not silently carry over. This behavior is important because reuse creates hidden coupling if the next request inherits the previous request's assumptions.
Asynchronous browser sessions can maintain a bounded pool of tabs for concurrent requests. When a free tab exists, the session reuses it. When the pool has room, it opens another. If every tab is busy, a new request waits for capacity and can eventually time out if a page never becomes available. Tabs that encounter an error are closed and replaced. This is a resource policy, not an unlimited concurrency switch: the pool size limits browser work, while the target site's capacity and the quality of the returned evidence still determine whether more parallelism is sensible.
The recent tab-reuse behavior also illustrates why version awareness matters. Earlier releases closed tabs after each request because reusing them could leak settings. The current release resets settings on reuse and keeps tabs open to improve subsequent work and enable automation across requests. A design that depends on session behavior should therefore be tested against the installed version rather than inferred from an old example.
Sessions can be combined rather than chosen once for an entire application. In Scrapling's spider framework, a Session Manager can register a fast HTTP session, a dynamic browser session, and a stealthy browser session at the same time. A request carries a session identifier, so a listing page can stay on HTTP while a protected detail page is routed to a browser. A session can also be started lazily, which reserves browser startup until a request actually needs it. When a follow-up request does not specify a new session, it inherits the originating request's session; an explicit session choice overrides that inheritance.
This arrangement is more than an optimization. It separates two kinds of continuity. The first is access continuity: cookies, connection state, browser context, fingerprint, locale, and proxy settings can be kept together for requests that belong to the same interaction. The second is task continuity: a later request may need a different access mode because its page has different JavaScript or protection requirements. Multi-session routing lets the workflow preserve the first kind without forcing every page into the most expensive version of the second.
The separation introduces responsibilities. A persistent browser consumes memory and may keep sensitive cookies or local storage alive. A session with a proxy or a browser fingerprint has a stable operating identity that may be useful for one sequence and inappropriate for another. A per-request override can change the evidence or network path while the session remains open. The session must be closed when the work is finished, especially in an AI-facing service or long-running process where an orphaned browser can accumulate resources.
The same principle applies to page actions and background capture. A page_setup callback belongs before navigation when an event listener or route must see requests from the beginning. A page_action belongs after navigation when the loaded page must be scrolled, clicked, or otherwise manipulated. If a session reuses the page and the next request runs setup before navigating away, the page's current state becomes part of the sequence. That can enable useful chained automation, but it also means that a debugging failure may come from stale state rather than from the target page alone.
The practical choice is therefore not simply “HTTP or browser.” It is a tuple: what content must be observed, what state must persist, how many requests will share it, what timing makes the evidence valid, and how much cost or sensitivity the workflow can accept. Use a one-off request when isolation and simplicity matter. Use a session when cookies, connections, browser state, or repeated setup have value. Use multiple sessions when different pages demand different transports, and make the handoff between them explicit.
Scrapling's fetchers make these choices composable by returning a common Response/Selector-shaped result. That common result should not hide the differences in how it was obtained. A static response, a rendered DOM, and a page captured after a challenge are different observations with different assumptions. Record enough metadata and validate the fields that matter before treating them as equivalent.
Once access is treated as a controlled choice rather than a feature contest, the larger architecture becomes easier to see. A single Response can be parsed and checked; a callback can then turn it into more requests or items. The next problem is not which fetcher is universally strongest, but how to preserve request identity, session choice, scope, and state when that response becomes the beginning of a crawl.