When Selectors Break: How Scrapling Recovers Structure
Scrapling's adaptive parser addresses a specific failure: the page still contains the information you need, but the selector that once found it no longer matches. This chapter explains how Scrapling turns an initial match into persisted structural evidence, how it uses that evidence to search a changed page, and how to decide when the result is a useful recovery attempt rather than a fact you can trust automatically.
Adaptive scraping begins with a remembered observation
A normal selector describes where an element is now. A durable extraction process also needs a description of what the intended element looked like when the scraper first found it.
Consider a product card. The first version of a page might place a product title inside an element with a particular ID, class, and nesting path. A scraper can select that element with CSS or XPath and extract its text. Later, the website may replace the ID, add a wrapper, rename a class, or move the title into a new child element. The product title has not necessarily changed as a piece of information, but the original route to it has become stale.
Scrapling's adaptive feature treats this as a two-phase problem. In the save phase, the parser records a selected element's properties. In the match phase, a later page is searched for an element with a sufficiently similar description. The parser does not ask an external model to interpret the page. It stores evidence from a known match and compares that evidence with the current document.
The feature must be enabled when the Selector or fetcher is configured. That initial condition matters because adaptive behavior is not a hidden fallback applied to every selection. With adaptive mode on, a selection can use auto_save to preserve the first matching element for later use. When the same selection is attempted in the future, the caller can enable adaptive fallback. If the ordinary selector still finds an element, Scrapling returns that ordinary result. It consults the saved description only when the ordinary path produces no result.
The distinction protects the normal case. A site may contain several candidates that happen to resemble the saved element, so there is no reason to run a whole-page similarity scan while the original selector still works. Adaptive relocation is a recovery path for a broken structural assumption, not a replacement for ordinary selection on every request.
The saved record needs two kinds of identity. The first is the site identity. Scrapling derives a domain key from the page URL so that an element saved for one website does not automatically become the evidence for another. The documentation also supports an adaptive_domain value for cases in which two URLs should be treated as the same site—for example, when comparing an archived representation with a current URL or when a site changes its address during a redesign. Without a shared identity, the saved evidence may exist but remain invisible to the later lookup.
The second is the identifier used to retrieve the record. For CSS and XPath calls, the selector can serve as the default identifier, but the documentation recommends supplying a meaningful identifier when the same conceptual element may later be expressed by a different selector. That separates the identity of the target—such as “primary product title”—from one temporary spelling of its location.
Scrapling's default storage is SQLite. The storage system normalizes the domain, stores the element description as data, and keeps the domain and identifier unique as a pair. The implementation uses a lock and SQLite's write-ahead logging mode, which supports the library's intended use in threaded environments. These details matter because adaptive scraping is not only a matching algorithm. It is also a persistence contract: the later request must use the same effective site identity and identifier, and the saved record must be available to the parser.
There is a deliberate asymmetry in the save operation. If a selector returns several elements, the adaptive save process records only the first one. That is useful when the target is one distinctive element, but dangerous when a broad selector represents a collection. Saving .product does not create a durable description for every product card. It creates the evidence for the first selected card, which is why a collection should be narrowed or handled through a structure that explicitly represents the collection before adaptive recovery is used.
Once the old observation is available, the later call follows a bounded sequence: try the original selection, retrieve the stored description if the selection is empty, compare the current page's candidates, and return a candidate only if it clears the configured minimum. The process is easier to reason about as a state transition than as a promise that a selector will “keep working.”
The flow also explains what adaptive scraping does not do. It does not silently replace every failed extraction with the first vaguely similar node. It has a saved identity, a candidate comparison, and a threshold. If any of those prerequisites is missing, the recovery path may produce no element at all. That is a more useful failure than silently accepting a result from an unrelated part of the page.
What the score can know—and where it must stop
Relocation works by comparing the saved element's structural description with elements in the current page. Scrapling's implementation scans page elements and calculates a score for each candidate. The candidate with the highest score is eligible, but only if it reaches the minimum percentage supplied to the relocation operation.
The score combines several signals because any single property can be unstable. Tag name is one signal: an old article is more plausibly related to a new article than to an unrelated footer. Text can contribute through a fuzzy comparison, which allows small changes rather than requiring an exact string. Attributes are compared as collections of names and values, and several important attributes—such as class, ID, link target, or source—receive separate comparisons when they were present in the original. The element's path through the tree, its parent's name, attributes and text, and its sibling pattern also contribute.
This combination is what lets structural evidence outlive a literal path. A wrapper can be inserted without changing the target's tag, nearby text, or relationship to its parent. A class can be renamed while the candidate still has a similar role and neighborhood. A URL can change, while the other attributes and the surrounding structure remain informative. The algorithm does not rely on one immutable identifier; it treats the element as a bundle of partially reliable observations.
That also means the score is a heuristic, not a semantic judgment. It can measure resemblance between DOM descriptions, but it does not know that two pieces of text represent the same business concept. Two product cards can share the same shape. A navigation item can resemble a content item. A redesign can preserve the structure while changing the meaning of a field. The score helps choose a candidate; it does not verify the candidate's business identity.
The current source snapshot uses a 40 percent default minimum for relocation. The project changed the default from an unconstrained match to this threshold and added a warning when no candidate crosses it. The threshold is therefore a guardrail against weak matches, not a claim that 40 percent similarity means 40 percent correctness. A higher threshold can reject legitimate changes; a lower threshold can admit misleading candidates. The right value depends on the page's regularity and on how costly a false match would be.
The warning is part of the mechanism's honesty. When the best candidate remains below the minimum, Scrapling reports the top score and returns no relocation result rather than presenting its best guess as settled data. An application can then log the page, inspect the redesign, save a new observation, choose a more specific identifier, or deliberately adjust the threshold after understanding the mismatch. Lowering the threshold without examining the page simply converts an explicit uncertainty into an implicit one.
There are several ways for the lookup to fail before scoring begins. Adaptive mode may not have been enabled at initialization, so a request to save or relocate is ignored or rejected according to the operation. The later page may use a different domain key from the one under which the observation was saved. The identifier may not match. Or the target may have disappeared entirely. These are not equivalent failures, and they should not be repaired with the same response. A missing record is a persistence or identity problem; a low score is a structural-evidence problem; an empty candidate set may mean the content is gone or that the selector's original meaning was too narrow.
The broad-selector limitation deserves special attention. Because only the first element in a selection result is saved, a selector that matches different locations on different pages can produce an apparently successful but incomplete relocation. The parser can recover the first saved target while leaving its siblings or peers unrepresented. The safest pattern is to save a distinctive container or element whose role is clear, then use that recovered element to navigate to related values. If the task is to extract a changing collection, the collection itself needs an explicit extraction strategy rather than one saved exemplar being treated as the whole dataset.
The same reasoning applies to find_similar. Scrapling can use an element found by text, filters, or another method as the starting point for finding comparable elements. It first narrows candidates by depth and surrounding tag relationships, then compares attributes, optionally considering text. This is useful for moving from one known product card or table row to its peers. But “similar” means structurally alike under the chosen rules. It does not mean that every returned element has passed a domain-specific completeness or relevance test.
A reliable consumer should therefore validate at the point where relocated structure becomes data. For a product record, that might mean checking that the recovered container contains the expected title and price fields, that the price matches an accepted format, and that the URL belongs to the intended domain. For a table, it might mean checking the expected headers or column count. For a document page, it might mean confirming a title, canonical URL, or content marker. These checks are not redundant with the similarity score. The score asks, “Which current node most resembles the saved node?” The consumer asks, “Does this node still represent the record I am allowed and intending to collect?”
This separation makes adaptive scraping useful without making it mystical. Scrapling remembers a selected observation, scopes it by site and identifier, compares multiple structural signals, refuses weak matches by default, and exposes a path for deliberate repair. It does not know the future layout of a site, preserve every item in a broad selection, or guarantee that a high-scoring node carries the same meaning.
Once the parser can cope with a changed structure, a different question remains: can the page be obtained in the first place? If the needed content is present in a plain response, the parser's recovery mechanism may be enough. If the content appears only after JavaScript or behind a protection layer, the next failure is not structural. It is an access decision, and that is where Scrapling's separate fetching modes become important.