Choosing the Extractor: Selectors, Semantics, or Language Models

Crawl4AI’s extraction layer answers a practical design question: once a page has been rendered and shaped into an HTML or Markdown view, which method can turn it into fields without adding unnecessary cost or uncertainty? This chapter compares selectors, regular expressions, semantic clustering, and language-model extraction as different responses to different page structures. The goal is not to find one universally “smartest” extractor, but to choose the least uncertain method that can express the task and to build checks around the result it returns.

The choice starts with the input, not with the model. A repeated product card, a list of dates, a group of conceptually related passages, and an irregular article do not present the same extraction problem. Crawl4AI exposes several strategies because structure, meaning, and operational cost are separate dimensions. A method is strong when its assumptions match the page and the task; it fails when those assumptions are hidden or violated.

Four routes from content to structure

The strategies share a common contract: they receive a URL and one or more content sections, then return records or blocks. Crawl4AI chooses the content format and chunking path before it invokes the strategy. HTML-like inputs can be kept as one identity chunk, while Markdown can be split by the configured chunker. That detail matters because an extractor can only reason over the boundaries it receives.

classDiagram class CapturedView class Chunker class ExtractionStrategy class CSS_XPath class Regex class Cosine class LLM class StructuredResult class UsageAndErrors class Validation CapturedView --> Chunker : supplies input Chunker --> ExtractionStrategy : provides sections ExtractionStrategy <|.. CSS_XPath ExtractionStrategy <|.. Regex ExtractionStrategy <|.. Cosine ExtractionStrategy <|.. LLM ExtractionStrategy --> StructuredResult : returns records or blocks LLM --> UsageAndErrors : tracks calls and failures Validation ..> StructuredResult : checks coverage

The diagram is deliberately not a ranking. The shared interface makes the strategies interchangeable at the pipeline boundary, but their internal assumptions are not interchangeable. A selector does not become semantic because it is wrapped in a common class. An LLM response does not become deterministic because it is placed in a JSON-shaped object. The rest of the decision depends on what each route can know from its input.

Repeated structure favors selectors

CSS, XPath, and LXML-based strategies are the most direct route when the page has a stable visual or structural pattern. A schema identifies a base selector—the repeated element that represents one record—and fields inside or beside it. A field can read text, an attribute, HTML, or a regular-expression match; it can apply a transformation such as trimming or case conversion; and it can supply a default when the target is missing.

This makes the schema an executable mapping rather than a list of desired names. Suppose a page contains repeated product cards. The base selector identifies one card. A title field selects the heading, a price field reads the price text, and an image field reads the image source attribute. The extractor repeats that mapping for each matching card. If the card contains a nested specification list, nested and list field types can walk into that structure and return a nested record rather than flattening everything into one string.

The repository also supports a sibling source for layouts where one logical record is split across neighboring elements. A common pattern is a primary row containing a title followed by a second row containing metadata. The schema can resolve the next sibling before applying a field selector. This preserves a useful distinction: the extraction unit is a logical record, even when the HTML unit is spread across multiple elements.

These capabilities are valuable because they make repeated extraction cheap and inspectable.