From Web Pages to AI-Ready Context

Scrapling's AI and RAG features extend the adaptive control problem beyond finding a DOM element: they decide how much of a fetched page should cross into a model, a vector store, or an agent. This chapter moves from a Response to targeted Markdown and from a site crawl to page-level corpus records, then examines the sanitizer, session, redirect, and transport controls that shape that handoff. The practical outcome is a scoped content pipeline whose convenience does not get mistaken for truth or complete security.

A useful AI input starts with less, better-scoped page content

Raw HTML is a poor default for language-model or embedding input. It contains markup that carries no useful meaning for the task, navigation that repeats across pages, scripts and styles that are not part of the article, and tracking or layout material that competes with the text a model is meant to analyze. Sending all of that content downstream increases the amount of context without increasing the amount of evidence.

It can also introduce a security problem. A page can contain hidden text or other content that a human does not see but that an extraction pipeline might pass to an AI system. If that text is phrased as an instruction, the model may treat page content as an instruction about how it should behave. The problem is not solved by calling the page “untrusted” after the entire document has already entered the model's context. The boundary needs to be narrowed before the handoff.

Scrapling's Response can be converted to Markdown, and the conversion can be limited to a CSS selector. With main_content_only enabled, the AI-oriented extraction path first focuses on the body, removes common noise, sanitizes hidden content, and then converts the selected content to Markdown or text. A CSS selector can further restrict the result to an article, a product description, or a specific group of fields. The selection is consequential: it controls both what the model sees and what the downstream pipeline is able to retrieve.

This is a different role for the parser from the one played during adaptive relocation. Adaptive relocation asks which current element most resembles a previously saved element. AI-targeted extraction asks which portion of the current page should be exposed to another system. The first is about preserving structural identity across time; the second is about limiting context at a handoff. Both depend on selection, but their failure modes differ. A relocated product card can be structurally plausible but semantically wrong. A perfectly selected article can still contain stale or false statements.

The choice of scope should follow the consumer. A question about a product price may need a product card or a price region, not a site's navigation and footer. A retrieval corpus may need the article body, title, URL, and enough surrounding structure to preserve meaning across chunks. A visual question may need a screenshot because layout, charts, or images carry evidence that plain text cannot represent. Smaller context can lower cost and distraction, but over-narrowing can remove qualifications that the consumer needs.

The RAG guide in Scrapling uses the same Markdown operation for one-page ingestion and for a whole-site corpus. A page-level record contains the page URL, its title, and the converted Markdown. Those fields matter together. Markdown provides a cleaner text representation, while the URL and title preserve provenance that can be attached to chunks in a vector store or used to return a reader to the source.