How a Public Page Becomes a Training Dataset
Web scraping for generative-AI training is not one technical event but a chain of state changes. The question here is how a public page, image, or caption becomes a reusable training-data object, and how each transition changes what is known, stored, and controlled. Following that chain helps the reader identify where privacy risk and responsibility arise before a model ever produces an output.
The distinction matters because “the scraper” is rarely the whole system. A site host publishes or serves material; an archiver may gather a broad snapshot; a curator selects and formats it; a distributor releases an artifact; a dataset user retrieves or transforms it; and a model developer trains on what remains. These roles can belong to different organisations, and one organisation can occupy several of them. The technical sequence is therefore also a sequence of decisions.
Collection is a decision before it is a crawl
A web crawler is software that visits pages and records server responses. Web scraping is the broader activity of extracting and storing information from publicly available web services. The extraction may concern a government register, a news article, a social-media post, a discussion forum, a blog, or an image and its alternative text. A page can look like a single object to a reader while presenting many possible fields to an automated system.
The first meaningful transition is not the network request. It is the choice of what the crawler is allowed to seek.
The European Data Protection Board distinguishes targeted scraping from untargeted scraping according to the specificity of the collection criteria, not according to whether the tool is sophisticated. Targeted scraping may limit collection to URLs in a particular domain, content in a particular language, or pages about a defined topic. Untargeted crawling gives software much wider permission to follow discovered links and add them to its queue. A short starting list can expand as each page reveals more links. The expansion matters because the controller’s knowledge of what personal data is being collected can shrink as the queue grows.
This is a control problem, not merely a performance problem. A restrictive criterion can make it easier to explain why a source belongs to the project, what kinds of information may enter, and which categories should be excluded. An unrestricted crawler may collect information because it was reachable from something else, rather than because anyone established that the information was necessary for the stated purpose. The EDPB does not say that targeted scraping is automatically lawful or that untargeted crawling is automatically unlawful. It identifies how restrictiveness affects knowledge, necessity, and risk.
The content itself can also require different collection paths. Static content remains in the page’s HTML and can often be extracted by fetching and parsing that structure. Dynamic content changes in response to an action, such as scrolling or selecting a control. A crawler may have to render the page and simulate interaction before the additional material becomes available. This technical difference does not decide whether the processing is permitted. It does change what the crawler can see, which requests it makes, what the site can communicate to it, and how a controller should describe the collection.
The EDPB describes a typical lifecycle in four broad movements. First, an organisation defines collection criteria and exclusions. Second, it extracts data from selected sources into storage. Third, it cleans the raw result by removing irrelevant markup, normalising formats, deleting unnecessary data, or avoiding duplicates. Fourth, it structures and stores what remains as a usable dataset. These are not sealed compartments. A filter can run during collection, cleaning can create metadata, and later model development can reveal that an earlier choice was too broad or inaccurate.
The movement from selection to storage also creates an accountability question: what did the system actually take? If a controller cannot connect a stored item to a source, date, criterion, or filtering decision, later claims about necessity and deletion become harder to substantiate. Website structures change frequently as well. A crawler that continues operating may need monitoring and updates, especially if accuracy depends on extracting current information. A process that was narrow at launch can become different in operation if its source patterns or inclusion rules drift.
The data changes state as it moves
DataComp CommonPool makes these transitions concrete. The dataset described in the research paper contains 12.8 billion image-text samples and was built from Common Crawl snapshots dating from 2014 to 2022. Common Crawl acts as a web archive: it collects broad web snapshots, but those snapshots are not yet a ready-to-train image-text dataset. A curator must decide what to extract and how to format it.
CommonPool’s documented curation process moved through several states. The curators gathered web snapshots, extracted image URLs and the accompanying alternative text, downloaded images, applied toxicity filters and face-detection annotations, deduplicated the result, and released a table of URL-caption pairs with metadata. The released artifact generally pointed to image assets rather than distributing the complete image collection itself.
Each transition preserves some information and loses or changes other information. A web snapshot preserves a record of a page as observed at one time, but it may not preserve all of the social context in which a person originally posted the material. Extracting a URL and alt-text retains a pointer and a description, but it separates them from the page’s surrounding audience, permissions, and later edits. Cleaning can remove irrelevant markup while retaining names or sensitive phrases. Face annotations can support a privacy filter, but they also become metadata that describes where a face was detected. A URL-caption table is smaller than a full image archive, but it still preserves a route to retrieval.
That last point is easy to underestimate. CommonPool users obtain a table and run a downloader to instantiate the images from their URLs. The released index therefore creates a new processing event when a user retrieves the assets. What the user receives may not be identical to what the original curator saw. A link may now be broken, blocked, redirected, or served behind a login. Alternatively, it may return updated content. The index is stable enough to distribute while the resources it points to remain subject to change.
This is why a URL-only release does not necessarily mean that the release has no privacy consequences. The URL, caption, annotations, and other metadata can themselves contain personal information. The table also helps multiple users perform a similar retrieval operation. It is an interface between a curated record and a new scrape, not merely a neutral description of one that happened in the past.
The diagram shows two different kinds of time. The archiver and curator may work from an older snapshot, while a dataset user may retrieve a current response years later. A source restriction arriving after curation can change future access without erasing prior copies. A user’s objection can arrive after the URL, caption, or image has been copied into another dataset. The lifecycle is therefore not simply a line from publication to training. It can loop back when a later event exposes a problem in an earlier state.
Handoffs create responsibility, not disappearance
The operational roles in this sequence should not be confused with legal labels. The EDPB says that organisations involved in creating an AI training dataset must be classified case by case, based on factual influence and how the processing is organised. “Controller” means the organisation that determines the purposes and essential means of processing personal data. A “processor” acts on behalf of a controller under that controller’s instructions. “Joint controllers” determine the purposes and means together.
Suppose an AI developer tells a separate scraping company which sources to use and which categories of data the training set should contain. If the scraper follows those documented instructions, the scraper may be a processor and the AI developer will generally be the controller for the relevant processing. The important fact is not who wrote the crawler. It is who decided why the collection existed and what essential choices defined it.
A different arrangement arises when an organisation obtains a dataset that someone else has already scraped. The original scraper and the AI developer may generally be responsible for separate processing activities, because each determines the objectives and essential means of its own work. The scraper is not automatically responsible for every later reuse, but the downstream developer does not inherit a free pass merely because the first collection happened elsewhere.
The parties can also be joint controllers. The EDPB’s example describes two organisations that jointly decide to develop a model, with one scraping the web and the other developing the model, while both agree on the collection criteria. Their separate tasks do not prevent joint responsibility if they determine the purposes and means together.
The CommonPool stakeholder map adds roles that are useful even when their legal classification remains fact-specific. A data subject is the person the information concerns. A data uploader may be someone who places the material online, which is not always the person depicted or described. A site host controls how the material is presented and what automated access it permits. A web archiver aggregates snapshots. A curator turns those snapshots into a dataset. An annotator adds labels or metadata, sometimes using automated tools. A distributor hosts the artifact. Dataset users process it, model users interact with systems built from it, and model subjects may be people affected by model decisions rather than people who appear in the training data.
These distinctions prevent a common reasoning error. The person who appears in a picture, the person who uploaded it, the site that hosted it, and the organisation that trained a model may all be different. A legal analysis that treats them as one actor will miss both the source context and the later consequences. It may also assign a safeguard to the wrong point in the chain. A site host can communicate restrictions, but a model developer still has to assess the data they choose to train on. A distributor can publish a notice, but that does not automatically make an individual request easy to fulfil across downstream models.
Persistence is the other reason handoffs matter. When a page becomes a URL-caption row, the surrounding context may be lost while the pointer remains. When a user downloads the image, a local copy may outlive the source. When the dataset artifact carries face boxes, or when a downloader extracts Exif fields, new information about the image may travel with the record. When a model is trained, some information may be represented in parameters or behaviour rather than in a row that can be located by searching for the original URL. Each transition changes the available remedy.
The paper reports examples of identity documents appearing on more than one image-hosting site. It also reports that, in its April 2025 audit, 21.4 percent of sampled image URLs failed to download while the corresponding captions, URLs, and annotations remained available. Lack of access permissions accounted for 19.0 percent of those failed links. The authors note that some “Forbidden” responses still rendered manually, and that the reasons for failure can include the site becoming unavailable, a permission change, or a server or connection issue. These observations support a careful conclusion: access at one moment is not a complete record of access over time.
A source-level change can therefore have several different effects. It may prevent a future downloader from obtaining an image. It may leave a previously downloaded copy untouched. It may not remove the caption or metadata that remains in a released index. It may arrive after a model has already absorbed patterns from the image. These are distinct states, and a controller cannot assume that changing one state automatically updates the others.
The lifecycle also clarifies what evidence is needed for later judgment. A controller should be able to say which sources and categories were in scope, what collection criteria governed expansion, when data was retrieved, which filters ran, what was removed, and who decided the purposes and essential means. The EDPB recommends recording source information, including domains and URLs where possible, and the date or period of collection. Such records do not transform questionable processing into lawful processing. They make the processing visible enough to test, explain, narrow, or stop.
The key lesson is that a scraper does not simply copy a page; it changes the page’s status. It turns a contextual publication into a stored object, then potentially into a shared index, a newly retrieved asset, and a model input. At every transition, some context can disappear while the ability to reproduce or redistribute the material increases. The next task is to examine the evidence of what survived those transitions in CommonPool—and to keep the audit’s findings, estimates, and limitations separate.