A Defensible Path Through the Scraping Decision

For generative-AI training, web scraping presents a deceptively simple choice: collect material that can be reached, or do not collect it. The evidence and guidance examined in this series support a more demanding question: can a particular collection be justified from source context through model output, with enough limitation, provenance, and control to remain accountable when conditions change? This final synthesis turns that question into a conditional decision path and makes clear where the evidence supports judgment—and where it does not.

The answer is not that public web data can never be used. The EDPB’s 2026 guidance describes circumstances in which a freely accessible source, a clear purpose, appropriate safeguards, and reasonable expectations may support a legitimate-interest analysis. Nor does the CommonPool audit prove that every web-scraped dataset has the same contents or that every downstream use is unlawful. The stronger conclusion is narrower: accessibility alone cannot carry the decision. A defensible program has to show why the data belongs in the system, what the controller knew about its context, what was excluded, what controls were tested, and what happens when those controls fail.

The decision starts before the crawler

The first discipline is to define the purpose before defining the crawl. “Build a general-purpose model” may describe a technical ambition, but it does not by itself explain why identifiable personal data is necessary. The EDPB treats purpose limitation as a requirement that personal data be collected for specified, explicit, and legitimate purposes and not later used in incompatible ways. A broad future use can be a feature of a model, but it increases the need to articulate the objective of the development and the role of each category of data within it.

That purpose must be connected to a collection boundary. Data minimisation does not ban large training datasets. It requires personal data to be adequate, relevant, and limited to what is necessary for the stated purpose. The useful comparison is not between a small collection and a large collection in the abstract. It is between the proposed data and a less intrusive alternative that could achieve the same objective with similar effectiveness.

This changes the first question from “How much can the crawler find?” to “What must it find, and why?” A project seeking to improve a narrowly defined language or vision capability may be able to exclude sources, categories, time periods, or direct identifiers. It may be able to use synthetic or pseudonymised data for some tasks. It may decide that contributor-provided material is preferable when the intended use requires a clear relationship with the people represented. These options may reduce coverage, cost more, or change model performance. Those are real trade-offs, but they are trade-offs that can be evaluated. Treating all potentially useful data as necessary avoids the decision rather than resolving it.

The next question is source context. A freely visible article, an image on a public blog, a login-limited discussion, a professional forum, and a page protected by CAPTCHA are not interchangeable sources. The EDPB identifies the nature of the website, the type and audience of publication, technical restrictions, the relationship between the person and the controller, and characteristics such as being a minor or public figure as relevant to reasonable expectations.

Anti-scraping signals belong here. Robots.txt, ai.txt, CAPTCHA, login barriers, audience restrictions, opt-out requests, and image-level response tags can indicate that automated collection is opposed, limited, or not what ordinary users experience. They do not amount to consent when absent, and they do not answer every legal question when present. They do change the factual record. A controller that ignores a clear restriction has a harder explanation for why the person or site should have expected the new use, especially when the same purpose could be served by a different source.

The distinction between source context and permission is crucial. A site that tells users their freely accessible content may be scraped for AI development may support an expectation analysis more than a site that blocks scraping and expressly rejects AI use. Neither fact automatically establishes a lawful basis, necessity, or permission to process special-category data. Context is an input to the decision, not the decision itself.

The CommonPool lifecycle shows why the source question cannot be separated from the artifact question. CommonPool contains 12.8 billion image-text samples assembled from Common Crawl snapshots from 2014 to 2022. Its process extracted image URLs and alt-text, downloaded images, applied filters and face annotations, deduplicated results, and released a URL-caption table with metadata. A user could later run a downloader to retrieve image assets from those URLs.