Public on the Web Is Not Permission to Train
Web scraping for generative-AI training is often described as a technical way to gather material that is already public. The more important question is not whether a browser can reach a page, but whether turning that page into personal-data input for a model is lawful and defensible. This series builds a way to answer that question by following information from its original context through collection, dataset construction, and model use.
That distinction matters because “public” compresses several different ideas into one convenient word. A page may be visible without a login. A person may have placed information there deliberately, or someone else may have posted it about them. A site may welcome ordinary reading while signalling that automated collection is not wanted. A crawler may copy the material for one purpose, a dataset curator may combine it with billions of other records, and a model developer may use the result for a purpose no one could have identified when the page was published. Technical accessibility is real, but it does not answer all of those questions.
The shortcut hidden in the word “public”
The European Data Protection Board’s 2026 guidelines start with a practical boundary: the GDPR applies to web scraping when the activity includes processing personal data. Processing is not limited to the instant a crawler fetches a page. Extraction, cleaning, organising, structuring, storing, and retrieving information can all be part of the activity. Personal data includes information that directly identifies someone, such as a name or contact detail, and information that can identify them indirectly when combined with other information.
That makes the subject broader than a list of obviously private records. A photograph of a face, a resume, an email address, a location, or a description connecting a person with a sensitive characteristic may all matter. So may information that looks harmless in isolation but becomes identifying when it is placed beside an image, a URL, a caption, or metadata. The scale of generative-AI collection intensifies the problem: a controller may process information about millions of people without knowing exactly which people or categories of information are present.
The legal questions therefore begin before anyone asks whether a model performs well. What is the purpose of collection? Which personal data is necessary for that purpose? What did the source context communicate about audience and reuse? Which organisation decides the essential means? What lawful basis supports the processing? What happens if a person objects, asks for deletion, or discovers that the model can reproduce or infer information about them?
The answers cannot be supplied by consent-by-default. The EDPB states that putting personal data online does not, by itself, mean that the person consented to its being scraped for a particular purpose. The absence of a robots.txt file is not consent either. Conversely, a technical restriction is not a complete legal analysis: it is evidence about the site’s preferences and the expectations that a controller should consider. The same URL can be technically reachable while still carrying a context that makes unrestricted reuse difficult to justify.
This is why the central claim is not that every use of public web data is unlawful. The stronger and more useful claim is that accessibility is only one fact in a larger judgment. Under the GDPR, a controller must still identify a lawful basis and respect principles such as purpose limitation, transparency, data minimisation, fairness, and accuracy. A public page may be a source; it is not automatically a permission slip.
The arrows are not meant to imply that information always travels in one neat direction. They show the object that must be judged. Controls can act before collection, during curation, or after training, but a late control may not undo an earlier copy. A model output can reveal a failure that requires a change to collection or filtering. The whole chain is the relevant unit because each stage changes what the next actor knows, holds, and can still remove.
What CommonPool changes in the question
The case study in the accompanying research paper is DataComp CommonPool, a large image-text dataset assembled from the web. The dataset contains 12.8 billion samples. Its source material came from Common Crawl snapshots collected between 2014 and 2022. The curation process gathered web snapshots, extracted image URLs and alternative text, downloaded images, applied toxicity filters and face detection, deduplicated the results, and released a table of URL-caption pairs with additional metadata rather than releasing all image files directly.
That release format is important. A URL table may look less consequential than a frozen archive of images, but it can still create a route to fresh processing. A user who downloads the table can run a tool that retrieves the images from their URLs. The artifact is therefore not merely a passive catalogue. It can serve as a central point through which personal information is distributed to dataset users, model developers, and later model users. If the original site changes, the next downloader may receive a different result; if an earlier user already copied the image, that earlier copy does not disappear.
The paper’s audit shows why this architecture matters. The authors downloaded a small-scale version in April 2025 containing 12.8 million randomly selected samples—about 0.1 percent of the full dataset. They used optical character recognition, named-entity and personally identifiable information detection, URL analysis, and manual inspection. The purpose was not to claim that every sample contained a privacy problem. It was to establish evidence that personal information remained present despite curation and to estimate some categories at the scale of the full dataset.
The findings span several forms of information. Captions and text extracted from images included names linked with descriptions of race, ethnicity, religion, or sexual orientation. The audit found images of identity documents, including credit cards, driver’s licences, Social Security numbers, passports, and birth certificates. It also found resumes associated with people who had a public online presence. From the sampled results, the authors estimated that at least 136,000 images in the full dataset depicted resumes linked to such individuals; their reported 95 percent interval ranged from roughly 136,000 to 200,000.
Other findings expose why a single notion of “the data” is inadequate. The URL investigation found children’s names, faces, birth certificates, passports, and health-related material in some samples. Metadata attached to images included timestamps, names, and precise geolocation information. The researchers also examined failed downloads. In their April 2025 sample, 21.4 percent of image URLs failed to download, and 19.0 percent of those failures were attributed to a lack of access permissions. Yet the captions, URLs, and annotations for those entries remained available. An image becoming inaccessible today did not erase the surrounding record or necessarily affect copies acquired earlier.
The face-blurring result is especially instructive, but it must be read with the paper’s qualifications. CommonPool supplied face-detection annotations so that a downloader could blur detected faces by default. In a random evaluation of 100,000 samples, the authors used another face detector to find images that the original detector missed. After manual review, they estimated that between 100 million and 119 million images in the full dataset contained human faces not covered by the default face-obfuscation annotations, with at least 100 million as the headline lower estimate. The result does not prove that every missed face is identifiable or that every blurring system performs this way. It does show how a residual rate that looks small in a sample can become a very large number at web scale.
The paper also reports that a blurred face may not end the identification problem. A caption can name the person. A location can narrow the possibilities. Exif data can expose a photographer or a precise place. A released bounding box can create another route to the underlying face. Automated detection adds its own uncertainty: the researchers observed both false positives and false negatives, and they focused primarily on English-language text. Their audit was large enough to produce useful evidence, but it was still a random subset rather than a complete census.
Those limits strengthen rather than weaken the main lesson. The audit can support the conclusion that personal information is present in CommonPool and that the default mitigations did not establish reliable absence. It cannot support the conclusion that every web-scraped dataset has the same prevalence, that every detected item falls into the same legal category, or that every downstream model was trained on CommonPool. Evidence of presence is not evidence that every item is present; an imperfect detector is not proof that an undetected item is safe.
The chain we need to judge
The EDPB’s guidance and the CommonPool audit meet at a point that is easy to miss if law and engineering are discussed separately. A controller does not assess only a crawler’s request or only a model’s final output. They assess a sequence in which purpose, knowledge, and control can change.
At the source stage, the relevant facts include the kind of website, the audience for the publication, whether a login or other barrier was used, whether the person or site communicated restrictions, and whether the information was posted by the data subject at all. At the collection stage, targeted criteria can narrow what is taken, while untargeted crawling can follow discovered links and greatly reduce the controller’s knowledge of what personal data enters the system. At the dataset stage, cleaning and deduplication may reduce exposure but can also create new metadata, annotations, indexes, and copies. At the model stage, memorisation, reproduction, inference, or profiling may create consequences that were not visible in the original page.
Responsibility follows the facts through those transitions. The organisation performing the scraping is not automatically the only controller. An AI developer that instructs a scraper about the sources and categories may determine the purposes and essential means. A scraper and an AI developer may be jointly responsible if they decide those purposes and means together. An organisation that obtains an already scraped dataset generally has its own processing to analyse. The physical location of a file does not answer who decided why personal data was processed.
This also explains why the phrase “we only used public information” is too weak to carry the argument. It does not say whether the collection was necessary, whether the source context supported the new purpose, whether people could reasonably expect the use, whether special categories of data were collected incidentally, or whether a person can exercise their rights once the information has moved through several systems. It does not say whether a model can retain data after the source is changed. It describes one access condition and leaves the rest unanswered.
A defensible judgment begins with a narrower question: what exactly is this system trying to accomplish, and what personal data is genuinely needed to accomplish it? The EDPB says large volumes are not forbidden simply because they are large. Minimisation instead requires the controller to avoid personal data that is unnecessary, irrelevant, or excessive for the intended purpose. That distinction leaves room for substantial datasets while rejecting the assumption that everything potentially useful should be collected first and justified later.
The practical implication is demanding but clear. Public access should trigger investigation, not automatic approval. A team needs to understand the source context, record what it collected and when, screen out categories and sites that it does not need, respect meaningful restrictions, and test whether its filters work across text, images, URLs, and metadata. It must also consider what happens after release and training, when deletion becomes harder and a single inclusion can travel far beyond the original page.
The next question is therefore not whether CommonPool contains a surprising item. It is how a technical lifecycle turns such items into legal and operational decisions: which rules apply, what evidence can support a lawful basis, and when a safeguard reduces risk without pretending to erase it. That is the point at which “public” stops being a description of visibility and becomes a question about context, purpose, and responsibility.