Web Scraping for AI Training

Overview

This series follows personal data from a public page through scraping, curation, release, model training, and output. It combines the CommonPool audit with EDPB 2026 guidance to examine purpose, necessity, expectations, lawful basis, safeguards, provenance, and remedies. Public accessibility is only source context: a defensible program narrows collection, measures residual risk, and may proceed, redesign, substitute, or stop.

Learning outcomes

Public accessibility does not by itself establish consent or a lawful basis for AI training.
Collection, curation, release, retrieval, training, and output form one accountability chain.
The CommonPool audit found personal information survived filtering, but its sample-based estimates do not describe every dataset or item.
A defensible program defines purpose, tests necessity and balance, preserves provenance, and can narrow, substitute, or stop.

Keywords

web scrapinggenerative AIGDPRCommonPoolpersonal dataprivacydata minimisationprovenancerobots.txtlegitimate interest

Chapters

Public Web Data for AI Training

Why accessibility does not settle lawful use

12 min · Why accessibility does not settle lawful use

12 min

How a Public Page Becomes an AI Dataset

The state changes behind collection and reuse

13 min · The state changes behind collection and reuse

13 min

DataComp CommonPool Audit

What survived filtering—and what the sample proves

15 min · What survived filtering—and what the sample proves

15 min

GDPR Tests for AI Web Scraping

Purpose, necessity, balance, and Article 9

15 min · Purpose, necessity, balance, and Article 9

15 min

Robots.txt and AI Scraping

What access signals can—and cannot—prove

13 min · What access signals can—and cannot—prove

13 min

AI Dataset Cleanup Failures

Why deletion and retraining reach different layers

16 min · Why deletion and retraining reach different layers

16 min

Defensible AI Scraping Decisions

When to proceed, narrow, substitute, or stop

16 min · When to proceed, narrow, substitute, or stop

16 min

Preview every chapter · Unlock full text, audio, and video from chapter 4

Plans

Ch.1-3 FreeCh.4-7 Premium

FAQ

Is public web data automatically legal to use for AI training?

No. Public visibility does not by itself establish consent or a lawful basis; the controller must still assess purpose, necessity, expectations, transparency, minimisation, and safeguards.

What did the CommonPool audit show?

It showed that personal information remained in CommonPool after filtering and release, including documents, sensitive-linked text, metadata, and faces missed by default annotations. Its estimates came from sampled research, not a census.

How should a team decide whether to scrape?

A team should define the purpose and necessary data first, then assess source context, access signals, lawful basis, safeguards, provenance, and downstream effects. The result may be proceeding narrowly, redesigning, substituting another route, or stopping.

Can deleting a dataset item remove it from a trained model?

Not necessarily. Source or dataset deletion may not reach copies or model representations; retraining, output controls, or unlearning have different reach and require verification.