How to Narrow a Crawl Before It Becomes a Dataset

The practical problem is to turn a defensible AI purpose into a dataset that collects less, preserves enough provenance to be understood, and remains accurate enough for its intended use. This requires decisions before the first page is fetched as well as controls after material has been collected. Readers should be able to distinguish upstream minimisation—preventing unnecessary data from entering—from downstream cleanup, which reduces exposure but cannot fully recreate a narrower original collection.

The EDPB does not interpret data minimisation as a ban on large AI datasets. It interprets it as a requirement to collect only personal data that are adequate, relevant, and limited to what is necessary for the purpose. That distinction matters. A large dataset may be justified for a defined objective, but its size does not justify every field, source, or category that happens to be available.

Let purpose decide what the crawler may see

A collection purpose becomes operational when it produces inclusion and exclusion criteria. “Use public web data to improve language capability” does not tell a project which domains, languages, topics, dates, or data fields it needs. A more bounded purpose can ask whether the model needs full pages or only a particular kind of text, whether names and profile links are relevant, and whether current material is enough. The point is not to find a perfect formula before work begins. It is to make the reason for each major category of collection visible.

The EDPB recommends a set of actions before collection: consider synthetic data instead of personal data; define precise collection criteria; map and inventory the data; apply filters that exclude unnecessary categories; and exclude categories of websites that structurally contain vulnerable or sensitive information when those categories are not needed. A site mainly used by minors may create a different risk from a site whose subject matter routinely contains financial or location information. Excluding such a source category can be more effective than trying to identify every sensitive record after a broad crawl has already captured it.

This is the difference between a boundary and a hope. A boundary tells the crawler where not to go and what it should not retain. A hope says that an expansive collection will be made safe later. Post-collection filtering is useful, but it operates after information has crossed the first boundary and may already have been copied, stored, or made available to other organisations.

Source signals also belong in the pre-collection decision. The EDPB recommends excluding websites that clearly oppose scraping through technical measures such as authentication, robots.txt, ai.txt, or CAPTCHA. These signals do not amount to consent when absent, and following or ignoring one does not by itself resolve every legal question. They do provide relevant evidence about the source context and whether the project’s planned access is consistent with the stated collection boundary and the expectations of affected people.