Robots.txt Is a Signal, Not a Consent Form

For web scraping used in generative-AI training, anti-scraping signals are part of the evidence about source context, not a shortcut to a legal answer. This chapter explains what robots.txt, CAPTCHA, login barriers, audience restrictions, opt-outs, and related controls communicate; how they affect reasonable expectations and collection choices; and how to read CommonPool’s access failures without mistaking a current response for a complete history. The practical goal is to decide when a source should be excluded, narrowed, documented, or revisited.

A useful distinction comes first. A signal can tell a crawler—or a human reviewer—that a site host limits automated access, expects an action before access, restricts an audience, or does not want a particular reuse. It does not, by itself, establish consent, a lawful basis, or the absence of personal data. The same is true in the other direction: a missing signal does not prove that people agreed to have their information collected for model training.

What a signal says—and what it does not say

The most familiar signal is robots.txt, a file through which a site can communicate instructions about what web crawlers may access. The EDPB treats robots.txt as one of the technical measures that can clearly oppose scraping, alongside robots authentication, ai.txt, and CAPTCHA. A CAPTCHA requires an action intended to be completed by a human. A login barrier limits access to an authenticated audience. An audience restriction on a platform can communicate that visibility is not the same as unrestricted reuse.

These mechanisms are related but not identical. A login requirement says something about who may enter the service. A CAPTCHA says something about whether automated access is welcome. A robots instruction says something about a site host’s preference for crawler access. An opt-out list says that a person or site wants an existing or future processing activity to stop. A privacy notice can explain intended uses and expectations. Treating all of these as one “permission setting” hides the facts that a controller needs to assess.

The distinction between site-level and image-level signals is especially important for a dataset built from URLs. The research paper describes X-Robots-Tag as a header that a site host can attach to an individual image response. It differs from the robots.txt protocol exposed at the main webpage or site level. A crawler may respect a tag attached to one image while ignoring, failing to observe, or not being configured to process a broader site-level instruction. A record of the response to one URL therefore does not necessarily describe the site’s overall position.

The EDPB’s legal point is deliberately more nuanced than “follow the file and you are safe.” First, the absence of a robots.txt file does not amount to consent under the GDPR. A site may never have created the file, may not know how it works, or may communicate its expectations elsewhere. Second, the presence of robots.txt, CAPTCHA, a login barrier, or another restriction can matter substantially. The EDPB recommends excluding websites that clearly oppose scraping through such measures, and it says people are less likely to reasonably expect their information to be scraped when the site uses them.

“Reasonable expectations” are one part of a broader balancing exercise. They ask what a person could reasonably anticipate from the source and the relationship around it. A person who publishes an article on a freely accessible platform that tells users their content may be scraped may be more likely to expect third-party AI use. A person who posts within a service that blocks scraping, uses CAPTCHA, or expressly rejects use for AI development is less likely to expect that use.