Google and AI rivals back common controls for training crawlers

Website publishers would gain a shared way to restrict collection of their content for generative AI training under a proposal backed by Google, OpenAI, Microsoft and Anthropic. All four companies have committed to respecting the directive, which remains under consideration by IETF, the internet standards organization.

Implementation schedules differ. Anthropic has begun honoring the directive, and Microsoft said its support would start on February 1, 2027. Google expects to introduce support in early 2027, while OpenAI has also promised to implement it.

The proposal was written by Kevin Kelley, Anthropic's technical lead for standards. Google and OpenAI said the current system requires website operators to identify crawlers separately and create rules for individual providers. The proposed approach instead gives participating companies a common instruction in a site's robots.txt file.

Publishers could block training-related collection while making exceptions for selected content, such as a press kit. The directive would neither halt every kind of crawling nor keep content out of Google search. Google said it would apply the instruction to Google-Extended, its crawler for collecting Gemini training material. The restrictions concern future collection; they cannot reverse scraping that has already happened.