Skip to main content

Google and AI rivals back common controls for training crawlers

Anthropic already supports the directive, while Microsoft and Google plan to introduce support in 2027.

By Mei Ling Tan

Illustration of website code and a web crawler
Photo: Yahoo

Website publishers would gain a shared way to restrict collection of their content for generative AI training under a proposal backed by Google, OpenAI, Microsoft and Anthropic. All four companies have committed to respecting the directive, which remains under consideration by IETF, the internet standards organization.

Implementation schedules differ. Anthropic has begun honoring the directive, and Microsoft said its support would start on February 1, 2027. Google expects to introduce support in early 2027, while OpenAI has also promised to implement it.

The proposal was written by Kevin Kelley, Anthropic's technical lead for standards. Google and OpenAI said the current system requires website operators to identify crawlers separately and create rules for individual providers. The proposed approach instead gives participating companies a common instruction in a site's robots.txt file.

Publishers could block training-related collection while making exceptions for selected content, such as a press kit. The directive would neither halt every kind of crawling nor keep content out of Google search. Google said it would apply the instruction to Google-Extended, its crawler for collecting Gemini training material. The restrictions concern future collection; they cannot reverse scraping that has already happened.

Keep reading: Tech News

All Tech News stories

Latest in Technology

All latest
  1. Tech News

    Etsy defends AI policy changes after sellers leave platform

  2. Tech News

    Oracle says funding is in place for OpenAI infrastructure contract

  3. Tech News

    Google Search adds Bond-themed effect for Robert Pattinson

  4. Tech News

    DeepMind philosopher urges caution over creating conscious AI

  5. Tech News

    Researchers claim AI-assisted Navier-Stokes proof without external force

  6. Gadgets

    Gigabyte’s compact 1600W power supply goes on sale in China