Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping
Reported by TechRadar1 source ↓
One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.
Key points
- The signal governs training use, not whether bots may crawl a site
- Anthropic has begun; OpenAI plans implementation in the coming months
- Microsoft plans to honor it from February 1, 2027; Google is aiming for 2027
- The signal defaults to allowing training and does not affect content already collected
- The IETF has not endorsed the draft as a formal standard
Details
TechRadar reports that Google, Microsoft, OpenAI and Anthropic have agreed to honor a shared robots.txt preference for AI training. The proposal uses the token "__usage_ai-training" in a robots.txt group, replacing the need for sites to specify separate training preferences for each participating company’s crawler.
The signal concerns how fetched content may be used, not whether a bot can fetch it. Sites that want to block crawling still need to use existing per-bot rules, and the shared preference does not apply to material already collected. It defaults to allowing training unless a site opts out.
Anthropic says it has started implementing the signal. OpenAI plans to implement it in the coming months without giving a date; Microsoft says it will honor the preference from February 1, 2027, and Google is targeting 2027. The proposal is an individual Internet-Draft that the IETF has not endorsed, and it is due to lapse on May 3, 2027 unless revised.
Written from coverage as of
Images
- TechRadar
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh