OpenAI, Google, Microsoft and Anthropic said on October 30 that they will back a single robots.txt token that lets website owners opt their content out of AI training across all four companies in one step. The token, written as __usage_ai-training, comes from an Internet-Draft that Anthropic's Kevin Kelley submitted to the Internet Engineering Task Force the same day.
Until now, a publisher that wanted to keep its pages out of training had to block each provider separately, listing crawlers such as OpenAI's GPTBot and Google's Google-Extended one by one. Under the new approach, a site adds one group to robots.txt with the shared token as the User-Agent and a Disallow rule. Allow lines can carve out exceptions, such as a press-kit directory that stays open.
The draft, titled "Machine-Readable Training Preference Signal," adds no new directive names and no new grammar. It reuses the path matching, longest-match precedence and group structure already defined in RFC 9309. A wildcard User-Agent group does not count as a training preference, and a file with no group for the token expresses no preference at all.
Key facts
- 01OpenAI, Google, Microsoft and Anthropic will honor one robots.txt token, __usage_ai-training, to opt sites out of AI training.
- 02Anthropic's Kevin Kelley filed the IETF Internet-Draft on October 30, 2026; it expires May 3, 2027.
- 03Anthropic has begun implementing support; OpenAI targets the coming months and Google targets 2027.
- 04Microsoft's new training and grounding preferences take effect February 1, 2027, replacing NoArchive and NoCache.
- 05The token covers training only and does not remove content already in existing datasets or models.
The four companies are on different timelines. Anthropic has already begun implementing support. OpenAI says it aims to honor the signal in the coming months, and Google said it plans to support it in early 2027 and has updated its common crawlers documentation to say so.
“We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress.”— John Mueller, Google Search Advocate
Google said that if a site uses the shared token to opt out of training without naming Google-Extended, it will treat the site as if Google-Extended were disallowed. Existing company-specific controls keep working alongside the new signal, and sites that explicitly allow Google-Extended stay eligible to appear in Gemini Apps responses.
Microsoft published the most detailed rollout plan. In a Bing blog post, principal product manager Krishna Madhavan said the new training and grounding preferences take effect on February 1, 2027, as pages are crawled, replacing the older NoArchive and NoCache tags. Sites that rely on those tags have until March 1, 2027, to publish new preferences, and none of the choices affect eligibility for Bing search results.
“Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”— Krishna Madhavan, Principal Product Manager, Microsoft Bing
Microsoft is also separating training from grounding, the step that pulls live web content into chatbot answers. Publishers can set grounding page by page to Allow, to DisallowExtended, which limits use to Microsoft-operated products such as Copilot and Microsoft 365, or to Disallow. Microsoft said the two restrictive settings significantly reduce a site's visibility in AI experiences.
The token is a usage preference, not an access control. The draft says it does not direct crawling behavior, and providers that have not implemented it can ignore the group, so existing provider-specific rules should stay in place. Updating robots.txt also does not remove material already collected into existing datasets or trained models.
The document is an individual Internet-Draft, which means it is not endorsed by the IETF and has no formal standing in its standards process. It expires on May 3, 2027. The IETF's AIPREF working group, chaired by Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, counts Meta, Cloudflare, the Common Crawl Foundation, the BBC and Creative Commons among its participants.
Not everyone involved considers the work settled. Nate Hake, who has taken part in the IETF discussions, said "it's not actually done at the ietf level" and that significant disputes remain, and he criticized the current draft for exempting user-initiated actions from the preferences.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
