Insights

robots.txt AI training opt-out: one token for Google, OpenAI, Microsoft and Anthropic

Alexandre

Oct 30, 2026

On October 30, 2026, Google, OpenAI, Microsoft and Anthropic said they will honor the same robots.txt line telling them whether your content may be used to train their generative AI models. In practice, it is a shared robots.txt AI training opt-out that replaces a separate rule for every AI provider. But this signal does not control crawling, and it does not always protect your visibility in AI answers. Here is what advertisers need to know, and what to do about it.

Key takeaways

The __usage_ai-training token goes where a crawler name would normally sit, on the User-Agent line of robots.txt.
Disallow signals a refusal of AI training, Allow signals consent, and each directory can carry its own rule.
Without this group, no preference is expressed and training remains possible. A User-Agent: * group does not count.
The token does not block crawlers and does not remove content already used in existing models.
At Google, an opt-out is treated as a Google-Extended block, which also affects Gemini answers.
Timeline: Anthropic has started implementing it, OpenAI will follow in the coming months, Microsoft on February 1, 2027 and Google in 2027.

What is the __usage_ai-training token?

__usage_ai-training is a “policy token” defined in a draft standard submitted to the IETF on October 30, 2026 by Kevin Kelley, an engineer at Anthropic, as part of the AI Preferences (AIPREF) working group. It does not name a crawler. It names a use: generative AI training, meaning the use of content to modify the learned parameters of a generative model. Models used only for ranking or classification are out of scope.

The robots.txt syntax itself does not change. The example in the draft refuses training across the whole site, except for the press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Status: this is an individual Internet-Draft, valid until May 3, 2027. It is not yet an IETF standard and may still change.

Who supports it, and when?

Provider	Commitment	Announced timeline
Anthropic (Claude)	Author of the draft, implementation under way	No date given for full support
OpenAI (ChatGPT)	Will honor the signal, GPTBot rules still apply	“In the coming months”
Microsoft (Bing, Copilot)	Will honor the signal, adds page-level preferences	From February 1, 2027, as pages are crawled
Google (Gemini)	Will honor the signal, Google-Extended still applies	2027 (early 2027 according to its crawler documentation)

Until then, each provider’s own controls (GPTBot, ClaudeBot, Google-Extended) remain in force and will keep working alongside the shared token.

What this token does not do

It does not block crawlers. It is a usage preference, not a crawl directive. Bots can keep visiting your pages and using your bandwidth. To limit crawling, you still need to target each crawler by name (GPTBot, ClaudeBot and so on).
It is not triggered by the wildcard. A site that already blocks every bot with User-Agent: * has, under the draft, said nothing about training.
It does not erase the past. Content already included in training datasets or existing models is not removed.
It does not affect rankings. Google and Bing search rankings are unchanged, and appearing in AI Overviews or AI Mode depends on Google Search settings, not on this token.

The Google trap: your visibility in Gemini

Google has said that if a site opts out of training with the shared token without explicitly mentioning Google-Extended, it will read that as a Google-Extended block. But Google-Extended does not only cover Gemini training: it also governs whether your pages can be used to ground answers in Gemini Apps and Vertex AI. A site-wide Disallow: / can therefore take you out of Gemini answers, along with the clicks they bring.

Conversely, Google says sites that explicitly allow Google-Extended remain eligible to appear and earn clicks in Gemini Apps.

The Bing deadline: NoArchive and NoCache will no longer be enough

Microsoft is using the announcement to split two preferences. Training is set with the shared token in robots.txt, or page by page through a meta tag or HTTP header. Grounding, meaning the use of your pages in Bing and Copilot answers, is set page by page. Both default to allow, and Bing rankings are unaffected. You can therefore refuse training while still being cited in Copilot.

Date to remember: after March 1, 2027, Bing will no longer read NoArchive and NoCache tags as AI preferences. If your site relies on them to stay out of Copilot, publish the new preferences before that deadline. A transition setting opens in Bing Webmaster Tools on December 15, 2026.

Advertisers: should you block AI training?

For most brands, no. Your product pages, brand pages and buying guides are exactly what lets AI models learn about your offer and recommend it. Refusing training across your whole site deprives AI of your own brand message, and at Google it risks removing you from Gemini.

The token becomes useful to protect high-value proprietary content: studies, industry barometers, white papers, customer-only content. Best practice is to target those directories rather than the whole site:

User-Agent: __usage_ai-training
Disallow: /studies/
Disallow: /white-papers/

The rest of the site stays open to training. At Google, those directories alone will also drop out of Gemini answers: that is the price of protection, to be weighed content by content.

Three actions to take now

Audit your robots.txt and tags. Which AI crawlers are you blocking today, since when, and why?
Write your policy in one sentence. Which content do you want AI to cite, and which do you want to protect?
Keep your current rules until the switch. Keep GPTBot, ClaudeBot and Google-Extended rules until each provider actually reads the token, and migrate away from NoArchive and NoCache before March 1, 2027.

Frequently asked questions

Is the __usage_ai-training token already active?

Not everywhere yet. Anthropic has started implementing it, Microsoft will read it from February 1, 2027, Google in 2027 and OpenAI in the coming months. The text is still an IETF draft with no formal standard status.

Does opting out of training remove me from ChatGPT or Copilot answers?

No, the token only covers training. At OpenAI, visibility in ChatGPT search depends on the OAI-SearchBot crawler. At Microsoft, it depends on the grounding preference. The exception is Google: an opt-out without an explicit Google-Extended rule also affects Gemini Apps.

Is “User-Agent: *” followed by “Disallow: /” enough to refuse training?

No. The draft states that the wildcard does not apply to this token. Without a __usage_ai-training group in the file, no preference is expressed.

Will content already used be removed from models?

No. The signal applies to future collection. It removes nothing from existing training datasets or models.

Before you touch your robots.txt, find out what AI engines already cite from your site. Affilae GEO measures your brand’s presence in ChatGPT, Gemini and Perplexity, identifies the sources that cite it and activates the publishers that matter. Request your free GEO audit.