AI Training Opt-Out in robots.txt: The New Shared Token
On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor one shared robots.txt token, __usage_ai-training, as an AI training opt-out: one rule group tells every supporting company not to train generative models on a site, instead of one rule per AI company. Publishers have spent three years adding crawler names like GPTBot, Google-Extended and ClaudeBot one by one, so a single line would save real work. It will take a while to arrive, though: only Anthropic says it has started implementing the signal, Microsoft starts on February 1, 2027, Google targets early 2027 and OpenAI says "the coming months."
Key takeaways
- A robots.txt group with
User-Agent: __usage_ai-trainingandDisallow: /states a preference against AI training of your content (IETF draft). - The token does not stop crawling. Normal crawler rules still decide who may fetch what.
- Google and Microsoft both say the training preference does not change eligibility for Google Search or Bing search results.
- Existing
User-Agent: *groups express no training preference under the draft, so current robots.txt files say nothing about training until you add the token. - The token is defined in an individual Internet-Draft, not a finished IETF standard, and each company has its own rollout date.
What the shared AI training opt-out is
__usage_ai-training is a robots.txt policy token that states whether a site's content may be used to train generative AI models. It comes from an Internet-Draft titled "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley of Anthropic and published on October 30, 2026. It is an individual draft: the IETF has not endorsed it, and it expires on May 3, 2027 unless revised (IETF Datatracker).
The draft builds on work in the IETF AI Preferences (AIPREF) working group, which is writing a shared vocabulary for how content may be used. The meaning of "AI training" comes from that group's vocabulary draft, draft-ietf-aipref-vocab-08.
Instead of inventing new robots.txt directives, the draft reuses the existing User-Agent line with a "policy token." A policy token names a use of content rather than a crawler. __usage_ai-training is the first one, and any crawler that honors it reads the group the same way.
Who supports __usage_ai-training and when
Each company published its own announcement, and the dates differ.
| Company | Status (Oct 30, 2026) | Start date | Existing control that keeps working |
|---|---|---|---|
| Anthropic | "Has already started implementing support" | Rolling out across services over time | ClaudeBot |
| Microsoft | Committed | February 1, 2027 | Bing page-level training and grounding settings |
| Planned | Early 2027 | Google-Extended | |
| OpenAI | Planned | "In the coming months" | GPTBot |
| Meta | Supports the idea, no commitment | None | Not stated |
Sources: Anthropic on the IETF ai-control list, Microsoft Bing blog, Google crawler documentation, OpenAI and SiliconANGLE, which also reported Meta's position.
Anthropic
Kelley announced the draft to the AIPREF mailing list and wrote that Anthropic "has already started implementing support for this draft," adding that it "complements our existing crawler controls rather than replacing them." He also said rollout across Anthropic's services will take time (IETF mail archive). ClaudeBot remains the crawler to block if you want to keep future content out of Anthropic's training data (Anthropic support).
John Mueller's post on the Google Search Central blog says Google aims to support the control "along with other AI providers in 2027." Google's crawler documentation is more specific: it intends to support the signal in early 2027 (Google Search Central).
Google also explained how the token maps onto its own control. If a site opts out with the shared token and does not mention Google-Extended, Google will treat it as if the site had disallowed Google-Extended. Google-Extended covers more than training. It also controls grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, so a shared opt-out can remove a site from those answers too. Google-Extended does not affect inclusion or ranking in Google Search (Google crawler docs). Gemini models, including newer releases like Gemini 4 Carbon, fall under this control.
Google also ties the move to the EU Code of Practice for general-purpose AI model providers, which it says "recognizes robots.txt as a primary method for web publishers to express machine-readable preferences."
Microsoft
Microsoft published the most detail. Its Bing post adds two separate preferences, AI Training and AI Grounding, on this timeline (Bing blog):
- December 15, 2026: temporary grounding preferences open in Bing Webmaster Tools for sites that use
NoArchiveorNoCacheon that date. - February 1, 2027: Microsoft starts honoring the new preferences as it crawls.
- March 1, 2027:
NoArchiveandNoCachestop being read as training or grounding preferences. - September 30, 2027: the temporary Webmaster Tools preferences expire, unless a page is recrawled first.
Training can be set in robots.txt with the shared token or per page with a Bingbot robots meta tag or an X-Robots-Tag header set to ai-training:disallow. A page-level setting overrides robots.txt. Grounding is page-level only, with three values: Allow (the default), DisallowExtended (only Microsoft's own experiences such as Copilot may use it) and Disallow.
Microsoft's framing: "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work." It also warns that "neither choice guarantees inclusion, traffic, or payment."
OpenAI
OpenAI said it will support the shared signal in the coming months and publish implementation details once support is live (OpenAI). Until then GPTBot is the control. OpenAI's crawler documentation says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models," and that each of its crawler settings is independent, so you can block GPTBot and still allow OAI-SearchBot for ChatGPT search (OpenAI crawler docs).
How to add the AI training opt-out to robots.txt
The draft's own example blocks training everywhere except a press kit folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
A disallowed path means "I prefer this content not be used for AI training." An allowed path means the opposite. To opt out of training for the whole site, two lines are enough:
User-Agent: __usage_ai-training
Disallow: /
Matching follows RFC 9309, the robots.txt standard (IETF draft):
- The token must be spelled exactly, though case does not matter.
- As with normal robots.txt rules, the longest (most specific) matching path applies.
- A
User-Agent: *group must not be read as a training preference. - If no group names the token, the site has expressed no preference at all.
- A crawler that does not know the token treats it like any unknown user agent and keeps applying its other rules.
- A private agreement between a site and an AI company can override the published preference.
Keep your existing crawler rules
Do not delete GPTBot, ClaudeBot or Google-Extended rules yet. Every company says its own controls keep working, and none of them reads the shared token today except Anthropic, which is still rolling it out. The safest robots.txt for the next few months has both: the shared token and the vendor-specific groups you already use.
The token does not control crawling either. If you want a bot to stop fetching your pages, block its crawler name, as before. The draft asks crawlers not to fetch content they cannot use, but nothing obliges them to.
Why one token matters
Opting out of AI training today means keeping a list. You add GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Gemini, CCBot for Common Crawl, and so on, and the list goes stale every time a new crawler appears.
A June 2026 sample of 1,000 top domains on Wikimedia Commons shows the pattern: GPTBot was the most-blocked named bot, at roughly 200 sites, followed closely by CCBot, ClaudeBot and Google-Extended (Wikimedia Commons). Each of those lines does the same job for a different company. The shared token replaces that per-vendor guessing with one statement about use, which any new AI company can read without the site owner knowing its crawler's name.
The chart also shows the largest bar is the wildcard group. Under the new draft those groups say nothing about training, which is deliberate: a site that wrote User-Agent: * years ago to manage search crawlers never meant to make an AI training decision.
What the token does not cover
The token covers training only. When an AI assistant fetches a page to answer a question, other controls apply: Microsoft's grounding setting, Google-Extended for Gemini grounding, or crawler-specific rules such as OpenAI's OAI-SearchBot and ChatGPT-User.
It also says nothing about data already collected. None of the announcements we could read says the signal applies to content gathered before a site adds it, and Anthropic's support page describes ClaudeBot blocks as excluding a site's future materials.
Support is voluntary. Meta backed the idea but made no commitment, and smaller crawlers have no obligation at all. Like everything else in robots.txt, the token is a request; WAF rules or bot management are still the only way to actually stop a crawler.
How it compares to Cloudflare's Content Signals
Cloudflare launched a similar idea in September 2025: a Content-Signal line in robots.txt with search, ai-input and ai-train values, for example Content-Signal: search=yes, ai-train=no. Cloudflare added search=yes, ai-train=no by default to more than 3.8 million domains using its managed robots.txt (Cloudflare blog).
__usage_ai-training | Cloudflare Content Signals | |
|---|---|---|
| Syntax | A User-Agent group with Allow/Disallow paths | A new Content-Signal line |
| Scope | AI training only | Search, AI input (grounding) and AI training |
| Committed readers | Anthropic, Microsoft, Google, OpenAI | Voluntary; Cloudflare notes some companies may ignore it |
| Standards track | Individual IETF draft | CC0 policy published by Cloudflare |
The two can live in the same file. If your site sits behind Cloudflare, check whether you already serve a Content-Signal line before adding the token, so the two do not contradict each other.
Frequently asked questions
Does __usage_ai-training remove my site from Google or Bing search?
No. Google says Google-Extended, which the token maps to, does not affect inclusion or ranking in Google Search. Microsoft says its training and grounding preferences do not change eligibility for Bing search results. Grounding in Gemini Apps can be affected, though, because Google-Extended also covers that.
Is the shared AI training token in effect today?
Only partly. As of October 30, 2026, Anthropic says it has started implementing support. Microsoft begins honoring it on February 1, 2027, Google in early 2027, and OpenAI "in the coming months." Until then, keep your existing GPTBot, ClaudeBot and Google-Extended rules.
Does User-Agent: * with Disallow: / opt me out of AI training?
Not under the new signal. The draft says wildcard groups must not be read as a training preference, so you must name __usage_ai-training explicitly. A wildcard block does still stop compliant crawlers from fetching your pages, which is a different thing.
Is __usage_ai-training an official IETF standard?
No. It is defined in an individual Internet-Draft by Kevin Kelley of Anthropic, dated October 30, 2026, and the IETF has not endorsed it. It is offered as input to the AIPREF working group, and Microsoft says it will keep participating in that work toward a final standard.
Can I allow training on some pages and block it on others?
Yes. Use normal robots.txt paths inside the __usage_ai-training group, for example Disallow: / plus Allow: /press-kit/. The longest matching path wins. On Bing, a page-level ai-training meta tag or header also overrides robots.txt.
Sources
- IETF Datatracker: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)
- IETF ai-control mailing list: Kevin Kelley, "New draft: draft-kelley-aipref-training-preference-signal-00"
- Google Search Central Blog: Supporting a shared AI training control for websites
- Google crawler documentation: Google-Extended
- Microsoft Bing Blog: Your content, your choice: clearer preferences for training and grounding
- OpenAI: Helping websites express AI training preferences
- OpenAI: Overview of OpenAI crawlers
- Anthropic support: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- SiliconANGLE: Google, Anthropic and others to support new AI crawling opt-out standard
- Cloudflare Blog: Content Signals Policy
- Wikimedia Commons: Blocked robots according to robots.txt, June 2026
