---
title: "One rule against AI training. What does __usage_ai-training change?"
description: "On 30 October, Google, Microsoft and OpenAI announced they will honor one robots.txt rule: __usage_ai-training. What it does, when it takes effect and why it doesn’t remove your site from search or AI Overviews."
date: 2026-11-01
author: "Ewelina Podrez-Siama"
url: "https://podrez.pl/en/usage-ai-training/"
categories:
- "SEO Blog"
---
As of 1 November 2026 · Google, Microsoft and OpenAI have announced support, Anthropic is rolling it out
On Friday, 30 October 2026, Google, Microsoft and OpenAI announced that they will honor one shared rule in robots.txt. The rule says it plainly: don’t use my content to train generative AI models. Instead of separate entries for GPTBot, Google-Extended or ClaudeBot, two lines are meant to be enough:
```
User-agent: __usage_ai-training
Disallow: /
```
Importantly, this is not a new bot or an access rule. It doesn’t change who can crawl your site, it doesn’t remove it from Google or Bing, and it doesn’t switch off AI Overviews. And for now… nobody has confirmed that they already honor it. Microsoft starts on 1 February 2027, Google “in early 2027”, OpenAI “in the coming months”, and Anthropic has started implementing it without giving a date.
Announcements of 30 October 2026 about the shared \_\_usage\_ai-training ruleWhat was announcedSince when**Google**The rule works like blocking Google-Extended (Gemini training and grounding in Gemini Apps), unless Google-Extended has its own group. Search is unaffected.early 2027**Microsoft**The robots.txt rule, the same preference for individual pages (meta tag and HTTP header) and a separate grounding setting. NoArchive and NoCache will stop controlling training and grounding.from 1 February 2027, as pages are recrawled**OpenAI**Support for the shared rule. Existing robots.txt settings for OpenAI’s crawlers still apply.“in the coming months”**Anthropic**The draft’s author is Kevin Kelley of Anthropic. According to his message on the IETF mailing list, the company has started implementing it, but its help page still describes only ClaudeBot, Claude-User and Claude-SearchBot.no date
**None of the companies has confirmed yet that the rule already works.** Google, OpenAI and Anthropic say their existing crawler controls stay in place. Microsoft only changes the meaning of NoArchive and NoCache. Apple and Meta aren’t part of the announcements, so Applebot-Extended or meta-externalagent still have to be blocked separately.
How does \_\_usage\_ai-training work?
-------------------------------------
Until now, every provider had its own name: GPTBot, Google-Extended, Applebot-Extended, ClaudeBot, meta-externalagent. The list kept growing, and a full training opt-out meant knowing all of them (I keep track of them in my [bot directory](https://podrez.pl/en/bots/)). Kevin Kelley’s draft rolls this into a single entry without any new syntax. It uses an ordinary `User-agent` group, but instead of a crawler name you write a so-called *policy token*. Parsers that don’t know it simply skip it.
- **1**### The wildcard doesn’t cover it
A `User-agent: *` group never applies to the token. Your existing `Disallow` for all crawlers is therefore not read as an objection to training.
- **2**### No group, no preference
If robots.txt has no `__usage_ai-training` group, the site expresses no preference. Microsoft then applies the default value, `Allow`.
- **3**### It works on paths
`Disallow` and `Allow` work as usual, and the longer match wins. You can opt the whole site out and leave, say, a press kit folder open.
- **4**### It doesn’t control crawling
It’s a statement about how content is used, not about access. To keep a crawler out entirely, you still need rules for its name. The draft does, however, allow companies not to fetch content they couldn’t use anyway.
Training is defined narrowly here: changing the parameters of a generative model based on content. Models that only classify or rank don’t count. The definition comes from the vocabulary of the IETF AI Preferences (AIPREF) working group.
Good to know
For now it’s an individual draft (*Internet-Draft*), version 00 of 30 October 2026, not a standard. The IETF notes that it hasn’t endorsed it, and the draft expires on 3 May 2027 unless someone updates it. So the syntax may still change. Microsoft promises to give advance notice of any changes.
Microsoft goes further: training and grounding set separately
-------------------------------------------------------------
Bing adds two things of its own on top of the shared rule. First, the training preference can be set for an individual page with a meta tag or an HTTP header, and it then overrides robots.txt. Second, there’s a separate grounding setting – whether a page may feed AI answers in Bing, Copilot and Microsoft 365. Grounding can only be set at page level; there’s no robots.txt option for it.
Opting a single page out of training, with a meta tag or an HTTP header (one of the two is enough):
```
X-Robots-Tag: Bingbot: ai-training:disallow
```
Grounding limited to Microsoft’s own services, again in two alternative forms. Instead of `disallowextended` you can use `disallow` or `allow`; the values are explained below:
```
X-Robots-Tag: Bingbot: ai-grounding:disallowextended
```
- ### Allow
The default; nothing to add. Content can feed AI answers, with links to the source.
- ### DisallowExtended
Grounding only in Microsoft’s own services (AI mode in Bing, Copilot, Microsoft 365), excluding the grounding services Microsoft provides to other companies, such as Web IQ or the API in Microsoft Foundry.
- ### Disallow
Content doesn’t feed AI-generated answers.
Microsoft says none of these settings affects visibility in regular Bing results. I wrote about grounding in Bing and how to measure it in my piece on [grounding queries in Bing Webmaster Tools](https://podrez.pl/en/grounding-queries/).
### If you use NoArchive or NoCache
These old directives will stop controlling training and grounding in Bing. Once Microsoft detects the new preferences on a site, NoArchive and NoCache no longer count as an AI signal there. The timeline:
1. 1**15 December 2026**Temporary grounding preferences open in Bing Webmaster Tools. They’re available only to sites that use NoArchive or NoCache on that date.
2. 2**1 February 2027**Microsoft starts honoring the new preferences as it recrawls pages.
3. 3**1 March 2027**Deadline to publish the new preferences and record the temporary one in Bing Webmaster Tools. After that, NoArchive and NoCache are no longer read as an objection to training or grounding.
4. 4**30 September 2027**Temporary grounding preferences expire at the latest. Earlier for a given page if Microsoft finds a new preference on it when recrawling.
What about AI Overviews and AI Mode in Google?
----------------------------------------------
That’s a completely different setting in Search Console, available to all sites since 31 August 2026. In the property settings there’s a “Search generative AI” section with two options:
- ### Include (default)
“Include my site’s links and content in Search generative AI features”. The site can appear in AI Overviews, AI Mode and AI features in Discover (including as a link) and get impressions and traffic from them.
- ### Exclude
“Exclude my site’s links and content from Search generative AI features”. Google won’t show the site in these features or use it to ground answers, so there will be no impressions or clicks from them at all.
The “Search generative AI” setting in Search Console. Screenshot taken on 1 November 2026.The setting applies to the whole property, including child properties (they inherit it by default, but their owners can override it). According to [Search Console Help](https://support.google.com/webmasters/answer/16908024?hl=en), it doesn’t affect rankings in classic results, and changes take effect within 1–2 days. One thing worth remembering: the Search Console toggle has nothing to do with training, and the new robots.txt rule has nothing to do with AI Overviews.
X has already added it
----------------------
X has the rule in its robots.txt, with a comment saying it is “honored by Microsoft, Google, OpenAI, and Anthropic”. For now that’s more wishful thinking than fact: Google, Microsoft and OpenAI have only announced support, and Anthropic is still implementing it. Unless we’re missing something. X’s robots.txt leaves nothing to chance anyway: it separately blocks Google-Extended, FacebookBot and Meta’s crawlers, and finally every other crawler with a wildcard group.
[](https://podrez.pl/wp-content/uploads/2026/11/x-robots-txt-usage-ai-training.webp)Excerpt from [X’s robots.txt](https://x.com/robots.txt), screenshot taken on 1 November 2026.Am I adding this rule to podrez.pl?
-----------------------------------
No. I work on brand visibility in AI, and I want models to know me. What ends up in training data can stay in a model’s base knowledge for a long time, and in my view reputation in that data builds much like [E-E-A-T](https://podrez.pl/en/e-e-a-t/), just with a delay. I don’t treat it as a guarantee of recognition, but I deliberately don’t rule this use out. That’s why podrez.pl lets AI crawlers in, including the ones that collect content for training. I wrote more about where models get their knowledge of brands in my piece on [Common Crawl](https://podrez.pl/en/common-crawl/).
I’d see it differently as a publisher of paywalled content, a photographer, or the owner of a site whose business model depends on exclusive content. There, one rule instead of a dozen entries for individual crawlers is a real convenience.
What to do now?
---------------
- **1**### Don’t force it
So far no company has confirmed that the rule already works. The first firm date is 1 February 2027 (Microsoft), OpenAI may be faster, and Anthropic is still implementing it. Existing entries for GPTBot, ClaudeBot or Google-Extended still apply, and the companies say that will stay so.
- **2**### Separate three decisions
Training (the new rule), AI answers from grounding (Google-Extended in Gemini, a meta tag in Bing), and search results with AI Overviews (Googlebot, nosnippet and the [Search Console toggle](https://support.google.com/webmasters/answer/16908024?hl=en)). These are three different settings.
- **3**### Check who actually visits
Before you block anything, check in your [server logs](https://podrez.pl/en/ai-bots-in-logs/) which AI crawlers actually visit your site and whether they’re genuine.
- **4**### Plan the NoArchive switch
If you use NoArchive or NoCache and want to keep your content out of AI answers in Bing, do it before 1 March 2027.
\_\_usage\_ai-training: frequently asked questions
--------------------------------------------------
### Will \_\_usage\_ai-training remove my site from ChatGPT or AI Overviews?
No. The rule is about training models. AI Overviews and AI Mode in Google are controlled through Googlebot, nosnippet or the Search Console toggle, and grounding in Bing through a separate meta tag. Google is the exception: there the rule works like blocking Google-Extended (unless separate rules are set for Google-Extended), so it also covers answers in Gemini Apps.
### Is User-agent: \* with Disallow: / enough?
No. A wildcard group never covers the `__usage_ai-training` token, so it isn’t read as an objection to training. On top of that, it blocks every crawler that doesn’t have its own group in robots.txt, which usually includes Googlebot and Bingbot.
### When does the rule take effect?
As of 1 November 2026, no company has confirmed that it already honors it. Microsoft starts on 1 February 2027, Google in early 2027, OpenAI promises support in the coming months, and Anthropic has started implementing it without giving a date.
### Is it an official standard?
No. It’s an individual IETF draft, version 00 of 30 October 2026, which the IETF hasn’t endorsed. The syntax may still change.
### Sources
Google Search Central Blog, John Mueller, [Supporting a shared AI training control for websites](https://developers.google.com/search/blog/2026/10/ai-training-control) (30.10.2026). Google, [Google-Extended documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended), Search Console Help, [Search generative AI settings](https://support.google.com/webmasters/answer/16908024?hl=en).
Microsoft Bing Blog, Krishna Madhavan, [Your content, your choice: clearer preferences for training and grounding](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) (30.10.2026). Bing Webmaster Tools, [FAQ – New AI Preferences](https://www.bing.com/webmasters/help/new-ai-preferences-2t1cc16d).
OpenAI, [announcement on honoring the shared robots.txt signal](https://openai.com/index/helping-websites-express-ai-training-preferences/) (30.10.2026).
IETF, Kevin Kelley, [Machine-Readable Training Preference Signal](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) (draft-kelley-aipref-training-preference-signal-00, 30.10.2026), [message on the ai-control list](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) (30.10.2026), AIPREF vocabulary [draft-ietf-aipref-vocab-08](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-vocab-08).
Anthropic, [help page on Anthropic’s crawlers](https://support.claude.com/en/articles/8896518) (updated 7.04.2026). X, [robots.txt](https://x.com/robots.txt) (as of 1.11.2026). Search Engine Journal, Matt G. Southern, [Google, Microsoft, and OpenAI Plan to Honor a Shared AI Training Opt-Out](https://www.searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out/595387/).
An LLM may have supported me in preparing this text – most often at the translation, research, proofreading or code-styling stage. The responsibility for the decisions, the claims made and the arguments cited is fully mine. More on [how I work with AI](https://podrez.pl/en/authorship-and-ai/).