🔍 Read the full analysis: OpenAI, Google, Microsoft And Anthropic Back A Shared Opt-Out—How It Works on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have each said they will support a shared robots.txt signal for publishers who do not want their content used to train generative AI models. The signal is voluntary, implementation dates differ, and it does not block crawling or cover content used to generate live AI answers.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal for websites that do not want their material used to train generative AI models. The proposal could replace the need for publishers to set separate training preferences for each company, but it is a voluntary request, not a technical block, and the companies have not set the same rollout date.
The signal appears in an Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Anthropic standards lead Kevin Kelley and published on October 30. A site would add a shared user-agent group, User-Agent: __usage_ai-training, followed by Disallow: /. The draft describes this as a machine-readable preference against using the site’s content to train generative AI models.
The four companies have given different commitments. Anthropic says it has started implementing support, while noting that extending it across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Microsoft says it will honor the shared training preference from February 1, 2027. Google aims to support it in 2027; its crawler documentation specifies “early 2027.”
Google says it plans to treat the shared signal like a disallow rule for Google-Extended when a website has not set a separate Google-Extended rule. Microsoft is also introducing a distinct page-level preference for AI Grounding, its term for using web content to inform AI answers in Bing, Copilot and Microsoft 365. Its new preferences take effect as it crawls from February 1, 2027. Microsoft says NoArchive and NoCache will stop counting as training or grounding preferences after March 1, 2027.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Simpler Signal, With Limits
A common rule could make it easier for publishers to express a preference across several major AI companies without maintaining a separate training instruction for each crawler. That matters to newsrooms, creators and other site operators seeking more control over whether their work is used in model development.
But the change does not give publishers a guaranteed way to prevent use. The signal is a request that participating companies say they will honor; it is not described as a technical access control. It also concerns training, not every use of online material. Microsoft’s separate grounding preference highlights the distinction: content can be excluded from training while still being available for some AI answers, depending on the company’s rules and the publisher’s settings.
The proposal also does not establish payment, guarantee traffic or determine whether a page appears in search results. Microsoft says its preferences do not change eligibility for Bing search results, and Google says Google-Extended does not affect Google Search. Publishers therefore need to distinguish among training, AI-generated answers and search rather than treating one robots.txt instruction as a universal opt-out.
Products Worth Considering
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
LG UltraGear 32GX870A-B 32-inch 4K OLED Gaming Monitor
LG UltraGear 27G640A-B 27-Inch QHD IPS Gaming Monitor, 300Hz, 1ms, G-SYNC Compatible, FreeSync Premium, DisplayHDR 400
As an affiliate, we earn on qualifying purchases.
From Individual Rules to One Draft
Before this proposal, site operators could use company-specific controls, including GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has used page tags such as NoArchive and NoCache for related preferences. The new rule aims to provide a shared expression for training, while existing company-specific rules remain relevant during the rollout.
The draft is part of work associated with the Internet Engineering Task Force’s AI Preferences working group. It is an individual Internet-Draft, not an adopted internet standard. At a working-group meeting on October 21, one chair said the group had not declared consensus on anything. The draft’s publication and company support therefore mark a proposal and implementation plans, not formal standards approval.
Meta’s communications chief, Andy Stone, said the company supports publishers’ ability to express preferences and called a common signal an important step. He did not state that Meta would honor this particular signal. The source material also reports no commitment from Apple, Amazon, xAI or Perplexity.
“Anthropic “has already started implementing support for this draft.””
— Anthropic standards lead Kevin Kelley
Adoption and Coverage Remain Open
It remains unclear how many AI companies beyond the four named will honor the signal, and Meta has not made an implementation commitment in the statement cited. The source material reports no response from Apple, Amazon, xAI or Perplexity. Even among the four companies, dates and service coverage differ, and Anthropic says its rollout will take time.
The draft has no formal standing in the IETF standards process at this stage. Its precise practical effect will depend on how companies implement and interpret the rule. It also does not cover content already used to train existing models, prevent a site from being crawled, or by itself govern AI answers that draw on live web content. Microsoft’s separate grounding setting addresses one such use for its services, but the shared training signal does not.
Company Rollouts Through 2027
Website operators can add the proposed group if they want to express a preference against training use, while keeping existing company-specific rules in place until each provider’s support is active. They should review company documentation as dates approach, especially if they currently rely on NoArchive or NoCache: Microsoft says those tags cease to count as training or grounding preferences after March 1, 2027.
The next milestones are Anthropic’s expansion across services, OpenAI’s planned implementation in the coming months, and the 2027 launches announced by Google and Microsoft. Further company commitments, and whether the IETF working group reaches consensus on the draft, remain to be seen.
Key Questions
What is the shared AI training opt-out?
It is a proposed robots.txt rule group that tells participating companies a website does not want its content used to train generative AI models. The draft’s example uses User-Agent: __usage_ai-training and Disallow: /.
Does the rule stop AI companies from crawling a website?
No. The shared signal is described as a training preference, not a block on crawling. It also does not remove content already used to train models or automatically cover material used to produce live AI answers.
When will the companies honor the signal?
Anthropic says implementation has started, and OpenAI aims to implement the signal in the coming months. Microsoft’s date is February 1, 2027; Google aims for 2027, with its documentation saying early 2027. The dates and service coverage vary.
Has Meta agreed to honor the opt-out?
Meta communications chief Andy Stone voiced support for publishers expressing preferences and for a common signal. He did not say that Meta would honor this specific draft.
Is the shared signal an official internet standard?
No. It is an individual Internet-Draft associated with the IETF’s AI Preferences working group, and it has no formal standing in the IETF standards process at this stage.
Source: ThorstenMeyerAI.com
