AI Training Opt-Out Robots.txt: One Line That Tells Google, OpenAI, Microsoft and Anthropic Not to Train on Your Site

AI Training Opt-Out Robots.txt: One Line That Tells Google, OpenAI, Microsoft and Anthropic Not to Train on Your Site

For years, keeping your website out of AI training data meant maintaining a separate robots.txt rule for every single AI company — GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and a growing list of others, each with its own name and its own quirks.

On 30 October 2026, that changed. Google, OpenAI, Microsoft and Anthropic announced on the same day that they will honour one shared AI training opt-out robots.txt signal: a single policy token, __usage_ai-training, that speaks to all four companies at once. Add one small group to your robots.txt file and you have stated your preference to four of the largest AI developers in the world.

It is the biggest simplification of website AI controls since Google-Extended appeared in 2023, and it arrives at a moment when publishers, bloggers and small business owners are asking the same question: is my content feeding someone else’s model, and can I stop it? This guide explains the new AI training opt-out robots.txt mechanism in plain English — what it is, the exact code to paste, when each company will actually start honouring it, the traps that silently void it, and the trade-off Google quietly built into the deal.

Everything here is based on the four companies’ own announcements and the Internet-Draft that defines the AI training opt-out robots.txt signal, so you can act on facts rather than rumours.

Table of Contents

AI training opt-out robots.txt code shown in a text editor

What Happened on 30 October 2026

On Friday 30 October 2026, four of the biggest names in AI made the same commitment within hours of each other. Google published a post on its Search Central blog, written by Search Advocate John Mueller, announcing plans to support a shared AI training control in robots.txt. Microsoft published its own commitment on the Bing blog, written by principal product manager Krishna Madhavan. OpenAI posted a short company statement saying it would honour a shared robots.txt signal.

And Anthropic’s Kevin Kelley, a technical lead for standards at the company, submitted the formal proposal to the Internet Engineering Task Force (IETF) — the body that standardises the robots.txt format itself — and announced it to the IETF’s AI Preferences (AIPREF) working group the same afternoon, noting that Anthropic had already started implementing support.

The proposal is an Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00). It is deliberately boring in the best sense: it invents no new grammar, no new directives and no new file. It takes the robots.txt file every website already has — the small plain-text file that has told search engines where they may and may not go since the early web — and gives it one new name to use on the User-Agent line. That name is the whole innovation.

Why did four fierce competitors agree on the same day? Publishers, civil society groups and regulators had been asking for industry-wide controls for years. The old system was a mess: every AI developer published its own crawler name, and site owners had to discover each one, write a rule for it, and update the list every time a new developer appeared. Google itself framed the change as a response to that feedback, and noted that the approach fits the European Union’s code of practice for general-purpose AI model providers under the EU AI Act, which already treats robots.txt as the main way publishers state machine-readable preferences.

The announcement also landed against a noisy backdrop of lawsuits and licensing fights over training data — the first shared answer, remarkably, is a few characters in a plain text file.

One more detail matters for anyone deciding what to do today: as of now, this is four commitments and almost no live implementation. Anthropic has started building support, Microsoft has named a firm date, Google aims for 2027 and OpenAI says “the coming months”. Nothing you write in robots.txt this week will be read by any of them for months. That is not a reason to ignore it — it is a reason to understand it properly before the switch-on dates arrive.

How the AI Training Opt-Out Robots.txt Token Works

The mechanism is simple enough to explain in a minute, which is exactly why it has a real chance of being adopted widely. A robots.txt file is organised into groups. Each group starts with a User-Agent line naming a crawler — Googlebot, Bingbot, GPTBot — followed by Allow and Disallow lines that say which paths that crawler may fetch. The new AI training opt-out robots.txt approach keeps that structure exactly, but the User-Agent line names a purpose instead of a crawler: __usage_ai-training.

Under that group, Disallow: / means “do not train on anything on this site”, and an Allow line can carve out exceptions. The draft’s own example opens a /press-kit/ folder — the kind of material a company would be glad to see a model learn from. Matching follows the path rules and longest-match precedence of RFC 9309, the Robots Exclusion Protocol the IETF formalised in September 2022, so the behaviour is predictable for anyone who already understands robots.txt.

The double underscore at the front is deliberate. Some robots.txt parsers in the wild match bot names loosely, and the prefix makes an accidental match with a real crawler unlikely.

A single group may list several policy tokens, and policy tokens can sit beside ordinary crawler names in the same file. Kelley, the draft’s author, wrote that he spent a long time studying how deployed parsers behave and designed the AI training opt-out robots.txt mechanism to stay away from the areas where they disagree, so software that knows nothing about the AI training opt-out robots.txt token simply skips the group, exactly as it would skip a group addressed to a bot it does not recognise. Your existing file keeps working as before; the new group is inert until a company that understands it reads it.

What counts as “training” is borrowed rather than invented. The draft takes its definition from the AIPREF working group’s vocabulary document, edited by Paul Keller of Open Future and Martin Thomson of Mozilla: AI training means using an asset “to modify the learned parameters of a generative AI model”. Models used only for classification, ranking or scoring fall outside that definition, even when they generate an explanation for their output.

And the preference that applies to a page is the one written in the robots.txt file in force at the moment the page was retrieved — not the file as it stands today, and not the file as it stood last year.

AI training opt-out robots.txt syntax highlighted in a code editor

The AI training opt-out code, line by line

Here is the draft’s own example, which is also the most useful starting point for most site owners:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read it as a sentence: for the purpose of AI training, nothing on this site is available, except the press kit. If you want no exceptions at all, drop the Allow line. If you want the opposite — to state explicitly that training is welcome — you can write Allow: / under the same token, though doing nothing at all already means the same thing, because a file with no token group expresses no preference.

That last point is easy to misunderstand, so it is worth stating plainly. The AI training opt-out robots.txt token is an opt-out mechanism in a world that defaults to opt-in. If your file says nothing about AI training, the companies read that as no preference expressed — not as a refusal, and not as permission, but as silence. The token exists so that silence can be replaced with a clear answer, in either direction.

The Exact Code to Add to Your Robots.txt

For a site that wants to opt out of AI training across the whole site, the complete addition is two lines:

User-Agent: __usage_ai-training
Disallow: /

For a site that wants to opt out everywhere except a specific folder — a press kit, a public documentation section, or a portfolio a creator is happy to share — add one Allow line under the same group:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

And for a site that wants to block training on one sensitive area only, such as a members’ section, while leaving the rest of the site open:

User-Agent: __usage_ai-training
Disallow: /members/

Where does it go? Anywhere in the file, though most site owners put it near the top, above or below their existing crawler groups, with a short comment line above it so the next person who edits the file understands what it is. Comments in robots.txt start with #. A tidy example for a WordPress site might look like this:

# Shared AI training preference (IETF draft, Oct 2026)
# Applies to Google, OpenAI, Microsoft and Anthropic
User-Agent: __usage_ai-training
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Notice what this file does not do: it does not touch your search rules. The draft says plainly that the AI training opt-out robots.txt token “is not a directive for crawling behavior”. It states a preference about how content is used, not about who may visit, so search keeps working.

Rules addressed to Googlebot, Bingbot or any other named crawler keep exactly the meaning they have today. A site that wants a bot to stay away altogether still has to say so in a group for that bot’s own name. That separation is the feature publishers asked for most: refuse training, keep the search traffic.

There is one more practical point. Keep your existing vendor-specific controls in place for now. Google still reads Google-Extended, OpenAI still reads GPTBot and OAI-SearchBot, and Anthropic still reads ClaudeBot. The shared AI training opt-out robots.txt token layers on top of them; it does not retire them. The older controls are the ones actually working today, and they cover companies the shared AI training opt-out robots.txt token does not yet reach. Removing them now would leave you with less protection, not more.

Four Companies, Four Timelines

The four announcements agreed on the AI training opt-out robots.txt mechanism but not on the calendar. Here is where each company stands, based on its own words:

Company Status When it starts
Anthropic Implementing now Rolling out since 30 October 2026; covering all services takes time
OpenAI Committed “In the coming months”; no fixed date given
Google Planned Aims to support it in 2027; crawler documentation says early 2027
Microsoft Firm date 1 February 2027, applied as Bing recrawls each page

Anthropic is furthest along. Kelley wrote in his post to the IETF mailing list that the company has already started implementing support, though he added that it will take time to cover all of Anthropic’s services, and that the company’s reading of these signals may change as the standards evolve.

Anthropic’s existing ClaudeBot, Claude-User and Claude-SearchBot controls stay in place. One nuance worth knowing: rules that explicitly name Anthropic’s training crawler, ClaudeBot, take precedence over the shared AI training opt-out robots.txt signal at Anthropic, so a site that blocks ClaudeBot and allows the AI training opt-out robots.txt token — or the reverse — gets the ClaudeBot answer.

OpenAI has committed without a date. Its statement says it will honour the shared robots.txt signal to help website operators communicate AI training preferences to participating providers, and that implementation will come “in the coming months”, with details to follow.

The company also says the new AI training opt-out robots.txt signal will not replace the robots.txt controls it already offers, such as GPTBot. OpenAI’s post describes the AI training opt-out robots.txt signal as covering “foundational generative AI training”, and it has not said whether that wording narrows the broader definition in the vocabulary draft — a small ambiguity worth watching as the details arrive.

Google is aiming for 2027. John Mueller’s Search Central post says Google aims to support the control along with other AI providers in 2027 and will provide updates on progress. Google’s crawler documentation is slightly more specific and says early 2027. Until then, Google-Extended remains the working control, and nothing in the announcement changes that.

Microsoft has the only firm date. It will start honouring the shared training preference on 1 February 2027, applying it as Bingbot recrawls each site. Microsoft’s calendar is the one site owners can actually plan around, and it comes with extra machinery of its own, covered in the grounding section below.

AI training opt-out robots.txt and the data centres where AI models are trained

What the AI Training Opt-Out Does Not Do

A new control is only useful if you understand its edges. The AI training opt-out robots.txt token has four limits that every site owner should know before relying on it.

First, it only looks forward. Whatever the companies crawled before the line appears is already in their training sets, and nothing in the draft forces any company to delete it. The preference applies to content retrieved after the rule is in place. If your site has been public for years, its older pages have almost certainly already been collected. The token protects what you publish next, not what you published last year.

Second, it is voluntary. The draft says the AI training opt-out robots.txt token “is not a directive for crawling behavior”, and only companies that sign up will honour it. Scrapers that ignore robots.txt will ignore this too. OpenAI itself calls its participation a “voluntary effort”. Private licensing agreements may override it, the draft adds, so a publisher with a content deal can post a blanket refusal and still have that deal govern its relationship with a given developer. It is a request that four firms have promised to respect, not a lock.

Third, it covers training, not retrieval. The token asks labs not to train on your content. It does not pull you out of live answer-engine citations — the fetches that happen when someone asks ChatGPT or a search assistant a question and the bot reads your page to answer it. That distinction is the difference between a model memorising your words into its weights and a model looking your page up to cite it.

If you want to disappear from AI answers as well, you need the separate grounding controls, which only Microsoft has announced so far — and which come with their own trade-offs.

Fourth, it is not a standard yet. The IETF’s datatracker lists Kelley’s text as an individual submission, not a working-group document, and it carries no formal standing. It expires on 3 May 2027 unless a new version replaces it. Kelley has offered it as input to the working group’s separate draft on attaching preferences to content, co-written by Google’s Gary Illyes, one of the authors of RFC 9309. The technical form may still change, and Anthropic has warned that how it applies the AI training opt-out robots.txt signal “may change” as the work goes on.

None of this makes the AI training opt-out robots.txt token useless. It makes it what it is: the first shared, machine-readable way for a site owner to say no to training across four major developers, with honest limits. The publishers who get the most from it will be the ones who add it now, keep their existing controls, and check back when the 2027 switch-on dates arrive.

The Wildcard Trap That Silently Voids Your Opt-Out

There is a catch hiding in the file most sites already have, and it is the single most important detail in this entire guide. Many sites block all crawlers from part of the site with the familiar wildcard rule:

User-agent: *
Disallow: /

Under the new draft, that rule says nothing about AI training. Kelley deliberately designed the AI training opt-out robots.txt token so that the wildcard group does not match it: the draft states that wildcard groups “MUST NOT be interpreted as matching the policy token”, and a file with no __usage_ai-training group expresses no preference at all — which, as Kelley pointed out, is true of every robots.txt file published so far.

A site that shuts out every bot today has, under the draft, said nothing about AI training. Owners who want out have to add the new group by name.

The same logic applies to the older vendor tokens. A Google-Extended block keeps its meaning at Google, but it does not become a shared opt-out for OpenAI or Microsoft. Each company’s own control still speaks only to that company. The shared token is the only line that speaks to all four at once, and it only speaks if it is written out in full.

The practical check is easy. Open your site’s robots.txt in a browser — it lives at yoursite.com/robots.txt — and search the page for __usage_ai-training. If it is not there, you have expressed no training preference, whatever else the file says. Add the group, save, and check again. It takes a minute, and it is the difference between an opt-out that works and one that never existed.

Microsoft Goes Further: Training vs Grounding

Microsoft went a step further than the other three, and its announcement is the most detailed of the four. Next to the shared training signal, Microsoft is adding a separate grounding preference, set page by page through a meta tag or an HTTP header, which decides whether a page can feed the answers that Bing, Copilot and Microsoft 365 generate on the spot. Training is memorising; grounding is looking things up. Microsoft is the only company so far that lets a site owner refuse one and allow the other.

The grounding preference takes three values. Allow is the default: the page can be used for grounding. Disallow keeps it out of grounding everywhere. And a new value, DisallowExtended, keeps a page available to Microsoft’s own AI experiences but out of the grounding services Microsoft sells to other companies, such as Web IQ. A site could therefore refuse training and still allow grounding, keeping its citations and links inside AI answers — the visibility many publishers depend on — while keeping its content out of model weights.

Microsoft also lets the training preference be set for a single page, and that page-level setting overrides robots.txt.

And it warns plainly that restricting grounding will reduce a site’s visibility in those experiences, stating that “neither choice guarantees inclusion, traffic, or payment”. The company is also retiring the NoArchive and NoCache tags it introduced in 2023 as AI controls. It will honour the new preferences as it crawls from 1 February 2027, and after 1 March 2027 it will stop reading the old tags as training or grounding signals. Sites that rely on those tags to stay out of grounding can record a temporary preference in Bing Webmaster Tools from 15 December 2026.

There is a subtle trap here that Microsoft itself flags. Once it detects either new preference on a site, it will disregard any old tags still there — so a publisher that adds only the shared training line could lose a grounding restriction it set earlier through NoArchive or NoCache. If you rely on those tags today, read Microsoft’s Bing blog post in full before you change anything, and set the new grounding preference explicitly rather than assuming the old one survives.

AI training opt-out robots.txt being added by a website owner on a laptop

Google’s Catch: The Gemini Grounding Trade-Off

Google’s version of the deal contains a trade-off that deserves careful reading, because it is the one place where opting out of training costs a site owner something visible. Google says that when a site opts out with the shared AI training opt-out robots.txt token and does not mention Google-Extended separately, it will read that as a Google-Extended block.

Google-Extended, which Google has offered since 2023, has no effect on inclusion in Google Search — your pages keep ranking — but it covers more than training. According to Google’s crawler documentation, it also decides whether a page can be used for grounding in Gemini Apps and on Vertex AI, the process of handing pages from the search index to the model while it answers.

On Google’s stated plan, then, the shared AI training opt-out robots.txt line also takes a site out of those grounded Gemini answers, unless the site adds an explicit allow group for Google-Extended, which Google treats as the more specific rule.

In plain terms: use the shared AI training opt-out robots.txt token alone and you lose both training and Gemini grounding citations; add an explicit Google-Extended allow and you keep the citations while still refusing training at the other three companies. Only websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps, so for a site that earns traffic from Gemini citations, that one extra group is worth adding.

AI Overviews and AI Mode in Google Search are a separate matter, governed by the Googlebot rules and by a Search Console setting the company rolled out to every site in August 2026. The shared token does not touch them either way. If your goal is to stay visible in Google Search while refusing training, the combination to use is the shared AI training opt-out robots.txt token plus an explicit Google-Extended allow — and then to check, once Google’s support goes live in 2027, that the combination behaves as Google described.

For the full detail, Google’s own announcement is the authoritative source: Supporting a shared AI training control for websites on the Google Search Central blog. The technical definition lives in the IETF draft itself: Machine-Readable Training Preference Signal.

Who Is Missing: Meta, Apple and Amazon

The most conspicuous absence from the announcement is Meta. Its head of communications, Andy Stone, told The Verge that a common signal reaching several providers at once is “an important step”, but did not say whether Meta will honour it.

The timing is awkward. Meta’s revised business terms, which let it send crawlers to websites tied to Facebook, Instagram and WhatsApp business accounts and use what they collect to train its products, were originally due to take effect on 30 October — the very day of the announcement. After publishers complained to competition authorities in the EU and the UK, Meta told Reuters on 23 October that it had moved that date to 20 November 2026.

Meta is not the only large company outside the commitments. Amazon was not among the four, and neither was Apple, whose Applebot-Extended token sits on many existing opt-out lists.

That matters because the AI training opt-out robots.txt token’s value grows with every company that adopts it, and shrinks with every one that does not. A site that adds the shared AI training opt-out robots.txt line has still said nothing to Meta’s crawlers, nothing to Apple’s, and nothing to the many smaller training-data collectors — Common Crawl’s CCBot, for example, whose archive feeds many training sets — that never signed up at all. Those still need their own rules, by name, in the same file.

There is a hopeful detail in the draft’s acknowledgments. Meta and Amazon staff do appear in its long acknowledgments table, next to people from Google, Microsoft and OpenAI, publishers such as News Corp, the Financial Times and the BBC, and groups including the Electronic Frontier Foundation, Creative Commons, the Internet Archive and Common Crawl.

The draft notes that those affiliations are listed for identification only, since IETF participants contribute as individuals — but the presence of Meta people in the room suggests the door is open. How much the line protects will depend on who signs up next, and the company with the most business pages on its platforms has not yet said yes.

Meta’s own open-weight work shows why this matters to site owners. Meta’s open-weight Muse Spark model is trained on vast amounts of web data, and the same is true of open-weight image models like FLUX 3 — every model in this wave is hungry for the same public web. Until the shared AI training opt-out robots.txt signal covers more of the industry, a robots.txt file needs both the new AI training opt-out robots.txt token and the old per-company rules.

How to Add the AI Training Opt-Out to Your Site

Adding the AI training opt-out robots.txt token takes about five minutes on most sites. The steps differ slightly by platform, but the file you are editing is always the same: the robots.txt at the root of your domain.

On WordPress (self-hosted)

If your site runs WordPress with an SEO plugin, open the plugin’s robots.txt editor — Rank Math and Yoast both have one under their settings — and paste the group at the top of the file.

If you prefer the file manager route, open your hosting control panel, find the robots.txt file in the public_html folder, edit it, and add the same two lines. WordPress does not create a physical robots.txt by default; if none exists, create a plain text file named robots.txt in the root folder. Save, then visit yoursite.com/robots.txt in a browser to confirm the new group is live.

On WordPress.com, Wix, Squarespace and Shopify

Hosted platforms handle robots.txt differently, and some do not let you edit the file directly. On WordPress.com, check the SEO settings for a crawler or AI control; on Wix and Squarespace, look for the robots.txt editor in the SEO panel; on Shopify, you can edit robots.txt through the theme editor’s robots.txt.liquid template. If your platform offers no direct editing, use whatever AI-training toggle it provides, and keep the vendor-specific crawler blocks it supports. The shared token is only useful where you can actually write it into the file.

Test it before you trust it

Three checks, in order. First, fetch the file in a browser and confirm the AI training opt-out robots.txt token group is present and spelled exactly __usage_ai-training — two leading underscores, one hyphen, no spaces inside the name. Second, check that no earlier group in the file contradicts it: a Disallow for one company and an Allow for the AI training opt-out robots.txt token on the same path can produce surprising results under longest-match rules, so keep the file simple.

Third, put a reminder in your calendar for February 2027: re-check the file then, confirm Microsoft is honouring it as Bing recrawls, and look for Google’s early-2027 update. A control nobody verifies is a control nobody can rely on.

AI training opt-out robots.txt protecting content from large-scale AI training data centres

Should You Opt Out? A Decision Guide for Site Owners

There is no single right answer, and the honest advice is that it depends on what your site is for. Here is how the trade-off breaks down for the three kinds of site owner most likely to be reading this.

Publishers and bloggers who earn from traffic. If your income comes from ads, affiliate links or subscriptions, your content’s value is in the visits it brings, not in the model weights it might train. For you, the shared AI training opt-out robots.txt line is close to free money in reverse: it costs nothing, it does not touch search rankings, and it keeps your articles out of future training runs.

The one cost is Google’s grounding trade-off — add the explicit Google-Extended allow if Gemini citations bring you clicks, or accept the loss if they do not. Keep your search rules untouched, and do not confuse this AI training opt-out robots.txt token with blocking Googlebot, which would cost you traffic.

Creators and small businesses. If your site is a portfolio, a shop or a service business, the calculation is similar but the stakes are lower: your pages are less likely to be training targets, and your main risk is a competitor’s model learning your distinctive copy or product descriptions. The token is a cheap precaution. Creators who also publish with AI video generators should note the symmetry: the tools you use are trained on other people’s public content, just as yours may train theirs. Opting out is a statement about your own work, not a judgment on the tools.

Developers and open-source projects. If your site exists to be read — documentation, tutorials, public code — training may actually help you: models that learned your documentation answer questions about your project better. Many open-source projects will reasonably choose to allow training, or to allow it for documentation while blocking it for a blog. The token supports that choice too: write Allow: /docs/ under the group and you have stated a nuanced preference instead of a blanket yes or no. That nuance is the point of the whole mechanism.

Whatever you decide, decide it deliberately. The default — saying nothing — is also a decision, and it is the one most sites are making by accident right now. A one-line statement of your actual preference is better than silence, in either direction.

What This Means for the Open Web

Step back from the syntax and the scale of the change is striking. After years in which the only answers to AI training were lawsuits, licensing deals and per-company blocklists, the first shared answer is a few characters in a plain text file that anyone can edit in a minute. It is voluntary, it is incomplete, and it only looks forward — and it is still the most practical thing to have happened for website owners since the argument began.

Three things will decide whether it becomes the norm or a footnote. The first is adoption: Meta, Apple, Amazon and the training-data collectors have to join, or the AI training opt-out robots.txt token stays a partial answer. The second is verification: nobody has yet described how a site owner can check that a provider actually honoured the AI training opt-out robots.txt signal, and without a way to verify, the promise rests on trust.

The third is the standard itself: the draft expires in May 2027 unless it is renewed or adopted, and the IETF’s AIPREF working group is still writing the wider vocabulary — grounding, search, and other uses — that this AI training opt-out robots.txt token deliberately left out.

For now, the practical step is short, and it is the same for a personal blog and a national newspaper: add a group for __usage_ai-training, keep the existing crawler rules, and check what Google’s translation does to Gemini citations before relying on it. The open web has spent thirty years running on polite requests in robots.txt files. This one is the first that four AI giants agreed to read together — and that, whatever its limits, is worth a line in your file.

Frequently Asked Questions

Does the AI training opt-out robots.txt token block Google from indexing my site?

No. The token is not a crawling directive; it states a preference about how content is used for training. Googlebot’s rules are separate, and the AI training opt-out robots.txt token has no effect on inclusion in Google Search or on rankings. Google-Extended, which the AI training opt-out robots.txt token maps to at Google, also has no effect on search inclusion.

Will the AI training opt-out remove my content from models that already trained on it?

No. The signal only looks forward. Content already collected into datasets or already trained into shipped models is unaffected, and nothing in the draft requires a company to delete it. It protects what you publish after the rule is in place.

Do I still need GPTBot, ClaudeBot and Google-Extended rules?

Yes, for now. The shared token layers on top of the existing controls; it does not retire them. The older controls are the ones actually working today, and they cover companies — Meta, Apple, Amazon — that have not joined the shared AI training opt-out robots.txt signal. Keep both until the shared AI training opt-out robots.txt token is live everywhere.

When will the AI training opt-out actually start working?

Anthropic has started implementing support since 30 October 2026. Microsoft starts on 1 February 2027. Google aims for early 2027, and OpenAI says “in the coming months”. Nothing reads the token yet, so add it now and verify again in February 2027.

Does User-agent: * with Disallow: / opt my site out of AI training?

Not under this draft. The draft states that wildcard groups must not be interpreted as matching the policy token, and a file with no __usage_ai-training group expresses no training preference at all. You must add the token group by name.

Can I opt out of training but stay in AI answers?

Partly. Training and grounding are different uses, and only Microsoft currently lets you refuse one and allow the other, through its separate grounding preference. At Google, the shared AI training opt-out robots.txt token maps to a Google-Extended block, which also removes you from Gemini grounding unless you add an explicit Google-Extended allow. There is no shared grounding token yet.

Published 9 November 2026. The announcement dates, timelines and draft status in this article reflect the companies’ own statements as of 9 November 2026; the IETF draft expires on 3 May 2027 unless renewed, so check the current draft before relying on the details.

Leave a Comment

Your email address will not be published. Required fields are marked *