One robots.txt Token for AI Training: What OpenAI, Google, Microsoft and Anthropic Agreed To, and What It Does Not Do

On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor a single robots.txt policy token to let website owners opt content out of AI model training. According to reporting by The Verge as summarized by AI Weekly, the token is named __usage_ai-training. A site owner adds a User-Agent line naming it, followed by Disallow: /, and can add Allow lines for exceptions. Google's John Mueller said the token "will help websites express their AI training preferences across multiple AI providers in a single step."

What changes for publishers

Until now, a publisher wanting to opt out of training generally had to list each company's crawler or product token separately, for example Google-Extended for Google. A shared token reduces that to one entry. Google will treat a token-based opt-out that does not name Google-Extended as if the publisher had disallowed Google-Extended, so existing and new rules are meant to line up.

The rollout is staggered

The statements are not simultaneous commitments to the same date:

• Anthropic has already started implementing support.
• OpenAI aims to implement the signal "in the coming months".
• Microsoft targets February 1, 2027, and also plans a separate setting for AI grounding, the retrieval step that pulls live content into chat answers.
• Google targets 2027.

For a publisher, this means the token is only partly effective on day one. Until each company ships support, the older, company-specific rules remain the dependable route.

What the document is, and is not

The specification is an individual IETF Internet-Draft titled "Machine-Readable Training Preference Signal", authored by Anthropic's Kevin Kelley, dated October 30, 2026 and expiring May 3, 2027. The IETF states that it "is not endorsed by the IETF and has no formal standing in the IETF standards process." The IETF's AIPREF working group, chaired by Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, has participants including Meta, Cloudflare, Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons.

So the situation is a joint company commitment built on a draft, not a ratified internet standard. Drafts can change before, or without, becoming standards, which is why the expiry date matters.

Three limits worth understanding

1. It governs training, not crawling. Bots may still fetch pages. A publisher who wants to block access altogether still needs conventional crawler rules or technical controls.
2. It is voluntary. OpenAI itself describes it as a coordination mechanism under RFC 9309 (the Robots Exclusion Protocol), not a security control, so scrapers that ignore robots.txt can ignore the token.
3. It is not retroactive. Updating robots.txt does not remove material already in existing datasets or trained models.

Why the grounding distinction matters

Microsoft's plan for a separate AI grounding setting points to a second axis of consent. Training uses content to change a model's weights. Grounding retrieves content at answer time and may display or summarize it, which publishers often treat differently because it can substitute for a visit. A single training token leaves that second question to separate settings, and the reporting does not say whether the other three labs will offer an equivalent.

What to watch next

• Whether OpenAI names a date and whether Google ships earlier than its stated 2027 target.
• Whether the IETF working group adopts or amends the draft before May 3, 2027.
• Whether publishers observe changes in crawler behavior after each company goes live. Compliance can be checked in server logs, though the reporting does not describe an audit mechanism.
• Whether training-opt-out signals become part of licensing talks, as several publishers sit in the working group.

For people building with AI, the practical takeaway is that training consent is becoming a configurable, machine-readable preference. Users of multi-model workspaces such as Metir will see the effect indirectly, as the models they choose are trained under increasingly explicit publisher signals.

OpenAI

Google

Microsoft

Anthropic

Sources:

• OpenAI, Google, Microsoft, Anthropic back one AI opt-out token, AI Weekly, October 30, 2026
• The Verge, original reporting by Alexis Dufresne, October 30, 2026

Image credits

• Server rack with multiple hard drives and network cables, by the Federal Bureau of Investigation, via Wikimedia Commons, public domain. Illustrative only; reviewed visually.