On October 30, 2026,1 four AI companies said they would, on varying timelines, read one line in a website's robots.txt file. That file sits at the root of a site and tells automated crawlers what to skip. The new line says one thing: do not train AI models on these pages. OpenAI, Google, Microsoft and Anthropic all said they would honor it.
Only one of the four says it has started acting on it. Anthropic's Kevin Kelley proposed the individual Internet-Draft. ppc.land reports that he told an Internet Engineering Task Force mailing list that same afternoon that his company had already started implementing support for the draft. The other three gave dates in 2027 or no date at all.
On November 1, Massive checked 50 news sites and found usable robots.txt files on 49.2 Zero of the 49 carried the new line. Meta, which is rolling out specialized versions of existing models to Muse, has made no promise to read it. One limit up front: nobody outside these companies can yet confirm what any crawler actually obeys.
The honoring gap, company by company
The honoring gap is the distance between the day a company says it supports a signal and the day its crawler acts on it. The facts sort into four bins, ranging from live to absent. Each bin holds at least one company. That spread is the whole problem.
| Microsoft honors the token3Bing Blog · Krishna Madhavan | Feb 1, 2027 |
|---|---|
| NoArchive and NoCache stop counting3Bing Blog · training and grounding preferences | Mar 1, 2027 |
| Draft expires unless revised4IETF · Machine-Readable Training Preference Signal | May 3, 2027 |
| News sites carrying the token2Massive · November 1 check of usable files | 0 of 49 |
Anthropic sits alone in the live bin. Its draft is titled "Machine-Readable Training Preference Signal," and the company says implementation has started. What Anthropic has not published is proof that every one of its crawlers and datasets already follows it.
Microsoft is the only backer with a hard date. Krishna Madhavan wrote on the Bing Blog that Microsoft will honor the token from February 1, 2027.3 Microsoft also says Bing will stop reading its older NoArchive and NoCache tags as training or grounding preferences after March 1, 2027.
Google is half dated. John Mueller wrote on the Google Search Central Blog that the token "will help websites express their AI training preferences across multiple AI providers in a single step." From early 2027, when no explicit Google-Extended rule exists, Google plans to treat an opt-out as though Google-Extended were disallowed. Google says that would also keep a site out of grounding in Gemini Apps and Vertex AI once its plan takes effect. Grounding means pulling live web pages into an AI answer at the moment someone asks.
OpenAI is the vague one. Its post says it "aims to implement the signal in the coming months." Meta is absent, with no commitment at all.
The mechanics fit on a sticky note. A site writes "User-Agent: __usage_ai-training" and then "Disallow: /". Massive quotes the draft saying the token "is not a directive for crawling behavior," so bots can still fetch the page. A plain "User-Agent: *" rule does not count, and the draft expires on May 3, 2027 unless revised4.
Meta's missing promise and Muse's appetite
We think Meta's silence is the most telling fact in this story. The four companies that signed took on a cost. Each agreed to shrink its own supply of training data whenever a publisher asks. Meta declined to put that cost on the record.
Will help websites express their AI training preferences across multiple AI providers in a single step.
Read the signatures the way an investor reads a business, trusting cash flow over reported numbers. An announcement is a reported number. A crawler that skips your pages is cash. By that test the industry has booked four announcements and roughly one payment, from Anthropic.
Meta's own calendar shows why the cost matters to it. On November 6, 2026,5 Meta announced specialized models for Muse and Meta AI, trained using Muse Wonder. Meta is also rolling out real-time voice and expressive avatars to the Muse app, starting with US users. Products like these eat large volumes of fresh, varied data.
Meta has made some moves toward publishers. It pushed revised commercial terms for websites linked to business accounts to November 20, 2026,6 after backlash over proposed AI uses, and has since paused them indefinitely. Its stated position is that content owners can use robots.txt to express preferences. TechRadar's report, syndicated on Yahoo, sums Meta up as "supportive but uncommitted."
Stating a preference and honoring one are two separate promises. A publisher can write anything in a robots.txt file. What matters is whether Meta's crawler and training pipeline read the shared token and obey it. Meta has not said.
Nobody knows yet whether Muse training touches pages that would opt out. Meta's November 6 announcement does not establish that, and we have seen no evidence either way. A fair reading also allows that Meta's holdout is commercially rational. It runs social platforms and training pipelines, and the draft leaves "training" loosely scoped.
The evidence supports one decision today: treat Meta as its own case in every publisher plan. Three events would change that call. Meta could name a date. A later Massive count could show real adoption, which raises the price of staying out.
The third is the draft lapsing on May 3, 2027 without revision4. That would let every backer drift, so watch that date as closely as Meta's next statement.
The industry has run this loop before. OpenAI's GPTBot and Google's Google-Extended each worked as one company's private opt-out. Microsoft is now retiring NoArchive and NoCache in favor of the shared signal. For years the norm was one rule per crawler. This token is the first broad attempt to replace it.
Will one rule cover everyone by 2028?
Start with the dates each company published. Microsoft goes live 94 days after the October 30 announcement. The draft expires 185 days after it. So unless Kelley revises it, Microsoft's reported support would run for 91 days7 before the document it implements lapses.
The best case for mid-2027 is easy to count. Anthropic is live today. Microsoft follows on February 1, and Google follows sometime in early 2027. That makes three of four backers. OpenAI joins whenever "the coming months" ends, and Meta stays at zero until it says otherwise.
The rollout faces a coordination trap. Publishers have little reason to add a line that few crawlers read. Crawlers feel little urgency when Massive counted 0 of 49 news sites using it2. Each side can point at the other and wait.
Traps like this usually break when one side moves first and absorbs the cost. Anthropic moved first and also wrote the spec. That is counterpositioning: rivals with data-hungry products find the honoring stance expensive to copy. If the stance wins publisher trust, the advantage compounds with every licensing talk that follows.
For publishers the risk runs heavily one way. Adding the group takes minutes, and ppc.land reports that rankings in Google and Bing search results are unaffected. The upside grows each time another crawler goes live. The main cost is Google grounding, which the opt-out also removes.
The signal protects future content only. TechRadar notes it "does nothing for content already scraped." Scrapers and data brokers that never signed can ignore it entirely.
If the draft is revised before May 3, 2027 and Meta commits4, one line could govern five major labs by 2028. If Meta holds out, the shared token covers four of five. Even then, a publisher goes from one rule per crawler to one shared rule plus a single exception.
Ship a two-layer robots.txt this Saturday
Layer one is the shared token. Layer two is the named rules for crawlers that do not read it yet. Budget an hour, including testing.
Step one: open the robots.txt file at the root of your site. Add a group that reads "User-Agent: __usage_ai-training" on one line and "Disallow: /" on the next. If you want press pages kept trainable, add a line such as "Allow: /press/" under it.
Step two: keep the named rules, because OpenAI and Google are not live yet. Add a "User-Agent: GPTBot" group and a "User-Agent: Google-Extended" group, each with "Disallow: /". Before you keep the Google-Extended group, decide on grounding. If AI answers that cite you send real traffic, measure that traffic first.
Step three: put three dates on your calendar. February 1, 2027 is when Microsoft starts reading the token3. March 1, 2027 is when NoArchive and NoCache stop counting at Bing, so swap them out before then. May 3, 2027 is the draft's expiry4, so check the IETF Datatracker for a revision that week.
Step four: write a short checker script. It fetches your robots.txt from outside your own network every week. It confirms the __usage_ai-training group is present and spelled exactly right. It sends you an alert when the group goes missing.
Things will break. Redeploys, CDN caching and redirects can silently replace the file you edited. A typo in the User-Agent line turns the group into noise, and the draft treats a file without the group as expressing no preference.
Step five: handle Meta by name. Look up the crawler names Meta documents and decide whether to block each one directly. The shared token binds nothing at Meta today.
Run the checker once by hand, break the file on purpose, and confirm the alert fires. Then fix the file and leave the weekly run in place.
Practise it step by step: Train the full skill in The Dojo
The bottom line
Write the line today, then verify who obeys it.
Four announcements and one live implementation leave publishers carrying the enforcement work. The token costs minutes to add and leaves Google and Bing search rankings untouched, so the case for shipping it now is strong. Keep the named rules until OpenAI and Google go live, treat Meta separately until it names a date, and watch the May 3, 2027 expiry4. One shared rule plus a short list of exceptions beats managing every crawler on its own.