On October 30, 2026 01, four AI labs backed one proposed opt-out signal for model training. Anthropic, OpenAI, Google and Microsoft all signed on. Under the proposal, publishers would write the instruction once in robots.txt, the plain text file that tells crawlers what a site allows. Every participating lab is supposed to read it.
Only one lab reads it today. The Verge reports that Anthropic already honors the signal. OpenAI says it "aims to implement" support in the coming months, with no date attached. Microsoft's Bing says its support will take effect February 1, 2027 03, and Google says only that it will ship in 2027.
The day before, Reddit told shareholders it is "shifting Reddit to a logged-in-only experience," with testing planned for the fourth quarter in select markets outside the U.S. Its third-quarter letter said October search referrals fell by more than 1 million 14 a week on average. One of the biggest sites on the web looked at a polite signal with a 2027 date and chose a locked door.
We owe you one admission up front. None of the October reporting says how many sites have adopted the token, or how often any lab complies. So the hardest numbers in this story are calendar dates and Reddit's own traffic figures.
Signals declare, locks enforce
The token, reportedly named __usage_ai-training, came out of an Internet Engineering Task Force working group. It states a preference about future model training, and that is all it does. A crawler that ignores it can still download every page. Microsoft says scraping stays allowed by default until a publisher changes the setting.
The calendar and traffic figures that define the opt-out gap
Every control a publisher owns sits on one of two layers. The signal layer says what the publisher wants. The lock layer decides what a crawler can fetch. The risk lives in the weeks between a signal going out and a lab honoring it.
October's facts sort neatly into those two layers. On the signal side sit the robots.txt token, OpenAI's description of its planned shared website preference signal as a "voluntary effort," and Microsoft's split controls. Microsoft principal product manager Krishna Madhavan wrote that publishers can now choose "training, grounding, or both while remaining eligible for Bing search results." Grounding means a model fetching your page to answer a live question.
On the lock side sits Reddit. Its shareholder letter says a logged-in-only test begins this quarter in select markets outside the U.S. According to 24 09/7 Wall St., Reddit holds licensing deals with OpenAI and Google. The same outlet reports Reddit is pursuing lawsuits against Anthropic and Perplexity AI.
The gap between the layers comes in four widths. Anthropic's is zero. OpenAI's is open-ended. Microsoft's is about 13 weeks 25, and Google's could run past a year. The IETF draft itself is dated October 30, 2026 13. Unless it is advanced or replaced, it reportedly expires May 3, 2027 04.
Reddit priced the gap first
Reddit's move is the clearest case study of the week. Steve Huffman told analysts the "vast majority" of scraping happens through logged-out access.
Still, Reddit's cash flow explains the wall better than its scraping complaints do. Referral traffic is revenue that Google's interface decides to send or keep. Licensing is contracted revenue that Reddit controls. 24 09/7 Wall St. reports the planned wall would protect Reddit's deals with OpenAI and Google and strengthen its case against Anthropic and Perplexity AI.
The wall costs something. 24 09/7 Wall St. reports U.S. daily active users fell 2% 15 to 51 million, the first U.S. drop since the 2024 IPO. GadgetReview reports Huffman warned daily visitors could fall by double digits. SEO Handbook's reading of the letter finds four stated reasons for the wall, and Google is not one of them.
The labs ran their own two-track play that week. OpenAI published "Fair use benefits everyone" alongside its post on training preferences. One post offers an opt-out. The other argues the training was lawful to begin with. Nieman Lab reports that, according to legal experts, Meta's new commercial terms let it scrape sites tied to business accounts on its apps, starting October 30.
Taken together, those moves show where consent is heading. The labs offer a voluntary signal while keeping a legal argument that they never needed one. Meta writes consent into the terms of a product publishers cannot easily leave. Reddit answers with identity checks and contracts.
We think Reddit read the timeline correctly for a platform its size. It has paying buyers for its corpus, so a wall raises the price of a product people already pay for. A small publisher with no licensing buyer takes the wall's traffic cost and gets none of the contract revenue. For most sites, the lock layer should cover only the material someone would pay to license.
The evidence already supports two decisions. Publish the token, because it costs one line and creates a dated record of your preference. Keep every lab-specific crawler rule too, since three of the four labs are not yet reading the shared one. Three events would change the call. OpenAI or Google could name a date, or any lab could give publishers audit access to its crawl logs.
Consent runs on each lab's own calendar
Anthropic already reads the token.
One line in robots.txt pays off with at least one lab right now. It also leaves a dated record of your preference that could matter in a later dispute.
Google could stay unprotected for over a year.
Google has named a year with no day attached. Any page crawled during that stretch can end up in a trained model, and an opt-out cannot pull it back out.
The standard itself may lapse first.
The IETF text could expire before Google ships support. Recheck the syntax before each review so your robots.txt matches whatever version survives.
Consent may move behind logins by 2027
The arc behind this week is short, and every step has a date on it. The Straits Times notes that The New York Times sued OpenAI and Microsoft in 2023 over unauthorized use of its articles. Private licensing deals, like Reddit's, came next. In October 2026, Reddit built its wall and Meta rewrote its terms.
Each step moved consent somewhere new. First it went to the courts, then to private contracts. Now it sits in defaults written by whoever owns the platform or the crawler.
The gap runs on a calendar anyone can check. October 30, 2026 13 to February 1, 2027 03 is 94 days 28, about 13 weeks. At Reddit's reported pace of more than 1 million 14 lost referrals a week, 13 weeks 29 works out to more than 13 million referrals. If Google ships on December 31, 2027 30, the wait stretches to about 61 weeks.
That napkin math has a catch. Huffman blamed the referral losses on Google's AI Overviews and AI Mode, which answer searches directly. The Straits Times reports the new token only stops future AI training. A publisher bleeding traffic to AI answers gets little help from a training opt-out, even on the day Google ships it.
The risk is lopsided. Publishing the token costs almost nothing and buys a small, real upside with Anthropic today. Relying on it alone leaves the downside open, because anything already inside a trained model stays there. The draft also expires about 26 weeks 31 after publication, possibly before Google ships at all.
Three consent shifts landed between 2023 and 2026. If that pace holds, two or three more arrive by 2031. Our read is that the open web splits in two: a free public layer of summaries for search, and full text behind logins or paid feeds. Once that split hardens, it is unclear whether the shared signal will still matter or survive mainly as evidence in court.
Ship a crawler ledger before February
Step one: add the shared token to your robots.txt. Check the exact syntax against the current IETF draft, since a draft can still change before May 3, 2027 04. Leave every existing lab-specific block in place. Until a lab's support is live, those older lines do the real work.
Step two: split your site into tiers. The public tier holds summaries and canonical pages that search engines should keep finding. The locked tier holds full text someone would pay to license, behind a login or a paid feed. Everything else stays open with the training opt-out switched on.
Step three: build the ledger. A crawler ledger is a weekly table of which bots fetched which pages. Pull seven days of server logs and filter by user agent, the name a bot sends with every request. Count requests per bot per day, then flag any request to a path your robots.txt disallows.
Step four: keep evidence. Save each flagged request's timestamp and path in a dated file. Those records are what a contract dispute or lawsuit will ask for. Reddit's litigation against Anthropic and Perplexity AI shows platforms already take that route.
Expect the first run to break. Your CDN, the content delivery network that caches pages near readers, may still serve an old robots.txt. Subdomains and staging sites may lack the file entirely. Fix what the ledger catches and run it again the next Monday.
Step five: put February 1, 2027 03 on the calendar for Microsoft and May 3, 2027 04 for the draft's expiry. Add a third review the day OpenAI or Google names a date. On each review, rerun the ledger and check whether that lab's crawler changed its behavior. A promised date only counts once your own logs confirm it.
Stand up a crawler ledger before Bing goes live
- Publish the shared token. Add the training opt-out line to robots.txt using the current IETF draft syntax, and keep every lab-specific crawler block in place.
- Run the first ledger. Pull seven days of server logs, count requests per bot per day by user agent, and flag any fetch to a disallowed path with its timestamp saved to a dated file.
- Book the review dates. Put February 1, 2027 03 and May 3, 2027 04 on the calendar, rerun the ledger each time, and check your CDN, subdomains and staging sites for a missing or stale robots.txt.
Publish the signal, then trust your logs.
The shared token is cheap, and it already works with one lab. Three of the four labs have yet to read it, and the draft could expire before Google ships. We think publishers should keep their old crawler rules, lock only what someone would pay to license, and run a weekly ledger. A promised date counts once your own server logs confirm it.