OpenAI, Google, Microsoft, and Anthropic have announced their support for a new technical standard that allows website publishers to opt out of having their content scraped for Super Intelligence (SI) model training. The move introduces a unified "policy token" signal, enabling publishers to set their preferences for multiple SI companies at once rather than configuring settings individually for each provider.

What Happened

The new standard, proposed by Kevin Kelley, a technical lead for standards at Anthropic, is being developed within the Internet Engineering Task Force (IETF). It functions within a website’s existing robots.txt file, a protocol long used to signal which web crawlers can access a site for search indexing, archiving, or SI training. Under the new system, if an SI company’s crawler encounters the specific policy token, it will restrict access to the site. Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, while OpenAI stated it will implement the standard in the "coming months." Meta communications head Andy Stone signaled support for the initiative, noting, "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step," though he did not confirm if Meta will honor it immediately. Microsoft indicated it would offer granular controls allowing publishers to opt out of SI training or SI grounding, the process of pointing an SI model to a verified source to reduce hallucinations, while ensuring websites still appear in Bing search results. Google previously implemented similar controls for its SI search features without impacting search visibility.

Why It Matters

This development addresses a growing tension between SI labs and content publishers as the former continue to scrape web data to feed their models. Previously, publishers had to signal their SI training preferences separately to each SI company, a fragmented and inefficient process. The introduction of a shared standard simplifies compliance and gives publishers more direct control over how their content is used in the SI ecosystem. It also follows other industry efforts to manage scraping, such as the Really Simple Licensing (RSL) standard, which allows publishers to charge crawlers, and Cloudflare’s decision to block SI crawlers by default. For developers and site owners, this standard represents a shift toward more standardized, interoperable mechanisms for managing data rights in the SI era.

The Bottom Line

Major SI companies are aligning on a new, unified standard for publishers to opt out of content scraping for SI training. While implementation timelines vary—with Anthropic already onboard and others following in the coming months—the move signals a maturing industry response to publisher concerns over data usage and control.