EntSun News

Robots.txt one line AI training. Anthropic, Microsoft, Google and OpenAI back the concept

EntSun News/11108224
How Lab451 Generator handles it: an opt-in training preference for robots.txt. No preference is the default; choose to opt out of the whole site or all but some paths. When you opt out, the per-crawler training groups are written to match.

VICTORIA, British Columbia - EntSun -- Lab451 Helps Website Owners Get Ready for the New AI-Training Signal in robots.txt Backed by Microsoft, OpenAI, Google and Anthropic

Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token

One line instead of a growing block list

Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.

More on EntSun News
A site that wants to opt out of AI training would add a group such as:

User-Agent: __usage_ai-training
Disallow: /

AI Training, Generate, Verify, Monitor
  • AI Training (new): Lab451's generator offers the __usage_ai-training group as an opt-in option clearly labelled "IETF draft". The checker recognizes and validates the token in existing robots.txt files, monitoring alerts when it is added, removed or malformed, and a short explainer page walks site owners through what it means.
  • Generate: Build four files from one crawl of a site — robots.txt, sitemap.xml, llms.txt and llms-full.txt — with robots.txt as the entry point that points search engines and AI agents to the other three.
  • Verify: Every generated file is checked for dead links and hostnames, unregistered package names, and text hidden from human readers before it is published. The free Checker runs the same validation on files a site already publishes.
  • Monitor: Pro and Agency plans watch published files six times a day, send a weekly report by email, and notify the owner immediately when a critical problem appears — such as a robots.txt change that accidentally blocks search engines or drops the AI-training preference.

Availability

More on EntSun News
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.

Media contact

Roger Remacle
hello@lab451.org
https://lab451.org

Contact
Roger Remacle
***@lab451.org


Source: Lab451

Show All News | Disclaimer | Report Violation

0 Comments

Latest on EntSun News