Popular on EntSun
- Disney Streamer Network Establishes Creator Code of Conduct - 379
- Commercial Manufacturing Validation Completed as November 10 FDA Milestone Approaches; Critical FDA Milestone Close: NRx Pharmaceuticals NAS DAQ: NRXP - 240
- Western Canadian Host James Cousineau Finishes 11th In Global "Top Host" Competition - 225
- Majo Aguilar inaugurates the Paseo de las Estrellas at Plaza Mexico with the unveiling of her Estrella Maya - 198
- Highsfield Showcases Earth Zoom Generator for Cinematic AI Videos - 186
- TÀBÍRA Studio Announces Permanent UK Entry and Upcoming London Showcase - 176
- Bacchus Wine & Spirits and Michelin Guide–Recommended Ethel's Fancy to Host Exclusive LOUIS XIII Limited Edition Diwali Decanter Celebration - 175
- OncoBeta Introduces Rhenium-SCT for Non-Melanoma Skin Cancer at Theageneio Cancer Hospital of Thessaloniki - 172
- The Return of Plaza México's 'Paseo de Las Estrellas' Honoring Recording Artist Majo Aguilar - 170
- Former Goddard Enterprises Managing Director Anthony Ali Joins STS Capital Partners as Managing Director and Partner - 165
Robots.txt one line AI training. Anthropic, Microsoft, Google and OpenAI back the concept
EntSun News/11108224
How Lab451 Generator handles it: an opt-in training preference for robots.txt. No preference is the default; choose to opt out of the whole site or all but some paths. When you opt out, the per-crawler training groups are written to match.
VICTORIA, British Columbia - EntSun -- Lab451 Helps Website Owners Get Ready for the New AI-Training Signal in robots.txt Backed by Microsoft, OpenAI, Google and Anthropic
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on EntSun News
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
Availability
More on EntSun News
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on EntSun News
- "Martin" Star Bruh Man and R&B Star Alisa Peoples-Yarbrough Join "Mother of Inmates" — Featured on FOX 4 News
- "Dick Dyman: Head Hunter" Episode 2, "The Client," Premieres November 28 on YouTube
- Pixel Swarm Drones Illuminates the Sky for The Bath Club's 100th Anniversary Celebration
- 12 Points Asia Launches Fan Hub After Historic First Eurovision Asia Contest
- Best Private Vacation Rentals in St. George, Utah: Experience Luxury Near Zion & Sand Hollow
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
- AI Training (new): Lab451's generator offers the __usage_ai-training group as an opt-in option clearly labelled "IETF draft". The checker recognizes and validates the token in existing robots.txt files, monitoring alerts when it is added, removed or malformed, and a short explainer page walks site owners through what it means.
- Generate: Build four files from one crawl of a site — robots.txt, sitemap.xml, llms.txt and llms-full.txt — with robots.txt as the entry point that points search engines and AI agents to the other three.
- Verify: Every generated file is checked for dead links and hostnames, unregistered package names, and text hidden from human readers before it is published. The free Checker runs the same validation on files a site already publishes.
- Monitor: Pro and Agency plans watch published files six times a day, send a weekly report by email, and notify the owner immediately when a critical problem appears — such as a robots.txt change that accidentally blocks search engines or drops the AI-training preference.
Availability
More on EntSun News
- Turkish Engineer Who Cracked Wall Street's Algorithm Launches Hermes AI Analyzer
- Nious Media Expands Gaming Coverage With 2026 Game & Geek Expo Special
- UK Financial Ltd Launches MayaSwap On MAYA CHAIN To Facilitate UKFLMS-For-Gold Swaps, Officially Transforming UKFLMS From Meme Into A Real-World Asset
- La'Nardo Myrick Sr Salutes House of Myrick Collaborators and Announces Two Baroque Albums
- Producer 9-0 Credits Southern Miss and Ole Miss as Baroque Project Takes Shape
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Source: Lab451
0 Comments
Latest on EntSun News
- Commercial Sales for U.S.-Made, Preservative-Free Ketamine Amid Long-Runnin Shortage Following FDA Approval by NRx Pharmaceuticals, Inc
- Independent Producer/Writer Karen Morris Announces The Swingin' Angle
- The King Returns: Satirical December 2026 Rematch Sees Challenger Delayed by a Christmas Light Installation
- Last Night of the Proms 2027 Tickets and Private Boxes on Sale at the Royal Albert Hall
- LCH2 Awarded Additional $1 Million to Advance Lewis County Clean Energy Campus
- Security Planning Tips For Your Holiday Event From Access Patrol Services
- Contracting Resources Group Receives 2026 HIRE Vets Platinum Medallion Award
- It's a Wonderful Life: Live in Door County Marks a Milestone for TAP
- Business News & Info Goes Inside Paragon FX Group's Hunt for Original Film Props
- Las Vegas Comedy Headliner Don Barnhart Returns to Pechanga Thanksgiving Weekend Nov. 27–28
- Kitchen Table Economics: An Invitation to Think Differently
- Who Won New York Fashion Week SS27? Tommy Hilfiger Led the Media as Tory Burch Won the Critics
- Win a $5,000 VIP Family Experience at the 100th Thanksgiving Day Parade in New York City
- CB Stuffer Announces Private Sale to Secure Future of Brand Growth
- Brother Marcus Celebrates 400th Episode of The Elements of Inspiration Podcast
- The Swarthy Channel Opens The Swarthy Studio for Creators Raising Capital in Entertainment
- Connecting is our passion. — From Power to Data, Taking Connectivity to the Next Generation
- Gayle Skidmore Partners with RM Records; New Single "Old Photographs," Recorded in Prague, Out November 13 via Virgin Music Group
- The Fifth Annual "Bamboo Lantern Festival" to Illuminate Tokyo Mizumachi and Sumida River Walk®
- MetroLagoons launches College Lagoon Pass for students