update on ai.robots.txt: the file (https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt) now contains 139 crawlers as opposed to the 129 back in december of 2025, and 37 back in february 2025. not a big difference but still worth updating your site's robots.txt file whenever you get the chance.
13 likes
Website Stats
Last updated
1 hour ago
CreatedFeb 3, 2025
![[N6] avatar](/site_screenshots/22/24/blankbarrel/index.html.50x50.webp)
![backto00.exe [v5.0] - Image Destroyer avatar](/site_screenshots/38/40/backto00/index.html.50x50.webp)
Thanks king! 🫡
Oh shit, I completely forgot to make one of these. Can I use that template as-is, or should I add more stuff too?
you can copy it as is and it will work just fine. the file basically lists out all ai-bot user agents, and the "Disallow: /" basically says "instruct those user-agents to not browse root (a.k.a. the entire site)"