The SEO Bench
ai.robots.txt: drop-in files to block the AI crawlers you do not want
A community-maintained, continuously updated list of AI crawlers, shipped as drop-in robots.txt, htaccess, Nginx and Caddy files so you can block or allow AI bots in minutes.
New AI crawlers show up faster than anyone can track by hand. GPTBot, ClaudeBot, Bytespider, PerplexityBot, and dozens more, each with its own user agent, each changing behaviour without much notice. Keeping a robots.txt current against that moving target is a tedious job nobody wants, so a community stepped in and started maintaining the list for everyone.
Why it made the cut
ai.robots.txt does the unglamorous, essential work of tracking every known AI crawler user agent and shipping it as files you can drop straight into your stack: a ready-made robots.txt, plus htaccess, Nginx, Caddy and HAProxy config equivalents for servers that need to enforce blocks at the server level rather than trust a crawler to respect robots.txt. There is also a machine-readable robots.json, useful if you want to build the blocking logic into your own tooling instead of pasting a static file.
How to use it
- Go to the repository and pull the current robots.txt, or the config format that matches your server.
- Merge it into your existing robots.txt, or drop the server-config snippet into your Nginx/Caddy/HAProxy setup.
- For programmatic use, pull robots.json and build your own allow or block logic against it.
- Pair it with an llms.txt file: robots.txt tells crawlers what they may not take, llms.txt tells the ones you welcome how to read you. Together they cover both sides of the GEO equation.
- Re-pull periodically since the bot list is updated frequently as new crawlers appear.
Worth knowing
The project is fully free and open source under the MIT licence, with roughly 4,000 GitHub stars and 170 forks. It is actively maintained, the latest release at the time of writing is v1.47, published 9 July 2026, with frequent updates to the bot list.