How to use this
The block above goes in the file at yoursite.com/robots.txt. If you already have a robots.txt, paste these rules into it, do not replace the whole file, or you will wipe out your existing sitemap and crawl rules. If you do not have one yet, this can be the whole file. Every one of the twelve bots gets an explicit entry so nothing is left to guesswork: Disallow: / blocks a crawler entirely, and an empty Disallow: allows it.
One thing this file cannot do: it cannot override a firewall. If Cloudflare or your WAF is set to block AI bots, it returns a 403 before robots.txt is ever read, and the crawler never gets your rules. So after you deploy this, run the AI crawler checker to confirm it reads the way you meant, and if a bot still shows as blocked, check your CDN. The full breakdown of every bot and the mistakes people make is in our guide to which AI crawlers can reach your site.