Is my robots.txt blocking AI crawlers?
Your robots.txt blocks an AI crawler if a group naming that bot (or the '*' group it falls under) has 'Disallow: /'; open yoursite.com/robots.txt and look for groups that name AI crawlers with 'Disallow: /', or run SiteScore's free mini-check, which tests 7 major crawlers in seconds.
The bots that matter for AI search
Two kinds: search crawlers and the crawlers AI assistants use to search and fetch pages when answering, which decide whether you can be found and quoted; and training crawlers, which collect pages for model training. Blocking training crawlers is a choice, not an error. Our audit checks 11 of them and reports any that are blocked.
Lines that allow them
'User-agent: *' followed by 'Allow: /' lets every crawler in; add 'Disallow:' lines only for private folders, and add 'Sitemap: https://yoursite.com/sitemap.xml'. Our full report includes a ready-to-paste version.
How often robots.txt is the problem
Less often than you might think. When we checked 48 small-business homepages on 14 October 2026, only 1 blocked an AI crawler in robots.txt. Missing business schema (18 of 48), missing or repeated H1s (12) and missing meta descriptions (9) were far more common, so check those too.
The block robots.txt doesn't show
Two of the first sites we tested (on SiteGround and on Cloudflare) served bot-challenge pages to our crawler even though robots.txt allowed everything. Verified search crawlers are usually let through; smaller AI crawlers may not be. The audit flags this.
Run the free mini-check See audit prices
Last updated .