robots.txt blocks AI crawlers

Medium prioritySocial & AI search

robots.txt disallows AI crawlers like GPTBot, ClaudeBot, PerplexityBot or Google-Extended from your site.

What triggers this issue

• One or more known AI crawler user agents are disallowed from the site root.

Why it matters

AI assistants like ChatGPT, Claude, Perplexity and Google’s AI features increasingly answer questions and recommend sites. If their crawlers can’t read your pages, they are less likely to cite or link to you.

Blocking can be a deliberate choice to keep content out of AI training; if so, ignore this warning. Some crawlers are used for live search answers rather than training, so you can allow those and block others.

How to fix it

1. Decide which AI crawlers you want to be visible in.
2. Remove the Disallow rules for those user agents, or add explicit Allow groups.
3. Test the result with our robots.txt tester, which includes the major AI crawlers.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /