Skip to content
SEO Audit Checker

robots.txt: how to set it up without blocking Google

robots.txt is a plain text file at the root of your site that tells crawlers which parts of the site they may visit.

Why it matters

A single Disallow: / under User-agent: * blocks every search engine from your entire site. This is a common leftover from development sites.

AI search tools such as ChatGPT, Claude and Perplexity use their own crawlers. If robots.txt blocks them, your content can’t be read or cited in AI answers. Whether to allow them is your choice, but it should be a deliberate one.

If robots.txt returns a server error, Google may stop crawling the site until it can read the file again.

How to fix it

  1. Create a file named robots.txt at the root of your domain, for example https://www.example.com/robots.txt.
  2. Allow everything by default and only disallow private areas like admin or cart pages.
  3. Add a Sitemap: line with the full URL of your XML sitemap.
  4. Decide on AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others) and add explicit rules for any you want to block.
  5. Test URLs against your rules with our robots.txt tester before publishing changes.
Safe default robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Related guides