robots.txt missing, erroring or blocking the site
Medium priorityTechnical
robots.txt is missing, returns a server error, or disallows Google from the whole site.
What triggers this issue
- No robots.txt file (404): not required, but it’s where you point crawlers to your sitemap.
- robots.txt returns a server error (5xx): Google pauses crawling the whole site.
- robots.txt blocks Google from the whole site with
Disallow: /.
Why it matters
robots.txt is the first file search engines request. A 5xx response makes Google treat the whole site as off-limits until it recovers, and Disallow: / (often left over from a staging site) removes everything from results over time.
How to fix it
- Serve /robots.txt with status 200, or 404 if you have no rules.
- Remove
Disallow: /for Googlebot and*unless the site is meant to be hidden. - Add a
Sitemap:line, and test rules with our robots.txt tester.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml