robots.txt tester
Test any URL against your robots.txt for Googlebot, Bingbot and AI crawlers. Runs in your browser; nothing is uploaded.
Every crawler, same URL
How rules are matched
This tester follows Google’s robots.txt rules: a crawler uses the most specific User-agent group that names it, falling back to *. Within that group, the longest matching rule wins, andAllow wins a tie. * matches any characters and $ marks the end of the URL.
Blocking a page in robots.txt stops crawling, not indexing: a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search. Read ourrobots.txt guide for a safe default file.