Why your page isn’t indexed: noindex, nofollow and robots.txt blocks

A page can only appear in search results if search engines are allowed to crawl it and allowed to index it. One stray tag or rule can remove a page, or a whole site, from Google.

Why it matters

A noindex directive in a robots meta tag or X-Robots-Tag header tells search engines to drop the page from results. It is often left behind from a staging site or set by a “discourage search engines” option in a CMS.

A nofollow directive tells search engines not to follow any links on the page, which stops them discovering and passing value to your other pages. A Disallow rule in robots.txt stops Google from crawling the page at all.

How to fix it

1. Look in the page source for <meta name="robots" content="noindex"> and remove noindex and nofollow from pages you want in search.
2. Check your server or CDN for an X-Robots-Tag: noindex response header.
3. In WordPress, make sure Settings → Reading → “Discourage search engines from indexing this site” is unchecked.
4. Open /robots.txt and remove any Disallow rule that matches the page for Googlebot or *.
5. After fixing, request indexing in Google Search Console with the URL Inspection tool.

<meta name="robots" content="index, follow">