How to fix: Pages blocked by X-Robots-Tag: noindex HTTP header

blocked by robots Semrush site audit tips

Issue: Pages blocked by the X-Robots-Tag: noindex HTTP header can’t be crawled or indexed by search engines, preventing them from appearing in search results.

Fix: Check your HTTP headers to ensure that valuable content isn’t accidentally blocked by the X-Robots-Tag: noindex.

How to Fix for Beginners

  1. Identify Affected Pages: Use SEO tools or browser developer tools to find pages with the X-Robots-Tag: noindex header.
    • Example: Your blog page is marked with X-Robots-Tag: noindex, but it should be indexed.
  2. Review Intent: Confirm if blocking the page was intentional (e.g., admin or test pages) or a mistake.
    • Example: You might want admin-dashboard.html blocked but not your main blog page.
  3. Update the HTTP Header: Remove or modify the X-Robots-Tag: noindex directive for important pages.
    • Example: Remove X-Robots-Tag: noindex from the blog page’s server configuration or CMS settings.
  4. Check Non-HTML Files: Ensure non-HTML resources, like PDFs, that need indexing are not unintentionally blocked.
  5. Test Crawling: Use Google Search Console’s URL Inspection tool to confirm that the page can now be indexed.

Tip: Properly configuring X-Robots-Tag ensures that search engines can index valuable content while ignoring irrelevant or sensitive pages.

Related Posts