How to fix: Pages that were blocked from crawling

blocked from crawling Semrush site audit tips

Issue: Pages blocked from crawling by search engines (due to robots.txt or noindex tags) won’t appear in search results, limiting your visibility.

Fix: Review your robots.txt and noindex tags to ensure that no valuable content is accidentally blocked from crawling.

Recommendations

  1. Check robots.txt: Verify that only pages you intentionally want hidden (e.g., admin pages) are blocked, and valuable content pages are accessible.
  2. Review noindex Tags: Make sure pages you want to rank don’t have noindex tags in their HTML.
  3. Run a Site Audit: Use Google Search Console or an SEO tool to check for blocked pages and confirm that key pages are open to search engines.
  4. Monitor Regularly: After updates, ensure that newly added or modified pages are not mistakenly blocked.

Tip: Allowing important pages to be crawled ensures they can appear in search results, helping with overall site visibility.

Related Posts