How to fix: Pages that were blocked from crawling

Updated on December 9th, 2024 at 11:18 pm

Estimated reading time: 1 minute

Issue: Pages blocked from crawling by search engines (due to robots.txt or noindex tags) won’t appear in search results, limiting your visibility.

Fix: Review your robots.txt and noindex tags to ensure that no valuable content is accidentally blocked from crawling.

Recommendations

1. Check robots.txt: Verify that only pages you intentionally want hidden (e.g., admin pages) are blocked, and valuable content pages are accessible.
2. Review noindex Tags: Make sure pages you want to rank don’t have noindex tags in their HTML.
3. Run a Site Audit: Use Google Search Console or an SEO tool to check for blocked pages and confirm that key pages are open to search engines.
4. Monitor Regularly: After updates, ensure that newly added or modified pages are not mistakenly blocked.

Tip: Allowing important pages to be crawled ensures they can appear in search results, helping with overall site visibility.

More articles relating to Blocked resources:

• How to fix: Issues with blocked internal resources in robots.txt
• How to fix: Pages that were blocked from crawling
• How to fix: Pages blocked by X-Robots-Tag: noindex HTTP header
• How to fix: Issues with blocked external resources in robots.txt