How to fix: Robots.txt not found

robots not found Semrush site audit tips

Issue: Without a robots.txt file, search engines may crawl unnecessary or sensitive content, wasting crawl budget and potentially exposing private information.

Fix: Create a robots.txt file to guide search engines on what to crawl and what to ignore.

How to Fix for Beginners

  1. Create a robots.txt File: Add a plain text file named robots.txt to your site’s root directory (e.g., https://example.com/robots.txt).
      • Example: 
        User-agent: *
Disallow: /private/
Allow: /
  2. Specify Directives: Use Disallow to block specific files or directories and Allow to specify what can be crawled.
    • Example: Block search engines from crawling your admin area with:Disallow: /admin/
  3. Test the File: Use Google’s robots.txt Tester in Google Search Console to ensure your file works correctly.
  4. Upload to Root Directory: Place the robots.txt file in the main directory of your website so search engines can find it.
    • Example: Upload the file to https://example.com/robots.txt.
  5. Monitor for Issues: Regularly review and update the file as your site structure changes.

Tip: A well-configured robots.txt file optimizes crawl efficiency and protects sensitive content.

Related Posts