How to Fix Robots.txt File Accessibility Issues in SureRank

If you’ve received a critical error in SureRank stating: “Robots.txt file is not accessible”.

This guide will help you understand what this means, why it’s important for SEO, and how to fix it even if you’re not a technical expert.

What’s the Problem

The robots.txt file is a simple text file in your website’s root directory that tells search engines which pages or sections of your site to crawl or avoid.

If this file is inaccessible:

• Search engines may not index your site properly.
• Important pages might be ignored or blocked from search results.
• SEO performance can be negatively affected.

Common causes:

• robots.txt file is missing from the root folder.
• Incorrect file permissions are preventing access.
• Server or security rules are blocking search engine crawlers.

Why This Matters

• Search engine visibility: If robots.txt is blocked, Google or other search engines may not index your pages correctly.
• Crawl efficiency: Search engines may waste resources crawling pages you don’t want them to or skip important pages entirely.
• SEO health: Without proper crawling, your site’s search ranking potential is reduced.

How to Fix

The Robots.txt file is automatically handled by SureRank after activation. If you don’t see it, follow the steps below.

Step 1 – Access the Robots.txt Editor

• Go to your WordPress dashboard.
• Navigate to SureRank → Advanced Settings → Robots.txt Editor.

Step 2 – Edit Robots.txt

• If the file doesn’t exist, SureRank will allow you to create a default robots.txt file.
• You can add rules to allow search engines to crawl your site. Example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap_index.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap_index.xml

Step 3 – Save Changes

• Click Save Changes to apply your edits.
• If you use a caching plugin, clear your site cache to ensure the updated robots.txt is served.

Step 4 – Check File Permissions

• Ensure the robots.txt file is readable by the web server.
• Recommended permissions: 644 (owner can write, everyone can read).

Step 5 – Test Robots.txt Accessibility

• Open your browser and visit: https://yourwebsite.com/robots.txt
• Ensure there are no 403 or 404 errors.

Step 6 – Re-run Site SEO Analysis

• After making changes, Re-run Checks in SureRank.
• The warning should disappear. Robots.txt file is accessible.