Fixing robots.txt Issues in SureRank

If your robots.txt file is not loading correctly, or SureRank’s Site Check is warning that it is missing or inaccessible, this guide will help you find and fix the cause. Most of the time, this happens because something on your server or hosting setup is answering the request before SureRank has a chance to, not because of a bug in the plugin.

This guide covers two situations:

• Your robots.txt page shows a 404 error or the wrong content
• SureRank’s Site Check says robots.txt is “missing or inaccessible”

Situation 1: robots.txt Shows a 404 Error or the Wrong Content

Cause 1: An Old robots.txt File Already Exists on Your Server

This is the most common reason. If your website already has a real robots.txt file sitting on the server, your site will always show that old file, and SureRank’s version will never be used, no matter what you have configured in the plugin.

How to check:

Go to SureRank, then Settings, then Advanced, then Tools, then Robots.txt Editor. If an old file is present, you will see a message letting you know, and the editor will be locked until it is removed.

How to fix it:

Delete the old robots.txt file from your server. You can do this using an FTP program or your hosting provider’s file manager. Once it is removed, visit yoursite.com/robots.txt again to confirm SureRank’s version now appears.

Cause 2: Your Hosting Provider or Security Service Is Blocking the Page

Some hosting providers and security services (like Cloudflare or Sucuri) can intercept robots.txt before it even reaches your site, showing a cached version, a security check, or a blocked page instead.

How to check:

Try loading yoursite.com/robots.txt in a private or incognito browser window, or from a different device or network. If it looks different from what loads normally, this is likely the cause.

How to fix it:

Contact your hosting provider or check your CDN/security dashboard, and ask them to allow robots.txt to load without being blocked or cached. If you recently updated your robots.txt content in SureRank, you may also need to clear your site’s cache.

Cause 3: Your Site Is Set to Hide From Search Engines

If your WordPress site is set to discourage search engines from indexing it, WordPress will show a robots.txt file that blocks everything, regardless of what SureRank has configured.

How to check:

Go to Settings> Reading in your WordPress dashboard. Look for the option labeled “Discourage search engines from indexing this site.”

How to fix it:

If you want your site to be visible in search results, uncheck this option and save your changes.

Situation 2: Site Check Says robots.txt Is “Missing or Inaccessible”

This message comes from a separate check that SureRank runs to confirm your robots.txt file is publicly reachable. Your robots.txt can load perfectly fine in a browser while still triggering this warning, because the check works a little differently.

Cause 1: You Are on a Local, Staging, or Password-Protected Site

If your site is still in development, running locally, or protected by a password, SureRank may not be able to access it to confirm it is working.

How to check:

If robots.txt loads fine when you visit it directly in a browser, but the warning still appears, this is most likely the reason.

How to fix it:

This is expected while your site is not yet live or publicly accessible. The warning will go away once your site is on a public domain without a password wall.

Cause 2: A Security Check Is Blocking the Request

If your hosting provider or a security service shows a challenge page or blocks automated requests, SureRank’s check will read this as the file being unreachable, even if a real visitor could load it normally.

How to fix it:

Ask your hosting provider or check your CDN/security dashboard to allow requests to robots.txt.

Cause 3: You Are Using an Older Version of SureRank

Older versions of SureRank had a stricter check that sometimes showed this warning even when everything was working correctly.

How to fix it:

Update SureRank to the latest version from your WordPress dashboard.

Quick Reference

What you are seeing	What is likely causing it	Where to look
robots.txt shows a 404 or the wrong content	An old file already on your server	Robots.txt Editor in SureRank
robots.txt loads fine directly, but looks different elsewhere	Hosting or security service blocking it	Your host or CDN/security dashboard
robots.txt unexpectedly blocks everything	Site set to hide from search engines	Settings, then Reading
Site Check warning, but the page loads fine in a browser	Local/staging site, or a security block on automated requests	Your site’s environment, or your host’s security settings
Site Check warning and the page genuinely does not load	Old file, security block, or password-protected site	Combine the checks above

Still Need Help?

If you have gone through these steps and the issue is still not resolved, reach out to our support team with a link to your site, and we will be happy to take a closer look.