SureRank – Robots.txt File 

SureRank automatically generates a clean and optimized robots.txt file for your website, ensuring that search engine crawlers access only what they need, improving crawl efficiency, and protecting sensitive or unnecessary pages from being indexed.

What is robots.txt?

The robots.txt file is a simple text file located at the root of your domain that gives instructions to search engine bots on which pages or directories they are allowed (or disallowed) to crawl.

SureRank creates this file dynamically based on your site’s SEO settings, offering you the right balance between visibility and protection.

Edit Your Robots.txt File

You can edit your site’s robots.txt file directly from the WordPress dashboard using SureRank.

1. Navigate to SureRank → Advanced Settings.
2. Scroll to the Robots.txt Editor section.
3. Add or modify the directives as needed.
4. Click Save Changes.

For example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap.xml

How to View Your Robots.txt File?

To check if the file is active on your site:

1. Go to your website URL.
   
2. Add /robots.txt to the end of your domain.
   

Example:

https://yourwebsite.com/robots.txt

If SureRank is active, you’ll see a file like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap.xml

Why Aren’t My Robots.txt Changes Appearing?

If your changes don’t appear after saving, your website may already have a physical robots.txt file in its root directory.

When a physical robots.txt file exists, the web server serves that file instead of the virtual robots.txt generated by SureRank. As a result, any changes made in the SureRank Robots.txt Editor will not be reflected on your website until the physical file is removed or renamed.

How to Check if a Physical Robots.txt File Exists

You can check whether your website has a physical robots.txt file by following these steps:

1. Log in to your hosting account.
2. Open File Manager or connect to your server using FTP.
3. Navigate to your website’s root directory (commonly public_html or the WordPress installation directory).
4. Look for a file named:

robots.txt

If this file exists, it is overriding the robots.txt managed by SureRank.

How to Resolve the Conflict

If you want SureRank to manage your robots.txt file, follow these steps:

1. Create a backup of the existing robots.txt file.
2. Delete or rename the physical robots.txt file in your site’s root directory.
3. Go back to SureRank → Advanced Settings → Robots.txt Editor.
4. Save your desired robots.txt rules again.
5. Visit:

to confirm that the updated content is being served.

Note: If your hosting provider or another plugin automatically creates a physical robots.txt file, you’ll need to remove or update that file before SureRank’s Robots.txt Editor can take effect.

Troubleshooting Tips

If your changes still don’t appear after removing the physical robots.txt file:

• Clear your website, server, or CDN cache.
• Refresh the robots.txt page in an incognito/private browser window.
• Temporarily disable any caching plugins and check again.
• Ensure no other SEO or security plugin is generating or managing the robots.txt file.