- How to Fix WWW and Non-WWW Version Redirects to Improve Your SEO
- How to Fix: No H1 Heading Found on Your Homepage
- How to Fix Missing H2 Headings on Your Homepage
- Re-run Checks Button in SureRank
- Fix Critical Error: Another SEO Plugin Detected in SureRank
- Fix Warning: Site Tagline Is Not Set in SureRank
- How to Fix Multiple SEO Plugins Detected on Your Site
- How to Fix: Homepage is Not Indexable by Search Engines
- Warning: Homepage Does Not Contain Internal Links
- How to Fix Missing Alt Text on Homepage Images
- How to Fix: Search Engine Title is Missing on the Page
- Page Level SEO: Broken Links Detected
- How to Fix Missing Alt Text on Images
- How to Fix Page URLs That Are Too Long
- Page Level SEO Warning: No Links Found on This Page
- Page Level SEO Warning: No Images or Videos Found
- Page Level SEO Warning: Missing Search Engine Description
- Page Level SEO Warning: No Subheadings Found on This Page
- Page Level SEO Warning: Canonical Tag is Missing
- Page Level SEO Warning: Open Graph Tags Missing
- Customize Final Title Tag surerank_final_title
- Customize Post Type Archive Title Output surerank_post_type_archive_title
- Customize Archive Page Title with surerank_archive_title
- CustomizeModify the Search Results Page Title surerank_search_title
- Customizing the 404 Page Title surerank_not_found_title
- Customizing the Title Separator surerank_title_separator
- How to Remove Archive Prefixes from Titles Using SureRank
- Customize Homepage Pagination Format surerank_homepage_pagination_format
- Customize Maximum SEO Title Length surerank_title_length
- Enable/Disable Pagination in SureRank Archives surerank_show_pagination
SureRank – Robots.txt File
SureRank automatically generates a clean and optimized robots.txt file for your website, ensuring that search engine crawlers access only what they need, improving crawl efficiency, and protecting sensitive or unnecessary pages from being indexed.
What is robots.txt?
The robots.txt file is a simple text file located at the root of your domain that gives instructions to search engine bots on which pages or directories they are allowed (or disallowed) to crawl.
SureRank creates this file dynamically based on your site’s SEO settings, offering you the right balance between visibility and protection.
How to View Your Robots.txt File?
To check if the file is active on your site:
- Go to your website URL.
- Add /robots.txt to the end of your domain.
Example:
https://yourwebsite.com/robots.txt
If SureRank is active, you’ll see a file like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourwebsite.com/sitemap.xml
We don't respond to the article feedback, we use it to improve our support content.