TL;DR: A small text file at the root of your domain that tells search bots which URLs to crawl.
In a nutshell
A robots.txt file gives crawlers like Googlebot instructions about which paths they may or may not request. It controls crawl access, not indexing, so a blocked page can still appear in results without a snippet. Example: blocking /cart/ and /checkout/ stops bots wasting time on pages that should never rank, freeing crawl budget for product pages.
Quick answer: robots.txt is a plain text file at the root of your domain that tells search engine bots which parts of your site they may or may not crawl. It does not control what is indexed (use noindex tags for that); it controls crawl access. Used wrongly, robots.txt can quietly block search engines from finding important parts of your site.
What Is Robots.txt?
Robots.txt is a plain text file located in the root directory of your website. It gives instructions to search engine crawlers (like Googlebot or Bingbot) about which parts of your site they’re allowed or disallowed to crawl. It’s one of the first files bots check when they visit your site, making it a critical tool for managing your site’s crawl behaviour.
It’s not a security feature, it doesn’t block access from humans, nor does it guarantee exclusion from indexing, but it’s a valuable part of your Technical SEO toolkit when used properly. The official spec lives at the Google Search Central robots.txt documentation.
Where Is Robots.txt Located?
The file is placed in the root of your domain. For example:
https://www.example.com/robots.txt
Every website can have only one robots.txt file, and it applies to the entire domain (including subfolders). If this file is missing, search engines assume they can crawl everything that isn’t blocked by other methods like meta robots or canonical tags.
What Can You Do With Robots.txt?
Here are some of the things you can control using your robots.txt file:
- Block crawlers from accessing certain pages or directories
- Allow specific bots while blocking others
- Point bots to your XML sitemap
- Prevent crawling of duplicate or unimportant content
This makes robots.txt especially useful for large sites or sites that generate lots of parameter-based URLs, like ecommerce stores using Shopify or WooCommerce.
How Robots.txt Works
The file consists of a set of simple rules that define:
- User-agent: Which crawler(s) the rule applies to
- Disallow: Which paths should not be crawled
- Allow: Exceptions to disallowed rules
- Sitemap: A link to your XML sitemap
Example:
User-agent: *
Disallow: /checkout/
Disallow: /cart/
Allow: /blog
Sitemap: https://www.example.com/sitemap.xml
This file tells all bots (“*“) not to crawl the /checkout/ or /cart/ paths but allows the blog section and links them to the sitemap.
When to Use Robots.txt
Some ideal use cases for robots.txt include:
- Blocking duplicate content pages, such as filtered product listings
- Preventing crawlers from indexing staging environments
- Reducing crawl waste on login, cart, or thank-you pages
- Directing bots to your primary XML sitemap
At BrisTechTonic, we always review the robots.txt file during a technical SEO audit. We often find important pages accidentally blocked, or worse, nothing restricted at all on a 200,000-page site.
What Robots.txt Can’t Do
It’s important to remember that robots.txt only controls crawling, not indexing. That means:
- If a page is blocked by robots.txt but linked from other pages, it can still be indexed (but without any content shown)
- Robots.txt doesn’t stop users from accessing a page or downloading files
- It’s a polite request, not all bots respect it (especially bad ones)
If you want to prevent indexing, use a noindex directive in the meta robots tag or response header. Learn more in our Noindex Tag glossary entry. See also indexability.
Robots.txt and Crawl Budget
One of the best uses for robots.txt is preserving your crawl budget. By blocking low-value pages like pagination, sort filters, or thank-you pages, you allow bots to focus their time on your most valuable content.
This is especially helpful on ecommerce sites, blogs with heavy tagging, or international websites with lots of language variations. Not sure what to block? Our SEO consultancy sessions help you figure it out.
Common Robots.txt Mistakes
- Blocking CSS or JS files – This prevents Google from rendering pages properly
- Disallowing important pages by mistake – Like your entire blog or /products/ folder
- Blocking a page you also want to “noindex” – Google won’t see the noindex tag if the page is disallowed
- Using wildcards incorrectly – Syntax errors can lead to unexpected blocking
- Disallowing everything – Always check with the Google Search Console robots.txt Tester before publishing
Monitoring Robots.txt with Search Console
Google Search Console includes a robots.txt testing tool (found in the “Legacy tools & reports” section). Use this to:
- Test specific URLs for crawlability
- Check for syntax errors
- Validate your latest robots.txt version
We also recommend testing with Screaming Frog or Sitebulb to simulate how different bots handle your site.
Robots.txt and WordPress
By default, WordPress generates a virtual robots.txt file if one doesn’t exist. Plugins like Rank Math let you manage and edit robots.txt directly from your dashboard, which is handy if you’re using WordPress SEO services. See the WordPress.org documentation for the underlying virtual file behaviour.
Using Robots.txt with Other Directives
Robots.txt is just one tool in the content control toolbox. Here’s how it works alongside others:
- Robots.txt – Blocks crawling of specific paths
- Meta robots tags – Controls indexing and follow behaviour
- Canonical tags – Consolidates duplicate content signals
- Sitemaps – Actively direct bots to your preferred URLs
Combining these techniques ensures search engines crawl and index exactly what you want, no more, no less. You can learn how to set this up yourself in our SEO Blueprint course.
Related Glossary Terms
Frequently Asked Questions
What does robots.txt actually control?
Crawl access. It tells bots which URLs they should or should not request. It does NOT prevent indexing on its own; URLs blocked by robots.txt can still appear in search results if Google discovers them via backlinks. To prevent indexing, use a noindex meta tag on the page itself, which means the page must NOT be blocked by robots.txt (Google needs to crawl the page to see the noindex).
Where does the robots.txt file live?
At the root of your domain, exactly: yoursite.com/robots.txt. It must be at the root; subdirectory locations are ignored. Each subdomain needs its own robots.txt (a different file at blog.yoursite.com/robots.txt).
Can robots.txt hurt SEO?
Yes, easily. The most common mistake is accidentally blocking important pages or sections. The second most common is blocking CSS, JavaScript, or image folders, which prevents Google from properly rendering and understanding the page. Always test robots.txt rules using Search Console’s robots.txt Tester before pushing changes live.
What should a basic WordPress robots.txt look like?
For most WordPress sites: User-agent: * / Allow: / / Disallow: /wp-admin/ / Allow: /wp-admin/admin-ajax.php / Sitemap: https://yoursite.com/sitemap.xml. This blocks the admin area while allowing AJAX requests and points to the sitemap. Modify only with deliberate intent.
Take this further
robots.txt is technical SEO hygiene. Misconfigured files have hidden important pages from Google for years before getting noticed; reviewing yours during any SEO audit is a five-minute task that prevents major issues.
Ready to apply this to your own site? Book a free discovery call, or explore our SEO strategy service.