Robots.txt, explained properly.
In a nutshell
Robots.txt is a small text file at the root of your domain that tells search bots which URLs to crawl. The file controls crawl access rather than indexing. For example, blocking /cart/ and /checkout/ preserves crawl budget for your product pages.
What is robots.txt?
Robots.txt is a plain text file in the root directory that instructs search engine crawlers which sections of your site they may or may not crawl. It is not a security feature and does not guarantee exclusion from indexing. It is a valuable part of your technical SEO toolkit when used properly.
Where is robots.txt located?
It sits at the root of your domain, for example https://www.example.com/robots.txt. There is only one robots.txt file per domain, and it applies to the entire domain including subfolders. Subdomains require their own separate files.
What can you do with robots.txt?
- Block crawlers from accessing certain pages or directories. - Allow specific bots while blocking others. - Point bots to your XML sitemaps. - Prevent crawling of duplicate or unimportant content.
How robots.txt works
The file is made up of: - **User-agent:** which crawler or crawlers the rule applies to. - **Disallow:** paths that should not be crawled. - **Allow:** exceptions to disallowed rules. - **Sitemap:** a link to your XML sitemap. Example: ``` User-agent: * Disallow: /checkout/ Disallow: /cart/ Allow: /blog Sitemap: https://www.example.com/sitemap.xml ```
When to use robots.txt
- Blocking duplicate content pages, such as filtered product listings. - Preventing crawlers from indexing staging environments. - Reducing crawl waste on login, cart, or thank-you pages. - Directing bots to your primary XML sitemaps.
What robots.txt can't do
- Robots.txt only controls crawling, not indexing. - Blocked pages can still be indexed if linked from elsewhere. - It does not stop user access or file downloads. - It is a polite request, and not all bots respect it. To prevent indexing, use the noindex directive in a meta robots tag.
Robots.txt and crawl budget
Blocking low-value pages such as pagination, sort filters, and thank-you pages lets bots focus on your valuable content. This is particularly helpful for ecommerce sites, blogs with heavy tagging, and international sites.
Common robots.txt mistakes
- Blocking CSS or JS files, which prevents proper rendering. - Disallowing important pages by accident. - Blocking a page you also want to noindex. - Using wildcards incorrectly. - Disallowing everything.
Monitoring robots.txt with Search Console
Google Search Console includes a testing tool in the Legacy tools and reports section to test specific URLs, check for syntax errors, and validate your robots.txt versions.
Robots.txt and WordPress
WordPress generates a virtual robots.txt file by default if none exists. Plugins like Rank Math let you manage it from the dashboard.
Using robots.txt with other directives
- **Robots.txt:** blocks crawling of specific paths. - **Meta robots tags:** control indexing and follow behaviour. - **Canonical tags:** consolidate duplicate content signals. - **Sitemaps:** direct bots to your preferred URLs.
Robots.txt: common questions.
What does robots.txt actually control?
Crawl access. It tells bots which URLs they should or should not request. It does not prevent indexing on its own. Blocked URLs can still appear in search results if discovered via backlinks. Use noindex meta tags to prevent indexing.
Where does the robots.txt file live?
At the root of your domain, exactly: yoursite.com/robots.txt. It must be at the root, and a subdirectory location is ignored. Subdomains need their own separate files.
Can robots.txt hurt SEO?
Yes. Common mistakes include accidentally blocking important pages or blocking CSS, JavaScript, or images, which prevents proper page rendering. Always test before publishing.
What should a basic WordPress robots.txt look like?
A typical example blocks /wp-admin/ while allowing /wp-admin/admin-ajax.php and points to the sitemap. Modify it only with deliberate intent.
What clients say about working with us.
“I've had the pleasure of working with Chris on several SEO projects for my websites in the health field. He was definitely been able to demystify things so that I felt able to make decisions with him and feel confident they were the right ones. For a non-techie, that is worth gold! I look forward to working with Chris again and warmly recommend him to anyone who - like me - is unsure about all things SEO.”
“I was recommended by my branding partner to use Chris and his team last year. I can’t recommend them highly enough. They are very responsive , easy to communicate with and great at translating techy stuff when needed. They are now an essential part of my business model”
“We've been working closely with Chris, Hannah and the team for over 12 months now. As a small company with ambitious growth plans, it was incredibly important that we made the right choice on who to trust with our SEO. 18 months later, we're totally thrilled with our experience so far and our growing online presence within our market. We'd happily recommend BTT and their services to other SMEs. Drop them an enquiry!”
Real people on your account, not a faceless agency.
You work directly with the senior people who do the work, a small Bristol team you will actually get to know, not a call centre or an account manager relaying messages.
Meet the team →
