Crawl Stats, explained properly.
What are crawl stats?
Crawl stats are the data that show how search engine bots such as Googlebot crawl and index your website. They reveal how often the search engine crawls your site, how many pages it indexes, and whether there are crawl errors or issues. Monitoring them helps you understand how effectively bots are crawling your site and whether all your important content is indexed and available in search. Regularly checking crawl stats is a key part of any SEO strategy, because it helps you spot and fix issues that could stop search engines fully indexing a site.
Why crawl stats matter for SEO
- **Identify crawl issues:** they highlight problems or errors with crawlability that need resolving so all pages get indexed. - **Improve indexing efficiency:** optimising crawl stats helps search engines crawl and index content efficiently, which can improve rankings. - **Understand search engine behaviour:** they show how often engines visit, which pages get crawled, and how much data is processed each crawl. - **Monitor site health:** tracking crawl stats over time keeps an eye on whether search engines can access your content without issues.
How to monitor crawl stats
### 1. Google Search Console Search Console shows how often Googlebot visits, which pages it crawls, and any errors, including server response codes like 404 (page not found) and 5xx server errors that point to broken links or missing pages. The Search Console help documentation is the official reference for what each report covers. ### 2. Crawl Stats report Inside Search Console, the Crawl Stats report shows how many requests Googlebot made in the last 90 days, how many succeeded, and how many failed. It helps you understand how well Googlebot can crawl your site. ### 3. Server logs Server logs give more detail than Search Console alone: how often bots visit and which pages they hit most frequently.
Common crawl issues and how to fix them
### 1. Crawl errors When Googlebot hits errors, it may not index all your pages. The most common are 404 (page not found) and 500 (server error). Fix them by redirecting broken URLs or resolving server problems. The IANA HTTP status codes registry lists every response code. ### 2. Blocked pages If pages are blocked by robots.txt or meta tags, Googlebot cannot access them. Make sure important pages are not blocked and adjust your robots.txt and meta tags accordingly. ### 3. Slow load times A slow site can cause timeouts or delays during crawling, which affects how often Googlebot returns. Optimise speed so bots can crawl every page efficiently. ### 4. Duplicate content Duplicate content confuses search engines about which version to index. Canonicalisation tells them which page is primary and avoids the issue. ### 5. Redirect chains Redirect chains, where one page redirects to another and then another, slow crawling and can block access to content. Monitor and fix them so Googlebot can crawl efficiently.
How crawl stats impact SEO
Crawl stats affect SEO indirectly by determining how well search engines can access and index your site. Better crawl stats mean more effective indexing, which can lead to higher rankings: - **Improved indexing:** the better engines crawl and index your site, the more likely your content shows in results. - **Faster content discovery:** more frequent crawling means new content is found and indexed sooner. - **Better crawl efficiency:** fixing errors and easing navigation improves efficiency and overall performance.
Crawl Stats: common questions.
Where do I find crawl stats in Search Console?
In Google Search Console, go to Settings (the cog icon at the bottom of the left navigation). Crawl Stats is one of the options. The full report is also at search.google.com/search-console/settings/crawl-stats. The data goes back about 90 days.
What should I look for in crawl stats?
Three things matter most. Average response time should be under about 1 second; rising response times indicate server performance problems. Total crawl requests trending consistently up or down suggests Google is finding more or fewer pages of interest. The breakdown by file type, response code and crawl purpose flags specific issues like excessive 404s or unexpected JavaScript crawl spikes.
Is high crawl frequency good or bad?
Generally good, with caveats. High crawl frequency means Google considers your site important and worth checking often. The exception: if Google is hammering low-value URLs (faceted filter combinations, tracking parameters), high crawl frequency wastes budget that should be spent on important pages.
What is crawl budget?
The number of pages Google is willing to crawl on your site within a given period. It is determined by site size, server capacity and how important Google considers your site. For sites under about 10,000 URLs, crawl budget is rarely the constraint. For larger sites, canonicalising or noindexing low-value URLs can be high-leverage.
“I've had the pleasure of working with Chris on several SEO projects for my websites in the health field. He was definitely been able to demystify things so that I felt able to make decisions with him and feel confident they were the right ones. For a non-techie, that is worth gold! I look forward to working with Chris again and warmly recommend him to anyone who - like me - is unsure about all things SEO.”
“I was recommended by my branding partner to use Chris and his team last year. I can’t recommend them highly enough. They are very responsive , easy to communicate with and great at translating techy stuff when needed. They are now an essential part of my business model”
“We've been working closely with Chris, Hannah and the team for over 12 months now. As a small company with ambitious growth plans, it was incredibly important that we made the right choice on who to trust with our SEO. 18 months later, we're totally thrilled with our experience so far and our growing online presence within our market. We'd happily recommend BTT and their services to other SMEs. Drop them an enquiry!”
You work directly with the senior people who do the work, a small Bristol team you will actually get to know, not a call centre or an account manager relaying messages.
Meet the team →
