Crawl Budget, explained properly.
What is crawl budget?
Crawl budget is the number of pages a search engine, typically Google, is willing and able to crawl on your website within a set period. It combines two ideas: crawl rate limit and crawl demand. If your site exceeds this budget, some pages may not get crawled or indexed as often, or at all. Google defines it as the number of URLs Googlebot can and wants to crawl. For small sites it is rarely a concern. For larger sites, especially ecommerce stores, blogs and marketplaces, inefficient use of crawl budget leads to wasted crawling, missed indexing and stagnant rankings.
Why crawl budget matters
Large sites often struggle with slow indexing, stale content in the results and unindexed products. Nine times out of ten, crawl budget is being wasted on: - URL parameter variations from filters and sorts - Out-of-stock or paginated content - Duplicate or low-quality pages - Broken links or redirect loops This is why technical SEO is critical. If you want your most important pages crawled frequently and indexed correctly, you need to optimise your crawl budget.
How Google determines crawl budget
### 1. Crawl rate limit This controls how many requests Googlebot can make without overwhelming your server. If your site slows down or returns errors, Google may crawl less aggressively. ### 2. Crawl demand This is based on how important Google thinks your pages are, which depends on popularity, freshness, internal linking and how often content updates. So crawl budget is not fixed; it fluctuates with your site's health and SEO signals.
Signs you are wasting crawl budget
- Pages take weeks or months to appear in search - Google Search Console shows "Discovered, currently not indexed" - Server logs show frequent crawling of unimportant URLs - You have thousands of pages but few are ranking If that sounds familiar, a full SEO audit with log file analysis can diagnose the issue.
How to improve crawl budget
### 1. Clean up low-value URLs Use robots.txt to block crawler access to unimportant sections like admin pages, tag archives or certain filters. Use canonical tags or noindex where appropriate. ### 2. Fix redirect chains Redirect chains slow crawling and waste resources. Fixing them improves efficiency and page authority flow. ### 3. Improve internal linking Pages that are not linked internally are hard for bots to find. An internal link map prioritises important pages and ensures none are orphaned. ### 4. Consolidate thin or duplicate content Trim or merge underperforming pages, and canonicalise duplicates to a single, stronger URL. ### 5. Improve site speed and server health If Googlebot detects server errors or sluggish responses, it slows its crawl. Solid hosting, CDN and caching keep it moving. ### 6. Use XML sitemaps effectively Submit sitemaps that include only your index-worthy URLs. Every page thrown in just clutters the crawl.
Tools to monitor and optimise crawl budget
- **Google Search Console:** check crawl stats, coverage and indexing status. - **Screaming Frog:** identify duplicate pages, thin content and redirect chains. - **LogFlare or server logs:** see which URLs bots are hitting. - **Sitebulb:** visualise crawl paths and prioritise pages.
Crawl budget and site size
- **Under 1,000 URLs:** crawl budget is rarely an issue. - **1,000 to 10,000 URLs:** optimisation starts to matter, especially on slower servers. - **10,000+ URLs:** you need a crawl budget strategy. If you run a Shopify or WooCommerce site with faceted filters, the number climbs fast.
What crawl budget does not affect
- **How well your page ranks:** that is a content and backlink game. - **Whether Google discovers your site:** that is more about links and sitemaps. - **How often pages are refreshed:** unless they are low priority or buried. It only governs how many pages Googlebot tries to visit in a timeframe. But if your best pages are not crawled, that is a real problem.
Crawl Budget: common questions.
Does crawl budget matter for small sites?
Rarely. Sites under 10,000 URLs typically get all their pages crawled regularly without crawl budget concerns. Crawl budget becomes a real constraint at the 100,000+ URL scale, especially for ecommerce sites with extensive faceted navigation.
How do I improve crawl budget efficiency?
Three approaches. Block low-value URLs in robots.txt or via noindex (filter combinations, internal search results, admin pages). Fix server response times, since Google reduces crawl rate when servers are slow. Improve internal linking so important pages are easy to discover; orphaned pages waste crawl signal.
Where do I see crawl budget data?
Google Search Console's Crawl Stats report (Settings, then Crawl Stats). It shows total crawl requests, average response time, and a breakdown by file type and response code. Spikes or drops reveal issues; steady patterns show healthy crawling.
Can I increase my crawl budget?
Indirectly. Improve site quality and authority, because Google crawls sites it considers important more frequently. Improve server performance, so faster responses let Google crawl more in the same window. Add new content regularly, which signals an active site worth checking.
“I've had the pleasure of working with Chris on several SEO projects for my websites in the health field. He was definitely been able to demystify things so that I felt able to make decisions with him and feel confident they were the right ones. For a non-techie, that is worth gold! I look forward to working with Chris again and warmly recommend him to anyone who - like me - is unsure about all things SEO.”
“I was recommended by my branding partner to use Chris and his team last year. I can’t recommend them highly enough. They are very responsive , easy to communicate with and great at translating techy stuff when needed. They are now an essential part of my business model”
“We've been working closely with Chris, Hannah and the team for over 12 months now. As a small company with ambitious growth plans, it was incredibly important that we made the right choice on who to trust with our SEO. 18 months later, we're totally thrilled with our experience so far and our growing online presence within our market. We'd happily recommend BTT and their services to other SMEs. Drop them an enquiry!”
You work directly with the senior people who do the work, a small Bristol team you will actually get to know, not a call centre or an account manager relaying messages.
Meet the team →
