BrisTechTonic is a Bristol SEO agency that helps UK businesses get found on Google, the local map pack and the AI assistants they now ask, like ChatGPT and Perplexity. Log file analysis for SEO means reading your server's raw activity records to see how search engine and AI crawlers actually behave on your site, rather than relying on what a scanning tool guesses from the outside. Every time a crawler visits, it leaves a line in the records: the page it asked for, which crawler it was, whether the page loaded or errored, the time, and often the size. Read properly, those lines show up time wasted crawling junk web addresses and dead redirects, important pages that are crawled rarely or never, and error and crawler-behaviour patterns no scanning tool can show. It is the only place you see what really happened rather than a guess, which is why it sits at the centre of serious technical SEO on large or complex sites.
Almost every technical SEO tool works by guessing. It starts at your homepage, follows the links it can see, loads what it can, and builds a picture of your site from the outside in. That picture is useful, but it is a guess about how a search engine might behave, not a record of how one did. Your server's activity records are that record. They are the one place where you are not guessing at Google's behaviour, you are reading it, visit by visit, exactly as your website saw it.
Read the full detail →
This matters most on the sites where technical SEO matters most: large online shops, publishers with deep archives, marketplaces, and any site where the number of pages runs well ahead of the number that actually need to rank. On a fifty-page brochure site, how much Google crawls is not a problem and the records tell you little you could not work out elsewhere. On a site with hundreds of thousands of pages, Google is constantly deciding what to crawl, how often, and what to quietly ignore, and those decisions are invisible until you open the records.
The money at stake is concrete. If Google is spending most of its visits re-crawling filtered category pages, expired listings and chains of redirects, it is not spending them finding your new products or re-checking the pages you have just improved. Content gets listed slowly or not at all. Fixes take weeks longer to register than they should. None of this shows up as a single alarm in a traffic report; it shows up as a site that feels sluggish to respond no matter how good the work upstream is.
There is a further reason the records have become essential. It is no longer only Google's and Bing's crawlers on your server. The crawlers behind ChatGPT, Claude, Perplexity and Google's AI (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and a growing list of others now visit your pages, and they behave differently from the traditional search crawlers. The records are where you see who is crawling, how hard, and whether your site is serving them or holding them back. If you care whether the assistants can reach your content, the records are where that question is answered.
If Google cannot crawl it, nothing else you do counts.
These are industry figures, not our own results, and we do not invent numbers. Technical work rarely shows on the page, but it is the ceiling everything else runs into. No outcome is ever guaranteed.
Log file analysis, broken down.
Crawl budget and frequency mapping →
We add up every crawler visit by part of the site, page type and page to show where Google's crawling genuinely goes. Which sections are crawled daily, which monthly, which never. The gap between where the crawling is spent and where your revenue sits is usually the whole story.
Status code and redirect diagnosis →
The records expose the not-found pages, server errors, fake not-founds and chains of redirects that crawlers hit again and again, not just the ones a scanning tool stumbles across. Persistent errors and long redirect chains waste crawling and eat away at trust in the pages affected, and we trace each back to its cause.
Orphan and rarely-crawled page discovery
By comparing the records against a full scan and your sitemaps, we find pages that exist and should rank but are crawled rarely or never, and pages Google visits that you did not know were exposed. Both distort how importance flows through the site.
Crawl versus render reality →
The records show whether crawlers are fetching your scripts and behind-the-scenes data at all, which is the first question behind any problem with pages not showing up. Where the two overlap, we hand off to a deeper look at how pages are built rather than guessing from the outside.
Bot verification and fake-crawler filtering
A meaningful share of traffic claiming to be Google's crawler is not. We confirm which crawlers are genuine by checking where they really come from, so the analysis rests on real search and AI crawlers, and so you can see where scrapers are pretending to be them and eating your resources.
AI and LLM crawler behaviour
We separate out the AI crawlers, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, so you can see which assistants are reaching your content, how often, and whether your site's rules or limits are quietly shutting them out.
Some of our recent work.
Book a free discovery call before you spend a thing
What clients say about working with us.
“I've had the pleasure of working with Chris on several SEO projects for my websites in the health field. He was definitely been able to demystify things so that I felt able to make decisions with him and feel confident they were the right ones. For a non-techie, that is worth gold! I look forward to working with Chris again and warmly recommend him to anyone who - like me - is unsure about all things SEO.”
“I was recommended by my branding partner to use Chris and his team last year. I can’t recommend them highly enough. They are very responsive , easy to communicate with and great at translating techy stuff when needed. They are now an essential part of my business model”
“We've been working closely with Chris, Hannah and the team for over 12 months now. As a small company with ambitious growth plans, it was incredibly important that we made the right choice on who to trust with our SEO. 18 months later, we're totally thrilled with our experience so far and our growing online presence within our market. We'd happily recommend BTT and their services to other SMEs. Drop them an enquiry!”
Real people on your account, not a faceless agency.
You work directly with the senior people who do the work, a small Bristol team you will actually get to know, not a call centre or an account manager relaying messages.
Meet the team →
The way we run Log file analysis.
Get the logs, in full
We work with you or your hosting team to pull the raw activity records covering a representative stretch of time, ideally several weeks and across busy periods, from every layer that serves requests: the main server, the content delivery network and any load balancers. Partial records give a partial picture, so we are particular about getting them complete up front.
Parse, verify and normalise
We turn the raw lines into a clean set of data, confirm the crawlers claiming to be search engines are genuine by where they come from to strip out the fakes, and tidy up the web addresses so different versions group together sensibly. This is the unglamorous step that decides whether every later number can be trusted.
Cross-reference against crawl and Search Console
The records on their own tell you what was crawled; joined to a full scan, your sitemaps and Google Search Console data, they tell you what it means. This is where wasted crawling, orphan pages and pages missing from Google become visible as specific, named pages.
Diagnose and prioritise
An experienced person reads the patterns, separates the findings that affect traffic and revenue from the long list of curiosities, and traces each real issue to its cause. You get an ordered plan, not a raw dump of everything a crawler ever touched.
Fix at source
We make the changes or write them up clearly for your developers: search-engine rules, tidying up redirects, internal-linking changes and handling of web-address variations, all applied in the templates and settings so the fix holds across the whole site.
Re-pull and confirm
After the fixes land we take a fresh sample of the records and check that crawler behaviour has actually changed: crawling shifting towards the pages that matter, error patterns clearing, important pages crawled more readily. We confirm it against what really happened rather than assuming.
How do we approach Log file analysis?
Ground truth, not simulation
A scanning tool like Screaming Frog guesses how a crawler might move through your site. The activity records show what Google's and the AI crawlers actually did. When the two disagree, and on real sites they routinely do, the records win. We start from what really happened, then use the scan to explain it.
Crawl budget is finite, so we spend it deliberately
On a large site Google will not crawl every page as often as you would like. Every visit spent on a filtered duplicate, a tracking web address or a chain of redirects is a visit not spent on a page that makes money. We find where that crawling leaks away and close the leaks at the source.
Experienced reading, not an automated export
Chewing through a million lines of records is easy. Knowing which patterns matter is not. An experienced person reads the totals and the oddities, weighs them against the revenue behind each part of the site, and tells you what is worth acting on and what is safe to leave.
Real people doing the work, and putting their name to it.
You deal with our small, experienced Bristol team directly, from the first call to every monthly report.
Meet the team →What Log file analysis puts on your desk.
Are we the right fit for you?
Thinking of switching from another agency?
Moving your Log file analysis to us is more straightforward than staying somewhere that is not working. You keep everything you already own, we pick up without a gap, and we can run the whole thing for you or work alongside your team and your developers, whatever suits.
A free look first
We review what your current agency has done, tell you straight what is worth keeping and where the quick wins are. No obligation to move.
You keep what's yours
Your site, content, Google Business Profile and analytics stay in your name. We work with what you already have; nothing is held hostage.
A clean handover
We map the work across, set a plain-English baseline in the first few weeks, and carry on from there. No starting from zero, no quiet ranking gap.
Three-month minimum, then rolling monthly on 30 days’ notice. No long lock-in, no exit fees.
Why log analysis is expert work?
Anyone can load an activity file into a tool and produce charts. The value is in reading them: knowing that a burst of one kind of response from Google on one page type is fine while the same pattern on another is a problem, or that a section crawled hard but converting nothing is wasted crawling worth stopping. That judgement comes from having done this on real sites for years, not from a tool's defaults. Our experienced team has deep technical SEO experience, and the person who reads your records is the one who plans and writes up the fix. Nothing is handed down a chain, so the reasoning behind a finding is never lost before it is acted on.
We read the record ourselves
Genuine crawler data, error patterns and how often pages are crawled are read by an experienced person, because the findings that matter rarely sit at the top of an automatic severity list.
Joined to the wider picture
The records are read alongside a full scan, your sitemaps and Google Search Console, so a finding is a named page with a cause and a cost, not a lone number on a chart.
Honest about scale
On a small site we will tell you the records add little and point you at cheaper wins. We recommend this work where how much Google crawls is genuinely a limit, not as a default line item.
How much does Log file analysis cost?
All + VAT · 3-month minimum, then 30-day rolling| What's included | Lite £299 + VAT / month | Max Most chosen £449 + VAT / month | XL £849 + VAT / month | Custom Contact us |
|---|---|---|---|---|
| Technical & on-page SEO | ✓ | ✓ | ✓ | ✓ |
| AEO / AI-search layer (found in ChatGPT, AI Overviews) | ✓ | ✓ | ✓ | ✓ |
| Local map-pack optimisation | ✓ | ✓ | ✓ | ✓ |
| Page creation | ✓ | ✓ | ✓ | ✓ |
| Blog posts / month | Add-on | Add-on | 2 | Tailored |
| AI Authority building | — | — | ✓ | ✓ |
| Plain-English monthly report | ✓ | ✓ | ✓ | ✓ |
| Best for | Getting the foundations right | Competing on your key terms | Going after it hard, incl. AI authority | Shaped around your goals |
Log file analysis: common questions.
What is log file analysis in SEO, and why does it matter?
It is the work of reading your server's raw activity records to see how search engine and AI crawlers actually behave on your site: which pages they ask for, how often, and whether those pages load or error. It matters because the records are the only data that show real crawler behaviour rather than guessing at it. On large or complex sites, that record shows up wasted crawling, rarely-crawled pages and error patterns that no scanning tool can see.
How is log file analysis different from a crawling tool like Screaming Frog?
A scanning tool starts at your homepage and follows links to guess how a crawler might move through your site. It is a simulation. Log file analysis reads what Google's and the AI crawlers genuinely did, visit by visit, as your server recorded it. The two work together: we compare them, and where a scan and the real records disagree, the records are what actually happened.
What server logs do you need, and how do I get them to you?
We need the raw activity records covering a representative stretch of time, ideally several weeks, from every layer that serves your site: the main server, the content delivery network, and any load balancers in front of them. Most hosting platforms can export these, and we will give your team precise instructions for the setup you run. Getting them complete matters more than anything here, because partial records lead to partial conclusions.
How does log file analysis help with crawl budget?
By showing exactly where crawler visits go. When we add up crawler activity by part of the site and page type, it usually becomes obvious that a large share of the crawling is spent on filtered duplicates, tracking web addresses, expired pages or chains of redirects, rather than on the pages that earn enquiries. We close those leaks at the source so more of Google's limited crawling lands on the pages you actually want crawled and listed.
Does log file analysis help with AI crawlers and assistants, not just Google?
Yes, and increasingly it is one of the clearest reasons to do it. The crawlers behind ChatGPT, Claude, Perplexity and Google's AI (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and others now visit your pages, and they behave differently from the traditional search crawlers. The records are where you see which assistants are reaching your content, how hard, and whether your site's rules or limits are quietly blocking them. If AI visibility matters to you, the records answer questions nothing else can.
How big does my site need to be for log file analysis to be worthwhile?
It earns its place when the number of pages runs well ahead of the number that need to rank: large online shops, publishers with deep archives, marketplaces, and anything with lots of filters or numbered pages. On a small brochure site, how much Google crawls is not a real limit and the records tell you little you could not learn more cheaply, and we will say so rather than sell you the work anyway.
Will log file analysis on its own improve my rankings?
It removes a specific kind of blocker: crawling wasted on the wrong pages, important pages crawled too rarely, and errors that eat away at trust in a section. On a site where crawling is limited, fixing those can help content get found and re-checked faster. But it is foundation work that lets your content and reputation count properly, not a substitute for them, and we will not promise rankings that depend on things outside the technical work.
What do you actually deliver from a log file analysis engagement?
A report on where Google's crawling goes, a breakdown of errors and redirects traced to cause, a list of orphan and rarely-crawled pages, a checked summary of genuine crawler activity with anything holding them back flagged, and a prioritised list of fixes tying each finding to a specific change. After the fixes land, we take a fresh sample of the records and confirm crawler behaviour actually moved the way it was meant to.
Book the call, or see the evidence first