This week we worked with a client whose website was completely invisible to ChatGPT and Perplexity, despite having strong rankings in classic Google search. The cause was not their content, their schema, or their authority signals. It was a single setting, switched on by default by their hosting infrastructure, that nobody had thought to check.
Their CDN had Cloudflare’s “AI Fight Mode” enabled, blocking GPTBot, ClaudeBot, PerplexityBot and Google-Extended at the edge. AI crawlers could not even read the homepage, let alone consider citing it. We turned the setting off, the dev agency confirmed the change, and within hours the AI tools could see the site.
That fix took about thirty minutes. The visibility loss it had caused was probably six to nine months long.
This is not an isolated case. Cloudflare ships AI Fight Mode on by default to a large percentage of their sites. WordPress security plugins occasionally add over-zealous AI bot blocks. Old robots.txt files from 2023 still contain blocks added when AI scraping was a hot topic and “lock the door against the bots” felt like the safe move.
If you want your business cited in AI search (and most do, by 2026), the first thing to check is whether you are accidentally blocking the bots you need to read your site. Here is the 30-minute audit, the distinction nobody talks about between training and reading, and how to fix it.
The distinction nobody talks about
There are two completely different reasons to block AI bots, and they get conflated all the time.
Reason 1: stop AI training on your content. If you have proprietary content (long-form research, paid courses, original photography, anything you sell), you may not want OpenAI or Anthropic to ingest it into the next training cycle of GPT-5 or Claude 5. That is a legitimate intellectual property concern. Blocking AI training crawlers is a defensible choice. OpenAI documents its bots in the GPTBot docs.
Reason 2: stop AI tools from reading your site at query time. When a user asks ChatGPT “what are the best vintage lighting shops in Bristol?”, ChatGPT runs a real-time web search, fetches a handful of pages, reads them, and synthesises an answer. If you block that, you are not protected from training, you are just invisible at the moment a real customer is asking about you. Anthropic’s Claude web search documentation describes the same retrieval pattern.
The two are different bots in most cases, and the meta tags that control them are now different too. The new draft standard splits them as ai-train=no (do not use my content for training) versus ai-input=yes (do read my content to answer real-time questions). Few sites use either yet, and most blanket-block both, which is the worst of both worlds for businesses that want to be cited.
For a typical UK SME wanting to be visible in AI search: you almost certainly want AI tools to read your site at query time. You may or may not want them to use it for training. The two settings should be separate, and right now most sites have them welded together.
The 30-minute audit
Three layers to check, in this order. Do all three.
Layer 1: robots.txt
Type your domain into a browser, followed by /robots.txt. So for us: https://bristechtonic.co.uk/robots.txt. You will see a plain text file. Read it.
The user agents to look for are:
GPTBot(OpenAI training crawler)ChatGPT-User(OpenAI live-search agent, used at query time)OAI-SearchBot(newer OpenAI search-specific crawler)ClaudeBot(Anthropic crawler)anthropic-ai(older Anthropic identifier, sometimes still in lists)PerplexityBot(Perplexity training/index crawler)Perplexity-User(Perplexity live agent at query time)Google-Extended(Google’s AI training crawler, separate from Googlebot)CCBot(Common Crawl, used by many AI training pipelines)
If any of those appear under Disallow: /, you are blocking them site-wide. Decide consciously whether that is what you want, then update accordingly. Most UK SMEs we audit have inherited a blanket block from an SEO plugin’s default and never thought about it. Google Search Central has the canonical guidance on user-agent directives if you want to read deeper.
The pragmatic recommendation: allow the live-query agents (ChatGPT-User, Perplexity-User, OAI-SearchBot) so AI tools can read your site at the moment a customer is asking. Make a separate, intentional decision on the training crawlers.
Layer 2: Cloudflare or your CDN
If your site sits behind Cloudflare (most do), log in and check two settings:
- Security, Bots, AI Bots: there is a managed list of AI bots that can be blocked en masse. Check whether it is on. Cloudflare ships this as opt-in, but many website setups (especially agency-built sites) have it enabled at handover. Switch it to “Allow” if your goal is AI visibility.
- “AI Fight Mode” or “Block AI scrapers”: a higher-level toggle that may exist in your dashboard. Same logic.
If you are on a different CDN (Sucuri, Akamai, Fastly), the equivalent settings are usually under “Bot management” or “WAF rules”. The bots to check for are the same list as Layer 1. This is part of a broader technical audit that should be done on every site annually.
Layer 3: meta tags and HTTP headers
Less common, but worth a check. View source on your homepage and search for any meta name="robots" tag with values like noai or noimageai, or any X-Robots-Tag headers (set in your server config or by a plugin) doing the same. These typically only get set deliberately, but they can sneak in via plugins. The noindex tag glossary entry covers related directives.
You can use a tool like Google’s Rich Results Test to fetch your page as Googlebot and see the rendered headers, or httpstatus.io to inspect the raw response.
llms.txt: the proactive complement
While you are in there, consider adding an llms.txt file to your site root. It is the emerging convention for telling AI tools how to interpret your site, similar in spirit to robots.txt but in plain English rather than directives. The format is still settling, but a basic version looks like:
# BrisTechTonic
> SEO consultancy based in Bristol, UK, specialising in AI search visibility,
> conversion-led SEO and small-business growth.
## Our services
- [SEO packages](https://bristechtonic.co.uk/services/seo-packages): three tiers...
- [SEO strategy](https://bristechtonic.co.uk/services/seo-strategy): one-off review...
## Our content
- [Blog](https://bristechtonic.co.uk/blog): regular posts on AI search...
- [Glossary](https://bristechtonic.co.uk/glossary): plain-English definitions...
The file lives at yourdomain.com/llms.txt. Not all AI tools read it yet, but a growing number do, and adding one costs nothing. It is also a useful exercise in itself: writing your llms.txt forces you to articulate, in plain language, exactly what you want AI tools to know about your business.
The rule of thumb
If your goal is AI search visibility (which it almost certainly is, given the way buyer journeys have shifted in 2026), the default position should be:
- Allow live-query AI agents at the
robots.txt, CDN, and meta-tag layers. ChatGPT-User, Perplexity-User, OAI-SearchBot. These are the bots that fetch your page when a customer is mid-conversation with the AI. - Make a deliberate choice on training crawlers (GPTBot, ClaudeBot, Google-Extended). Most UK SMEs are happy to be in the training set; some prefer not to be. Both are valid, but make it a decision, not a default.
- Add an
llms.txtif you have any structured services or content worth flagging. - Re-audit annually, because the bot landscape and the meta-tag standards change every six to twelve months. This sits alongside your crawlability and indexability checks.
For more on what to do once your site is reachable by AI tools, see our framework for getting found in AI search and the explainer on what AEO is and why it matters. If you are reading this because your analytics has gone strange, run our traffic drop diagnostic as well. The three posts work as a set: this one removes the blockers, the others build the visibility and read the numbers. The work is included in every SEO package we run.
If you want help
The audit above is something most marketing managers can do themselves with thirty minutes and a coffee. If you want a second pair of eyes (or you do not want to be the person who has to log into Cloudflare for the first time), get in touch and we will run it with you on a screen-share. The fix is rarely complicated. The visibility unlock can be enormous.
By Chris McDowell, founder of BrisTechTonic. We do the technical SEO conversations that nobody else seems to be having yet, including the boring ones about CDN settings. See the SEO packages or book a call.
