There are two kinds of AI crawler and almost every robots.txt guide treats them as one.
One kind fetches your page at the moment somebody asks a question, so that the assistant can quote you in its answer. The other kind collects text for a future training run. Blocking the first removes you from the answer. Blocking the second changes nothing about whether you get cited, ever.
Most advice tells you to block both, or neither, and counts the total as if the number meant something.
The four that decide whether you get cited
These fetch pages to answer live questions:
- OAI-SearchBot — what indexes you for ChatGPT search
- PerplexityBot — Perplexity's retrieval crawler
- Googlebot — the same crawler as always, now also feeding AI Overviews
- Bingbot — feeds Copilot
Block any of these and you are not ranked lower in that assistant's answer. You are absent from it. There is no partial credit, because the engine cannot cite a page it could not fetch.
Googlebot is the one that catches people out. It is not an AI crawler in the sense anyone means when they say "should I block AI bots," and blocking it has never been a serious suggestion. But AI Overviews are built on the same index, so a site that has quietly disallowed some section from Googlebot has also removed that section from AI Overviews without ever deciding to.
The five that only take training data
- GPTBot — OpenAI's training crawler
- ClaudeBot — Anthropic's
- CCBot — Common Crawl
- Google-Extended — Google's training opt-out control
- Applebot-Extended — Apple's
Blocking every one of these is a legitimate position. If you do not want your writing in the next training run, disallow them and the models will not train on it.
What it will not do is affect whether an assistant cites you tomorrow. Those are different systems with different crawlers. A site that blocks all five and none of the first four remains fully citable.
Why the counting is the problem
Tools in this category like to report a number: you are blocking 23 AI crawlers. The number is impressive and close to meaningless, because it treats a training crawler and a retrieval crawler as the same unit.
Twenty-three blocked training crawlers cost you nothing. One blocked retrieval crawler costs you every citation from that assistant. A single total cannot express that, and a tool reporting the total is answering a question nobody should be asking.
The useful question is not how many you block. It is which.
How this happens by accident
Nobody sits down and decides to disallow OAI-SearchBot. It happens three ways.
A blanket User-agent: * / Disallow: / on a staging rule that outlived the staging site. A plugin that added an "AI blocker" list in an update, bundling retrieval and training crawlers together because its author did not distinguish them either. Or a robots.txt that names a crawler nobody at the company remembers adding, in a line dated before anyone currently working there arrived.
The common factor is that none of it produces an error. Your site stays up, your Google rankings hold, and the only symptom is an absence — you are not in answers you would otherwise appear in, and nothing tells you.
Check it yourself
Open your own robots.txt and look for those four names, plus any blanket disallow that would catch them. That is the whole check, and it takes a minute.
If you would rather have it read for you, the AI crawler robots checker reads your live robots.txt the way each crawler reads it and reports them separately, because blocking each one has a different consequence. No account, and nothing is stored.
The distinction is the entire point. A number that merges the two tells you how cautious you have been. It does not tell you whether you can be quoted.