AI crawler access
Most sites that cannot be cited are not badly written. They are blocked, usually by a robots.txt rule nobody remembers adding.
The fifteen crawlers we check
Each one is checked separately, because blocking them has different consequences. Blocking OAI-SearchBot removes you from ChatGPT search results. Blocking GPTBot only opts you out of training.
- OAI-SearchBot, GPTBot and ChatGPT-User for OpenAI
- PerplexityBot and Perplexity-User for Perplexity
- Claude-SearchBot, ClaudeBot and Claude-User for Anthropic
- Googlebot, Google-Extended and Bingbot
- Meta-ExternalAgent, Applebot, Applebot-Extended and CCBot
Severity reflects consequence, not tidiness
Blocking OAI-SearchBot, PerplexityBot, Bingbot or Googlebot is a block level finding, because it removes you from an answer surface entirely. Blocking GPTBot or Google-Extended is informational, because it is a legitimate choice about training data and plenty of companies make it deliberately.
Page level snippet controls too
A site can be fully crawlable and still be uncitable. We separately check for nosnippet, max-snippet caps and data-nosnippet sections, which quietly stop an engine quoting the page even when it can read it.
Questions
Is blocking GPTBot bad?
Not necessarily. GPTBot is training data. If you do not want your content training a model, blocking it is a reasonable decision, which is why we report it as informational and not as a failure. OAI-SearchBot is the one that affects whether you appear in answers.
Do you need access to my server?
No. robots.txt is public. The check runs against the same file any crawler reads.
More in Can AI engines read you?
Rendering gate
We fetch your page as raw HTML and again in a real browser, then compare. Content that exists only after JavaScript runs is content most AI crawlers never see.
Read moreStructured data values
Most schema validators tell you the markup parses. We check whether the fields inside it actually contain anything an answer engine can quote.
Read moreCitation drivers
Three things make a page quotable to an answer engine: a concrete number, a date it can trust, and language that commits. We check all three.
Read moreEntity consistency
Answer engines need to know the company on your site is the one on your Google profile. We check whether that identity is machine readable and consistent.
Read more
Run it against your own site.
No signup, no card. It reports what it could not check as well as what it found.