AI crawler robots checker
Blocking GPTBot is a choice. Blocking OAI-SearchBot removes you from ChatGPT search. Most people do not know which one they did.
Reads your site's crawler settings and your homepage. Stores nothing.
What this checks
- OAI-SearchBot, GPTBot and ChatGPT-User
- PerplexityBot and Perplexity-User
- Claude-SearchBot, ClaudeBot and Claude-User
- Googlebot, Google-Extended, Bingbot and six more
Each crawler is reported separately for a reason
Most robots checkers tell you whether you have a robots file and whether it parses. That answers almost nothing, because the consequence of blocking depends entirely on which agent you blocked. Blocking GPTBot opts you out of training data, which many companies choose deliberately. Blocking OAI-SearchBot removes you from ChatGPT search results, which almost nobody chooses deliberately.
The rules are matched the way a crawler matches them
User agent matching is case insensitive and the most specific matching group wins, so a wildcard disallow does not necessarily apply to a bot that has its own group. We evaluate each of the fifteen agents against your file the way that agent would, rather than searching the text for its name, because a file can mention a bot and still allow it.
What it does not do
It checks one page, not your whole site, and it does not store the result. For a crawl across every page we can reach, with findings you can dismiss and re-audit over time, that is the paid product. This tool is the same code, scoped to one URL.
Other free tools
Live check
AI visibility checker
Three things decide whether an answer engine can cite you. This checks all three against your live site.
Live check
Schema value validator
A Product schema with an empty price is valid markup and useless data. Validators pass it. Answer engines skip it.
Generator
llms.txt generator
llms.txt is a plain text summary of your site for language models. It takes a minute and nobody does it.
The same check, across every page.
The paid audit runs this check and the rest of the catalogue on every page we can reach, and names the checks it could not run.