Search Overflow

Can AI engines read your website?

2 min read

How to check whether ChatGPT, Perplexity and Google AI Overviews can actually reach and parse your pages, and what silently blocks them.

Most sites that are invisible in AI answers are not invisible because their content is weak. They are invisible because the engine never read the page.

The three ways a page becomes unreadable

An answer engine has to do three things before your content can be cited. It has to be allowed in, it has to see content in the HTML it receives, and it has to find structured data that carries real values. A failure at any one of those makes everything downstream irrelevant.

1. The crawler is blocked

AI crawlers are separate user agents from Googlebot. Blocking or forgetting them is common, and it is usually accidental. A robots.txt written in 2021 cannot allow a crawler that did not exist then.

The retrieval crawlers that matter for citation are not the same as the training crawlers. OAI-SearchBot fetches pages to answer a live question in ChatGPT. GPTBot gathers training data. Blocking GPTBot has no effect on whether you get cited today. Blocking OAI-SearchBot removes you from the answer entirely.

2. The content only exists after JavaScript runs

If your page ships an empty div and fills it on the client, a crawler that does not execute JavaScript receives nothing. Some engines render, many do not, and the ones that do often apply a budget you cannot see.

The test is simple. Fetch your own page with JavaScript disabled and read what comes back. If the answer you want quoted is not in that HTML, it does not exist as far as a non-rendering crawler is concerned.

3. The schema is present but empty

Structured data that declares a Product with no price, or a Review with no rating, is markup without meaning. It validates. It passes most testing tools. It tells an answer engine nothing it can quote.

Why the order matters

These three checks are a gate, not a checklist. There is no value in optimising your heading structure for an engine that is blocked at robots.txt. Search Overflow runs them in that order deliberately, and suppresses lower priority findings on a page that fails an earlier gate, so the one fix that matters is not buried under thirty cosmetic suggestions.

What to do first

Check robots.txt for the retrieval crawlers by name. Fetch a page with JavaScript off. Look at whether your structured data carries values or just tags. Those three answers tell you more than any content audit will.