robots.txt
One wrong line here removes the whole site from search.
What this check looks at
robots.txt is one of the Technical SEO checks. Whether engines can crawl and index the site at all. It is judged from what your site served at the time of the crawl, and a page we could not reach is reported as unreachable rather than skipped.
How this one runs
This check answers entirely from a crawl of your own site. We fetch the page the way a search crawler would, parse what came back, and test the specific property this check is responsible for. There is no account to connect, no data set to license, and nothing to install, so it runs on the first audit you ever request and on every one after it.
The result is derived from what your site actually served us at that moment, not from a cached third party index that may be weeks stale. That means it reflects your site as it is today, and it also means a page behind a login, a firewall, or a robots rule we respect is a page we will report as unreachable rather than quietly skip.
What you get from it
- Runs on every audit, on every page we are allowed to reach
- No Google connection, no vendor subscription, no tracking script on your site
- Re-runs on a schedule, so a regression shows up as a change rather than as a surprise
What it reports when it fails
These are the findings this check can raise, taken from the code that raises them rather than rewritten for the website. An ellipsis stands where the report fills in what it measured on your page.
- fix now/robots.txt missing or returns …
- fix nowrobots.txt disallows everything, the whole site is closed to crawlers
- worth fixingrobots.txt does not declare the sitemap
- worth fixing… is blocked outright by robots.txt, this site will not appear in
- worth fixing… is not named in robots.txt, AI crawler access is implicit, not explicit
Where it sits in the audit
See every technical check or the full catalogue of 77 checks. The documentation covers the order checks run in and how a finding is fingerprinted, and the about page explains why we publish the checks that do not run.
See this check run on your site.
No signup. The report names what it could not check as clearly as what it found.