Can AI crawlers read your site?
Enter a domain and this checks its live robots.txt against the fifteen crawlers that feed AI answers — using the same RFC 9309 evaluation the full audit runs. No account, no email.
Why this check matters more than it sounds
A robots.txt rule is the cheapest way to disappear from AI answers, and the easiest to write by accident. Fifteen crawlers now fetch pages so that an assistant can cite them. If one of them is disallowed, that engine has nothing to cite you from — and nothing on your site will tell you.
The usual cause is not a deliberate block. It is a rule written years ago to stop a scraper, which happens to match a bot that did not exist at the time.
- Answer crawlers fetch a page when someone asks a question, so an assistant can quote it. Blocking one removes you from that engine entirely.
- Search crawlers feed the index that powers Google, AI Overviews and Copilot. Blocking these is almost always fatal for organic visibility too.
- Training crawlers only opt your content out of model training. Blocking them is a legitimate choice and does not cost you citations.
The error nobody notices: a robots.txt that fails to load
Under RFC 9309, a crawler that receives a 5xx error or a network failure on robots.txt must assume the entire site is disallowed. A 4xx, by contrast, means there are no rules and everything is allowed.
That asymmetry is why a robots.txt which renders through an application — one that times out, or returns 500 when a header is missing — can silently remove your entire site from every compliant crawler. The page itself is fine; the gate is broken.
This tool reports the status it actually received, because the difference between "no rules" and "could not read the rules" is the difference between being visible and being invisible.
How to read the results
Each row shows the bot, who runs it, what role it plays, and which rule decided the outcome. The rule matters: if the decision came from a wildcard rather than a named user-agent, that is the signature of a block written for a different purpose now catching AI crawlers by accident.
Training crawlers are listed alongside the others for completeness, not because you should allow them. Blocking GPTBot is a legitimate opt-out and has no effect on whether ChatGPT cites you, since that goes through its search crawler instead.
This is one part of the full audit
Crawler access is the cheapest thing to fix and the most common silent killer. The full scan also tests whether AI engines actually name your brand when a buyer asks.
Questions people ask after using this
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot is a training crawler: blocking it opts your content out of model training and has no effect on whether ChatGPT cites your page. Citations come through OAI-SearchBot and ChatGPT-User instead, which are separate user agents. Blocking those does remove you from answers.
My robots.txt returns a 500. Does that matter?
It matters more than almost any other config error. RFC 9309 requires a crawler that receives a 5xx or a network error on robots.txt to assume the entire site is disallowed. Every compliant crawler treats your site as off-limits, and the pages themselves look perfectly fine to you.
What is the difference between a 404 and a 500 on robots.txt?
A 404 means "no rules exist", so everything is allowed. A 500 means "I could not tell you the rules", which crawlers must treat as a full block. That asymmetry is why a robots.txt which renders through your application — and times out under load — can silently remove the whole site.
Should I add AI crawlers to robots.txt at all?
You already affect them whether or not you name them: a wildcard rule applies to every crawler. Naming them explicitly is useful when you want a deliberate exception — allowing answer crawlers while blocking training crawlers, for example.
Does robots.txt control whether Google AI Overviews can use my page?
Not directly. AI Overviews draw on the Google index, so Googlebot must be allowed. Blocking Google-Extended only opts out of Gemini training and grounding; it does not stop AI Overviews. Blocking Googlebot is what removes you from both.
Why does the tool show training crawlers I do not need to allow?
For completeness, so the picture is not misleading. A list that silently omitted them would leave you unaware that a wildcard rule was also blocking them, and you would be unable to tell whether a block was deliberate.
Also worth checking
Google, Bing & Copilot readiness
Whether the index can rank the page at all: crawl access, indexability, measurable page experience, structured data and the Microsoft controls that block Copilot.
llms.txt generator
Build a real llms.txt from your own pages, with a note on each one.
AI visibility scan
Whether AI engines name your brand when a buyer asks a question in your category.