AI 爬虫能读到你的网站吗？
Enter a domain and this checks its live robots.txt against the fifteen crawlers that feed AI answers — using the same RFC 9309 evaluation the full audit runs. No account, no email.
Why this check matters more than it sounds
A robots.txt rule is the cheapest way to disappear from AI answers, and the easiest to write by accident. Fifteen crawlers now fetch pages so that an assistant can cite them. If one of them is disallowed, that engine has nothing to cite you from — and nothing on your site will tell you.
The usual cause is not a deliberate block. It is a rule written years ago to stop a scraper, which happens to match a bot that did not exist at the time.
- Answer crawlers fetch a page when someone asks a question, so an assistant can quote it. Blocking one removes you from that engine entirely.
- Search crawlers feed the index that powers Google, AI Overviews and Copilot. Blocking these is almost always fatal for organic visibility too.
- Training crawlers only opt your content out of model training. Blocking them is a legitimate choice and does not cost you citations.
The error nobody notices: a robots.txt that fails to load
Under RFC 9309, a crawler that receives a 5xx error or a network failure on robots.txt must assume the entire site is disallowed. A 4xx, by contrast, means there are no rules and everything is allowed.
That asymmetry is why a robots.txt which renders through an application — one that times out, or returns 500 when a header is missing — can silently remove your entire site from every compliant crawler. The page itself is fine; the gate is broken.
This tool reports the status it actually received, because the difference between "no rules" and "could not read the rules" is the difference between being visible and being invisible.
How to read the results
Each row shows the bot, who runs it, what role it plays, and which rule decided the outcome. The rule matters: if the decision came from a wildcard rather than a named user-agent, that is the signature of a block written for a different purpose now catching AI crawlers by accident.
Training crawlers are listed alongside the others for completeness, not because you should allow them. Blocking GPTBot is a legitimate opt-out and has no effect on whether ChatGPT cites you, since that goes through its search crawler instead.
这只是完整审计的一部分
爬虫访问权限是最便宜能修好、也最常见的隐形杀手。完整扫描还会测试：当买家提问时，AI 引擎到底会不会提到你的品牌。
Questions people ask after using this
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot is a training crawler: blocking it opts your content out of model training and has no effect on whether ChatGPT cites your page. Citations come through OAI-SearchBot and ChatGPT-User instead, which are separate user agents. Blocking those does remove you from answers.
My robots.txt returns a 500. Does that matter?
It matters more than almost any other config error. RFC 9309 requires a crawler that receives a 5xx or a network error on robots.txt to assume the entire site is disallowed. Every compliant crawler treats your site as off-limits, and the pages themselves look perfectly fine to you.
What is the difference between a 404 and a 500 on robots.txt?
A 404 means "no rules exist", so everything is allowed. A 500 means "I could not tell you the rules", which crawlers must treat as a full block. That asymmetry is why a robots.txt which renders through your application — and times out under load — can silently remove the whole site.
Should I add AI crawlers to robots.txt at all?
You already affect them whether or not you name them: a wildcard rule applies to every crawler. Naming them explicitly is useful when you want a deliberate exception — allowing answer crawlers while blocking training crawlers, for example.
Does robots.txt control whether Google AI Overviews can use my page?
Not directly. AI Overviews draw on the Google index, so Googlebot must be allowed. Blocking Google-Extended only opts out of Gemini training and grounding; it does not stop AI Overviews. Blocking Googlebot is what removes you from both.
Why does the tool show training crawlers I do not need to allow?
For completeness, so the picture is not misleading. A list that silently omitted them would leave you unaware that a wildcard rule was also blocking them, and you would be unable to tell whether a block was deliberate.
也值得看看
Google、Bing 与 Copilot 就绪度
索引能不能把页面排上去：抓取权限、可否收录、可测量的页面体验、结构化数据，以及会挡住 Copilot 的微软设置。
llms.txt 生成器
用你自己的页面生成一份真实的 llms.txt，每条都带说明。
AI 可见性扫描
当买家在你的品类里提问时，AI 引擎会不会提到你的品牌。