Why AI crawlers matter
ChatGPT, Claude and Perplexity answer questions by reading the web. Each relies on its own crawler — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity. If your robots.txt blocks them, your pages can't be read, and an assistant can't cite or recommend you.
Many sites block them without knowing: a security plugin, a CDN setting or a copied robots.txt with a broad Disallow rule is enough.
Search bots vs. training bots
Not every AI bot does the same job. Search and answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch pages to answer a user right now: blocking them makes you invisible in AI answers. Training bots (Google-Extended, CCBot, Applebot-Extended, Bytespider) collect content to train future models: blocking them is a legitimate choice that doesn't hide you from today's answers.
This checker flags the first group as critical and the second as informational, so you can decide with the facts in hand.
How to allow AI crawlers in robots.txt
Add an explicit group per bot at the root of your site (https://your-site.com/robots.txt). An explicit Allow wins over a wildcard rule for that bot:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /Frequently asked questions
Does allowing GPTBot mean ChatGPT will cite me?
No, it's a prerequisite, not a guarantee. A blocked site can't be read, so it can't be cited. Once the bots are allowed, citation depends on your content and authority — Beakon's citation tests measure exactly that.
My site has no robots.txt. Is that a problem?
No: without a robots.txt every crawler is allowed by default. Having one is still recommended, to point to your sitemap and keep control.
Can a bot be blocked outside robots.txt?
Yes. A firewall, a CDN bot-protection rule or a server rule can block AI crawlers even when robots.txt allows them. This tool checks robots.txt; a full Beakon audit goes further.