How to refuse AI training without disappearing from AI answers
Published
To keep their content out of AI training, many sites block AI crawlers in robots.txt. Most do not realise they also vanish from the answers of ChatGPT, Claude and Perplexity, where more and more buyers now look.
What changed
On October 30, 2026, OpenAI announced that it will honour a shared robots.txt signal for AI training preferences, backed with Google, Microsoft and Anthropic. It comes from an IETF draft (aipref): a site states a preference for a use, training, instead of naming crawlers one by one.
Support is rolling out over the coming months, provider by provider.
The problem with blocking crawlers
Until now, the only lever was to block named crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot and the rest. The list is long, it changes, and some crawlers serve both training and browsing.
Many sites settled it with one blanket rule against every AI bot. That rule also shuts out the crawlers that fetch pages to answer users' questions, so the site is no longer read, and no longer cited, when someone asks an assistant for a recommendation.
The impact
A prospect who asks ChatGPT or Perplexity "which tool for…" or "which agency in…" gets your competitors' names, not yours. Your search rankings do not move, so nothing in your usual reports shows the loss.
The block does not even guarantee what it was meant to: your content can still reach training sets through copies and third-party datasets. And sites with no rule at all give their consent by default.
Who should check: publishers, e-commerce sites, SaaS documentation, and anyone who added AI rules to robots.txt in 2023 or 2024.