Should I block AI crawlers like GPTBot?

Each major AI company publishes its crawlers and what they do. The distinctions that matter most:

• OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search results) and ChatGPT-User (fetches pages when a user asks). OpenAI says each setting is independent of the others.
• Google: Google-Extended controls whether content is used to train Gemini models and for grounding. Google says it doesn't affect inclusion or ranking in Google Search, which includes AI Overviews and AI Mode.
• Perplexity: PerplexityBot surfaces sites in Perplexity's results; Perplexity-User fetches pages for a user's question and generally ignores robots.txt.
• Anthropic: ClaudeBot and Anthropic's other bots honour robots.txt directives.

A common setup for brands that want AI visibility but not training use is to disallow GPTBot and allow OAI-SearchBot, PerplexityBot and the other search-oriented agents. Publishers with licensing concerns may choose to block more.

Check the firewall too. robots.txt is only a request, and a CDN or bot-protection rule can return errors to AI crawlers even when robots.txt allows them. Our AI search readiness study found sites that allow AI crawlers in robots.txt but block them at the firewall without realising it.