Definition
ClaudeBot is Anthropic's crawler for web content that may be used to train its AI models, separate from its Claude-SearchBot and Claude-User agents.
ClaudeBot explained
Anthropic, the company behind Claude, uses three bots. ClaudeBot collects web content that could contribute to model training. Claude-SearchBot crawls to improve the quality of search results for Claude users. Claude-User fetches pages when a person asks Claude a question that needs web access.
Anthropic says all three respect robots.txt, and it supports the non-standard Crawl-delay directive for site owners who want to slow requests down. It also publishes the IP addresses its crawlers use, which helps you check that traffic claiming to be ClaudeBot is genuine.
As with OpenAI's bots, the decisions are separate. Blocking ClaudeBot affects training only. If you want Claude to find and reference your pages when people search with it, Claude-SearchBot and Claude-User need access. Review each group in robots.txt rather than blocking every Anthropic agent with one rule.
Example
Your robots.txt blocks ClaudeBot for licensing reasons. When a user asks Claude to compare vendors in your category, Claude-SearchBot and Claude-User can still read your comparison page because you left them allowed, so Claude can describe your current plans accurately.
Why it matters
Claude is widely used for research and buying decisions at work. Getting the bot rules right lets you make a training decision without losing visibility in Claude's search answers.
Related service
Generative Engine Optimization (GEO)
We help your brand get found, understood and cited by ChatGPT, Perplexity, Gemini and Google AI Overviews.
Generative engine optimization servicesSources
Published by Vidern, founded and led by Malhar Shah. Updated .
See how AI assistants describe your brand
The free GEO audit scores your readiness for ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews from 0 to 100, with an action plan, within 24 hours.
Related terms
- AI crawlerAn AI crawler is an automated bot run by an AI company that fetches web pages to train models, to build a search index, or to answer a user's question.
- GPTBotGPTBot is OpenAI's web crawler for collecting content that may be used to train its generative AI models, and it follows robots.txt.
- PerplexityBotPerplexityBot is Perplexity's crawler for surfacing and linking websites in its answers, and Perplexity says it isn't used to train AI models.
- robots.txtrobots.txt is a plain-text file at the root of a website that tells crawlers which URLs they may or may not request.