Definition
GPTBot is OpenAI's web crawler for collecting content that may be used to train its generative AI models, and it follows robots.txt.
GPTBot explained
OpenAI runs separate bots for separate jobs. GPTBot is the training crawler: according to OpenAI's documentation, it crawls content that may be used in training its generative AI foundation models. It is not the bot behind ChatGPT's search results; that is OAI-SearchBot, and page visits requested by users come from ChatGPT-User.
GPTBot respects robots.txt, so you can block it with a User-agent: GPTBot group and a Disallow rule without affecting whether your site can appear in ChatGPT search. The reverse also holds: allowing GPTBot doesn't put your pages into ChatGPT's search answers, because that depends on OAI-SearchBot.
Whether to allow GPTBot is a business decision. Publishers that license their content often block it. Brands that want future models to know their products, category and facts often allow it, because training data shapes what a model says when it answers without searching the web.
Example
Your robots.txt blocks GPTBot because a developer copied a template years ago. Your pages still appear in ChatGPT search through OAI-SearchBot, but you decide to allow GPTBot too, so future models learn your current product line rather than older descriptions from third-party sites.
Why it matters
Allowing or blocking GPTBot changes what future OpenAI models may learn about your brand, not your current ChatGPT search visibility. Mixing the two up is a common mistake.
Related service
Generative Engine Optimization (GEO)
We help your brand get found, understood and cited by ChatGPT, Perplexity, Gemini and Google AI Overviews.
Generative engine optimization servicesSources
- OpenAI: Overview of OpenAI crawlers (checked 30 Sep 2026)
Published by Vidern, founded and led by Malhar Shah. Updated .
See how AI assistants describe your brand
The free GEO audit scores your readiness for ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews from 0 to 100, with an action plan, within 24 hours.
Related terms
- OAI-SearchBotOAI-SearchBot is OpenAI's crawler for surfacing websites in ChatGPT search, and blocking it keeps your pages out of ChatGPT search answers.
- ChatGPT-UserChatGPT-User is the user agent OpenAI uses when ChatGPT visits a web page because of a user's request, rather than as part of automatic crawling.
- AI crawlerAn AI crawler is an automated bot run by an AI company that fetches web pages to train models, to build a search index, or to answer a user's question.
- robots.txtrobots.txt is a plain-text file at the root of a website that tells crawlers which URLs they may or may not request.
- Google-ExtendedGoogle-Extended is a robots.txt token controlling whether Google-crawled content can train and ground Gemini models; it doesn't affect Google Search.