Blog · 7 July 2026
GPTBot, ChatGPT-User and OAI-SearchBot: which OpenAI crawler does what?
OpenAI operates three distinct crawlers with three distinct jobs. GPTBot gathers content for model training. OAI-SearchBot builds the index behind ChatGPT's search. ChatGPT-User fetches a page live, in the moment, because a person just asked ChatGPT something your page can answer. Blocking one does not block the others, and most robots.txt policies we see were written with only one of the three in mind.
Why one company runs three crawlers
Because an AI assistant touches the web in three different ways, at three different times. Long before anyone asks a question, content is collected to train the model itself. Continuously, an index of the live web is maintained so the assistant can search. And at the moment of a conversation, individual pages are fetched so the answer can quote what a site says right now. Each job gets its own agent, its own user-agent string, and its own robots.txt token, all documented in OpenAI's bot documentation, which means each can be allowed or blocked independently. The other major providers have converged on the same split: Anthropic pairs ClaudeBot with a live Claude-User agent, and Perplexity pairs PerplexityBot with Perplexity-User. Understanding one family means understanding them all.
The three agents, one by one
GPTBot
The crawler that collects public web content used to train OpenAI's models. It visits on its own schedule, unprompted by any user. Blocking it in robots.txt is the standard way publishers keep content out of future training runs. It is also the crawler people usually mean when they say they have "blocked ChatGPT", which is where the confusion starts, because blocking GPTBot does not touch the other two.
OAI-SearchBot
The crawler behind ChatGPT's search features. It builds and refreshes the index that decides which sites can be found and linked when the assistant searches the web. Block it and your site stops being discoverable in that index, whatever your training policy says. In our audits this is also the agent most often locked out by accident, because security layers tend to treat newer crawlers with the least trust.
ChatGPT-User
The agent that fetches a specific page during a conversation, when a user asks about something and the assistant goes to read the source directly. These visits are triggered by real people in real time, often asking about your brand specifically. This is the agent that gets served your page at the moment of decision, and if what it receives is an empty application shell, the assistant answers from whatever else it can find.
Where policies go wrong
The most common failure is the mismatch between intention and rule. A publisher decides to keep its content out of AI training, writes a robots.txt rule for GPTBot, and believes the job is done. The rule does exactly what it says and nothing more: OAI-SearchBot and ChatGPT-User carry on as before. The reverse mistake costs more. A brand blocks everything with "AI" in the name, or switches on a CDN bot-protection mode that challenges declared crawlers wholesale, and quietly disappears from the index and from live answers at once. The training question and the answering question are different questions, and a deliberate policy answers them separately.
"Blocking GPTBot keeps you out of the training data. Blocking ChatGPT-User keeps you out of the answer."
What should your policy be?
For most brands the goal is simple: be present at answer time. That means OAI-SearchBot and ChatGPT-User should be allowed and, more importantly, should be served the real page rather than a shell, which is a rendering question no robots.txt can fix. The training decision is genuinely yours to make: a publisher monetising content has a case for blocking GPTBot, while a brand that wants AI systems to know it deeply has a case for allowing it. Either position is defensible. Holding it by accident is not. And the accidental version is about to get more common: from 15 September 2026, Cloudflare's new defaults will block Agent-category crawlers on ad-bearing pages for new domains, making live-fetch access something owners must consciously allow.
Whatever you decide, verify it from the outside. Rules say what should happen; only a fetch shows what does. The two diverge more often than anyone expects, usually at the CDN, and the divergence never appears in your analytics.
Test all three OpenAI agents on your site now
The free Agent Parity Check fetches any public page as GPTBot, ChatGPT-User and OAI-SearchBot, plus PerplexityBot, ClaudeBot and CCBot, and shows the status and size each is served. 30 seconds, no access needed.
Run the free check