Blog · 7 July 2026 · Updated 20 October 2026
GPTBot, ChatGPT-User and OAI-SearchBot: which OpenAI crawler does what?
OpenAI documents four crawlers, and three decide how ChatGPT sees your site: GPTBot gathers training data, OAI-SearchBot builds the search index, and ChatGPT-User fetches your page live when a question in ChatGPT calls for it. The fourth, OAI-AdsBot, only checks ad landing pages. Blocking one does not block the others.
Why one company runs separate crawlers
Because an AI assistant touches the web in three different ways, at three different times. Long before anyone asks a question, content is collected to train the model itself. Continuously, an index of the live web is maintained so the assistant can search. And at the moment of a conversation, individual pages are fetched so the answer can quote what a site says right now. Each job gets its own agent, its own user-agent string, and its own robots.txt token, all documented in OpenAI's bot documentation, which says each setting is independent of the others. robots.txt sets the rules for the crawlers that visit on their own schedule, while a fetch that a person starts may not follow it. The other major providers have converged on the same split: Anthropic pairs ClaudeBot with a live Claude-User agent, and Perplexity pairs PerplexityBot with Perplexity-User. Understanding one family means understanding them all.
GPTBot, OAI-SearchBot and ChatGPT-User, one by one
GPTBot
The crawler that collects public web content used to train OpenAI's models. It visits on its own schedule, unprompted by any user. Blocking it in robots.txt is the standard way publishers keep content out of future training runs. It is also the crawler people usually mean when they say they have "blocked ChatGPT", which is where the confusion starts, because blocking GPTBot does not touch the other two.
OAI-SearchBot
The crawler behind ChatGPT's search features. It builds and refreshes the index that decides which sites can be found and linked when the assistant searches the web. Block it and your site stops being discoverable in that index, whatever your training policy says. In our audits this is also the agent most often locked out by accident, because security layers tend to treat newer crawlers with the least trust.
ChatGPT-User
The agent that fetches a specific page during a conversation, when a user asks about something and the assistant goes to read the source directly. These visits are triggered by real people in real time, often asking about your brand specifically. This is the agent that gets served your page at the moment of decision, and if what it receives is an empty application shell, the assistant answers from whatever else it can find. OpenAI also notes that because a user starts these fetches, robots.txt rules may not apply to them.
Where policies go wrong
The most common failure is the mismatch between intention and rule. A publisher decides to keep its content out of AI training, writes a robots.txt rule for GPTBot, and believes the job is done. The rule does what it says and nothing more: OAI-SearchBot and ChatGPT-User carry on as before. The reverse mistake costs more. A brand blocks everything with "AI" in the name, or switches on a CDN bot-protection mode that challenges declared crawlers wholesale, and disappears from the index and from live answers at once. The training question and the answering question are different questions, and a deliberate policy answers them separately.
"Blocking GPTBot keeps you out of the training data. Blocking ChatGPT-User keeps you out of the answer."
What should your policy be?
For most brands the goal is simple: be present at answer time. That means OAI-SearchBot and ChatGPT-User should be allowed and, more importantly, should be served the real page rather than a shell, which is a rendering question no robots.txt can fix. The training decision is yours to make: a publisher monetising content has a case for blocking GPTBot, while a brand that wants AI systems to know it deeply has a case for allowing it. Either position is defensible. Holding it by accident is not. And the accidental version is getting more common: since 15 September 2026, Cloudflare's new defaults for new ad-supported domains block Agent-category crawlers on pages that show ads, making live-fetch access something those owners must consciously allow.
Whatever you decide, verify it from the outside. Rules describe intent. A fetch shows the result. The two diverge more often than anyone expects, usually at the CDN, and the divergence never appears in your analytics.
Updated 20 October 2026: OpenAI now documents a fourth crawler, OAI-AdsBot, and says robots.txt rules may not apply to ChatGPT-User. Both are covered above, and the Cloudflare change is now described in the past tense.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot gathers content for model training. ChatGPT's answers about current web content come through OAI-SearchBot's index and ChatGPT-User's live fetches. A site that blocks only GPTBot can still be searched, fetched and cited by ChatGPT, and a site that allows GPTBot but blocks the other two can be in the training data yet invisible at answer time.
Should I block ChatGPT-User?
For most brands, no. ChatGPT-User fetches a page because a real person asked ChatGPT something your page can answer, often a question about your own brand. Blocking it means the assistant cannot read your page at the exact moment a buyer is asking about you. A robots.txt rule alone might not stop it anyway, because OpenAI says those rules may not apply to fetches a user starts.
How do I check which OpenAI crawlers can read my site?
Check your robots.txt for rules naming GPTBot, OAI-SearchBot or ChatGPT-User, check your CDN's bot settings, and then test from the outside. The free Agent Parity Check at getwellknown.ai fetches any public page as GPTBot, OAI-SearchBot and ChatGPT-User, plus PerplexityBot, ClaudeBot and CCBot, and shows what each is served.
See what OpenAI's crawlers are served from your site
The free Agent Parity Check fetches any public page as GPTBot, ChatGPT-User and OAI-SearchBot, plus PerplexityBot, ClaudeBot and CCBot, and shows the status and size each is served. All it needs is a public URL, and results come back in about 30 seconds.
Run the free check