AI crawler reference
GPTBot: OpenAI's training crawler
GPTBot is the crawler OpenAI uses to collect pages that may inform future foundation models. It is separate from the bot that builds ChatGPT's search index, so blocking GPTBot does not remove a site from ChatGPT search results. It obeys robots.txt and publishes its IP ranges.
The facts
Taken from OpenAI's own documentation.
- robots.txt token
- GPTBot
- Operator
- OpenAI
- What it feeds
- Model training
- Obeys robots.txt
- Yes. OpenAI documents robots.txt as the control for it.
- User-agent string
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages may be used to train future OpenAI models. There is no citation and no referral traffic attached to this — training is not retrieval.
User-agent: GPTBot Allow: /
If you block it
Future models may hold less about your brand from your own pages. Your presence in ChatGPT search results is unaffected, because that comes from OAI-SearchBot.
User-agent: GPTBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- GPTBot and OAI-SearchBot are different bots doing different jobs. Blocking the first does not block the second, and teams regularly block one meaning to block the other.
- Training is not citation. A page GPTBot reads earns no link and sends no traffic; a page OAI-SearchBot indexes can do both.
- The user-agent string can be forged. Verify against the published IP list before acting on server logs.
Verify it
A user-agent header proves nothing on its own, because anything can send one. OpenAI publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Does blocking GPTBot remove us from ChatGPT?
No. ChatGPT's search results are built by OAI-SearchBot, which is a separate token. Blocking GPTBot only opts your pages out of training.
Is there any visibility benefit to allowing it?
Indirectly, over time. What a model holds from training is what it can say without searching, so a brand well represented in training needs less retrieval to be named.
How do we confirm a request really was GPTBot?
Check the source IP against openai.com/gptbot.json. The user-agent header alone proves nothing, because anything can send it.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap