AI crawler reference
OAI-SearchBot: the crawler behind ChatGPT search
OAI-SearchBot builds the search index ChatGPT draws on when it looks something up. It is the bot that decides whether your pages can be surfaced and linked in ChatGPT's answers, which makes it the one crawler in OpenAI's set with a direct line to citations and referral traffic.
The facts
Taken from OpenAI's own documentation.
- robots.txt token
- OAI-SearchBot
- Operator
- OpenAI
- What it feeds
- Search index
- Obeys robots.txt
- Yes. OpenAI documents robots.txt as the control for it.
- User-agent string
- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages can be surfaced and linked in ChatGPT's search results, which is where citations and referral clicks come from.
User-agent: OAI-SearchBot Allow: /
If you block it
Your pages stop being eligible to appear in ChatGPT search results. Of the OpenAI bots, this is the expensive one to block.
User-agent: OAI-SearchBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- This is the one to allow if you allow only one. It is the bot attached to citations.
- Blocking it does not stop ChatGPT talking about your brand. It stops ChatGPT linking to your pages while it does.
- It is easy to catch in a blanket rule. A robots.txt block written for training crawlers often takes this one with it.
Verify it
A user-agent header proves nothing on its own, because anything can send one. OpenAI publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Is this the same as GPTBot?
No. GPTBot collects pages for model training. OAI-SearchBot builds the search index ChatGPT queries, and only the second one leads to citations.
We blocked all AI bots. What did that cost?
Most likely eligibility for citation in ChatGPT search. Allowing OAI-SearchBot while blocking GPTBot is a common and coherent position.
Does allowing it guarantee we get cited?
No, it makes you eligible. Which pages get cited still depends on whether they answer the question asked.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap