AI crawlers, and what each one actually does
There is no single list of AI crawlers, because every operator publishes its own in its own help centre. This is that list in one place: the exact robots.txt token, what each bot actually feeds, and whether a robots.txt rule stops it — three of them ignore robots.txt, which is the part most rules miss.
OpenAI
GPTBotModel trainingYes. OpenAI documents robots.txt as the control for it.OAI-SearchBotSearch indexYes. OpenAI documents robots.txt as the control for it.ChatGPT-UserLive user fetchYes, per OpenAI's documentation for this agent.OAI-AdsBotAd safetyYes. OpenAI documents robots.txt as the control for it.
Microsoft
Ready to Dominate AI Search?
Get started now
Allswap