AI crawler reference
PerplexityBot: the search crawler
PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity states plainly that it is not used to crawl content for AI foundation models, so allowing it is a citation decision rather than a training one — an unusually clean choice for publishers.
The facts
Taken from Perplexity's own documentation.
- robots.txt token
- PerplexityBot
- Operator
- Perplexity
- What it feeds
- Search index
- Obeys robots.txt
- Yes. Perplexity documents robots.txt as the control.
- User-agent string
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages can be surfaced and linked in Perplexity answers. Perplexity cites heavily, so this is a direct route to visible attribution.
User-agent: PerplexityBot Allow: /
If you block it
You give up citation in a product built around citing sources, and gain nothing on the training side, because this bot does not feed training.
User-agent: PerplexityBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- The no-training statement makes this the easiest allow decision of any bot on this list.
- Perplexity shows its sources prominently, so a citation here is more visible to a reader than in most assistants.
- Perplexity-User is separate and behaves differently — it generally ignores robots.txt.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Perplexity publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Does allowing it mean training on our content?
Perplexity states PerplexityBot is not used to crawl content for foundation models. Allowing it is about search visibility.
What does blocking cost?
Eligibility for citation in a product whose answers lead with sources. There is no training benefit to offset it.
How do we verify it?
Match the source IP against perplexity.com/perplexitybot.json.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap