AI crawler reference
Perplexity-User: fetches robots.txt does not stop
Perplexity-User visits a page because someone asked Perplexity a question that needs it. Perplexity documents that, because a user requested the fetch, this agent generally ignores robots.txt rules — so a robots.txt block is not the control here.
The facts
Taken from Perplexity's own documentation.
- robots.txt token
- Perplexity-User
- Operator
- Perplexity
- What it feeds
- Live user fetch
- Obeys robots.txt
- Generally no. Perplexity documents that this agent ignores robots.txt because a person requested the fetch.
- User-agent string
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
What each choice costs
Neither option is free. This is the trade.
If you allow it
Nothing to do — it proceeds on user request regardless of what robots.txt says.
User-agent: Perplexity-User Allow: /
If you block it
A robots.txt rule will not reliably stop it. Blocking has to happen at the network or server layer, using the published IP list.
User-agent: Perplexity-User Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- This is the fact most often missed on this list. A robots.txt entry for Perplexity-User is not an effective block.
- The published IP list exists for verification, and it is also what you would use to block at the edge.
- The reasoning is consistent across the industry: an agent acting for a named person is treated as that person browsing.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Perplexity publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Our robots.txt blocks it. Are we covered?
Probably not. Perplexity documents that this agent generally ignores robots.txt, so an effective block needs to be enforced at the network layer.
Is that allowed?
It is the same reasoning several operators apply to user-triggered fetches: the request stands in for a person visiting the page.
How do we block it properly?
Match on the published IP ranges at your CDN or firewall rather than relying on robots.txt.
The engine it serves
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap