AI crawler reference
Meta-ExternalFetcher: may bypass robots.txt
Meta-ExternalFetcher supports agentic AI capabilities, including helping AI navigate websites to complete tasks for people. Meta documents that it may bypass robots.txt rules, so a robots.txt entry is not a reliable way to stop it.
The facts
Taken from Meta's own documentation.
- robots.txt token
- meta-externalfetcher
- Operator
- Meta
- What it feeds
- Live user fetch
- Obeys robots.txt
- No. Meta documents that it may bypass robots.txt rules.
- User-agent string
- meta-externalfetcher/1.1
What each choice costs
Neither option is free. This is the trade.
If you allow it
Nothing to configure — it proceeds regardless of robots.txt.
User-agent: meta-externalfetcher Allow: /
If you block it
A robots.txt rule is not reliable here. An effective block needs network or server-level rules.
User-agent: meta-externalfetcher Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- One of three agents on this list that may ignore robots.txt, alongside Perplexity-User and facebookexternalhit.
- It is an agentic fetcher, so it is trying to complete a task on a page rather than index it.
- Agentic traffic is growing, and it is worth watching in logs separately from crawling.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Meta publishes no IP list for this agent, so a request carrying this name cannot be confirmed as genuine.
Questions
Can we block it with robots.txt?
Not reliably. Meta states it may bypass those rules, so blocking has to happen at the network or server layer.
What is it doing on our site?
Helping Meta's AI navigate pages to complete a task someone asked for, rather than collecting content for an index.
Is it the same as Meta-ExternalAgent?
No. Meta-ExternalAgent is the training crawler and it does obey robots.txt.
The engine it serves
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap