AI crawler reference
Meta-ExternalAgent: Meta's training crawler
Meta-ExternalAgent crawls for training foundation AI models and for improving products by indexing content directly. It is the one Meta bot that reliably obeys robots.txt — the other two may bypass it, which makes the distinction worth knowing.
The facts
Taken from Meta's own documentation.
- robots.txt token
- meta-externalagent
- Operator
- Meta
- What it feeds
- Model training
- Obeys robots.txt
- Yes, unlike Meta's other two agents.
- User-agent string
- meta-externalagent/1.1
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages may be used to train Meta's models and to index content for Meta products.
User-agent: meta-externalagent Allow: /
If you block it
You opt out of that. Meta-ExternalFetcher and facebookexternalhit are not covered by the same rule.
User-agent: meta-externalagent Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- Meta publishes three tokens and only this one reliably honours robots.txt.
- Meta AI reaches a very large audience through WhatsApp, Instagram and Facebook, so brand representation here has wide reach.
- A robots.txt rule naming only this token leaves the other two Meta agents untouched.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Meta publishes no IP list for this agent, so a request carrying this name cannot be confirmed as genuine.
Questions
Does this cover all of Meta's crawling?
No. Meta-ExternalFetcher and facebookexternalhit are separate, and Meta documents that both may bypass robots.txt.
Does blocking it affect link previews?
No. Previews on Facebook, Instagram and WhatsApp come from facebookexternalhit.
Is it worth allowing?
It is the usual training trade: wider representation in a model that reaches a very large audience, with no citation attached.
The engine it serves
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap