AI crawler reference
facebookexternalhit: link previews
facebookexternalhit fetches a page shared on Meta's apps to build the preview: title, description and thumbnail. It reads your Open Graph tags. Meta documents that it might bypass robots.txt when running security or integrity checks.
The facts
Taken from Meta's own documentation.
- robots.txt token
- facebookexternalhit
- Operator
- Meta
- What it feeds
- Live user fetch
- Obeys robots.txt
- Not always. Meta states it might bypass robots.txt during security or integrity checks.
- User-agent string
- facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
What each choice costs
Neither option is free. This is the trade.
If you allow it
Links to your site shared on Facebook, Instagram, WhatsApp and Threads render with a proper title, description and image.
User-agent: facebookexternalhit Allow: /
If you block it
Your shared links can lose their preview and render as bare URLs, which cuts click-through sharply.
User-agent: facebookexternalhit Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- This is the oldest bot on the list and the one most often blocked by accident in a blanket AI rule.
- It reads Open Graph tags, so a missing og:image shows here before anywhere else.
- Blocking it is a social sharing decision, not an AI one.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Meta publishes no IP list for this agent, so a request carrying this name cannot be confirmed as genuine.
Questions
Why do our shared links have no preview?
Most often this agent is blocked, or the page has no Open Graph tags for it to read. Check both.
Is it an AI crawler?
Not really. It predates all of them and exists to build link previews, but it gets swept up in blanket AI blocking rules.
Does it obey robots.txt?
Usually, but Meta states it might bypass it during security or integrity checks.
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap