AI crawler reference
ClaudeBot: Anthropic's training crawler
ClaudeBot collects web content to improve Anthropic's models. It is one of three Anthropic tokens, alongside Claude-SearchBot for search quality and Claude-User for fetches a person asked for. All three obey robots.txt and share one published IP list.
The facts
Taken from Anthropic's own documentation.
- robots.txt token
- ClaudeBot
- Operator
- Anthropic
- What it feeds
- Model training
- Obeys robots.txt
- Yes. Anthropic honours standard robots.txt directives and also supports Crawl-delay.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages may inform Anthropic's models, which shapes what Claude can say about your brand without looking anything up.
User-agent: ClaudeBot Allow: /
If you block it
Your pages are excluded from that. Claude can still reach you through Claude-SearchBot and Claude-User unless those are blocked too.
User-agent: ClaudeBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- Anthropic publishes three tokens. A rule naming only ClaudeBot leaves the other two untouched.
- Crawl-delay is honoured here, which is unusual — it is a way to slow a bot down rather than shut it out.
- Older robots.txt files often use anthropic-ai, a legacy token. The current set is ClaudeBot, Claude-SearchBot and Claude-User.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Anthropic publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Is anthropic-ai still the right token?
It is the older one. Anthropic's current documentation names ClaudeBot, Claude-SearchBot and Claude-User, so a modern rule should list those.
Can we slow it down instead of blocking it?
Yes. Anthropic supports the Crawl-delay extension, which most operators ignore.
How do we verify it?
Check the source IP against claude.com/crawling/bots.json, which covers all of Anthropic's bots.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap