AI crawler reference

Claude-User: fetches on request

Claude-User fetches a page because a person using Claude asked for it. It supports user requests rather than building an index, so traffic from it reflects someone choosing your page. Anthropic states it honours robots.txt, which is stricter than several comparable user agents.

Analyze your AI presence Book a demo

The facts

Taken from Anthropic's own documentation.

robots.txt token: Claude-User
Operator: Anthropic
What it feeds: Live user fetch
Obeys robots.txt: Yes. Notably stricter than some user-triggered fetchers, several of which do not.

What each choice costs

Neither option is free. This is the trade.

If you allow it

People using Claude can open your pages when they ask, and the content is available to answer them.

User-agent: Claude-User
Allow: /

If you block it

Someone who explicitly asked Claude to read your page is told it cannot be read.

User-agent: Claude-User
Disallow: /

Deciding this alongside the other bots is easier in one place: build the whole file.

Worth knowing

Anthropic applying robots.txt to a user-initiated fetch is a deliberate choice. Perplexity and Meta take the opposite position on their equivalents.
Requests here are a demand signal rather than crawl volume.
It has no effect on whether Claude can find you in the first place.

Verify it

A user-agent header proves nothing on its own, because anything can send one. Anthropic publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.

https://claude.com/crawling/bots.json Anthropic documentation