AI crawler reference
Claude-SearchBot: search quality for Claude
Claude-SearchBot navigates the web to improve the quality of Claude's search results. Where ClaudeBot feeds training, this one feeds retrieval, which makes it the Anthropic token most directly connected to whether your pages can be surfaced and cited in Claude's answers.
The facts
Taken from Anthropic's own documentation.
- robots.txt token
- Claude-SearchBot
- Operator
- Anthropic
- What it feeds
- Search index
- Obeys robots.txt
- Yes, alongside Anthropic's other bots.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages stay eligible to be found and cited when Claude searches the web.
User-agent: Claude-SearchBot Allow: /
If you block it
You lose that eligibility. As with OpenAI, the search token is the costly one to block.
User-agent: Claude-SearchBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- The training and search split mirrors OpenAI's. Allow search, decide separately on training.
- It is the newest of Anthropic's three tokens, so it is missing from most existing robots.txt files.
- Blocking it does not stop Claude discussing your brand from what it already knows.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Anthropic publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
How is this different from ClaudeBot?
ClaudeBot collects content that may improve the models. Claude-SearchBot supports search quality, so it is the one tied to retrieval and citation.
Should we allow it if we block training?
That is the common position, and a coherent one: stay citable while opting out of training.
Does it obey robots.txt?
Yes. Anthropic states all of its bots honour standard directives.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap