AI crawler reference
CCBot: the crawl behind many models
CCBot is Common Crawl's crawler. It builds a free public dataset that many organisations use as training input, so blocking it is unusually wide-reaching: one rule affects every model builder who draws on that archive rather than a single company.
The facts
Taken from Common Crawl's own documentation.
- robots.txt token
- CCBot
- Operator
- Common Crawl
- What it feeds
- Model training
- Obeys robots.txt
- Yes. Common Crawl documents a CCBot disallow as the control.
- User-agent string
- CCBot/2.0 (https://commoncrawl.org/faq/)
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your pages enter a public archive that many model builders use, including ones with no crawler of their own.
User-agent: CCBot Allow: /
If you block it
You are excluded from that archive going forward. Crawls already published are not retrospectively changed.
User-agent: CCBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- One block, many downstream effects. No other token on this list has that reach.
- It is a non-profit archive rather than a company collecting for its own product.
- Common Crawl warns that other crawlers falsely identify as CCBot, so verify against its published IP list before drawing conclusions from logs.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Common Crawl publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Why does this one matter more than it looks?
Because the dataset is public. Many model builders use it instead of crawling themselves, so a single rule reaches all of them.
Does blocking it remove us from past crawls?
No. It applies going forward. Archives already published stay as they are.
We see heavy CCBot traffic. Is it real?
Possibly not. Common Crawl states it is aware of crawlers falsely identifying themselves as CCBot, and recommends verifying rather than trusting the user-agent string. It publishes IP ranges and supports reverse DNS for exactly that.
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap