AI crawler reference
Bytespider: ByteDance's crawler
Bytespider is ByteDance's crawler, collecting content for its AI models. It is the one bot on this list with no operator documentation behind it: ByteDance publishes no crawler page, no user-agent specification and no IP list, so there is nothing authoritative to check a request against.
The facts
ByteDance publishes no documentation for this bot, so these are observed rather than stated.
- robots.txt token
- Bytespider
- Operator
- ByteDance
- What it feeds
- Model training
- Obeys robots.txt
- Undocumented. ByteDance publishes nothing stating either way, so confirm it in your own logs rather than assume.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your content may be collected for ByteDance's AI models. There is no citation or referral attached.
User-agent: Bytespider Allow: /
If you block it
Worth confirming in your logs rather than assuming. If volume continues after a robots.txt rule, a firewall rule is the usual next step.
User-agent: Bytespider Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- Every other operator here publishes documentation. This one does not, so treat anything stated about Bytespider — including on this page — as second-hand until your own logs confirm it.
- With no published IP list there is no way to tell a real Bytespider request from something wearing the name.
- Blocking costs little in visibility terms, since there is no documented citation or referral path here.
Verify it
A user-agent header proves nothing on its own, because anything can send one. ByteDance publishes no IP list for this agent, so a request carrying this name cannot be confirmed as genuine.
ByteDance publishes no crawler documentation to link to.
Questions
Our robots.txt rule does not seem to work. What now?
Confirm it in your logs first. There is no published compliance statement to appeal to, so if requests continue the practical step is a block at the CDN or firewall.
What do we lose by blocking it?
Representation in ByteDance's models. There is no citation or referral traffic to give up.
Is the traffic genuine?
There is no way to be sure. The user-agent string is trivial to forge and ByteDance publishes no IP list to check it against, which is not true of any other bot here.
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap