AI crawler reference
GoogleOther: the general-purpose crawler
GoogleOther is a generic crawler used by Google product teams for research and development. It does not power any particular user-facing product, which makes it the lowest-stakes Google token: blocking it has no documented effect on Search or on AI features.
The facts
Taken from Google's own documentation.
- robots.txt token
- GoogleOther
- Operator
- What it feeds
- Model training
- Obeys robots.txt
- Yes.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Google product teams may fetch your pages for internal research and development work.
User-agent: GoogleOther Allow: /
If you block it
Those fetches stop. Google does not tie this crawler to any user-facing product, so nothing visible changes.
User-agent: GoogleOther Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- It shares infrastructure with Googlebot, so it can look like Googlebot in logs until you read the token.
- Because it is not tied to a product, there is no visibility argument either way.
- It is often blocked by mistake in rules aimed at Googlebot, which is a far more costly block.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Google publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Is blocking it safe?
It carries no documented effect on Search or AI features. It is the least consequential Google token to block.
Why does it look like Googlebot in our logs?
It shares crawl infrastructure. The user-agent token is what distinguishes it, and the IP ranges are the same.
Does it feed Gemini?
Gemini training and grounding are controlled by Google-Extended, not by this token.
The engine it serves
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap