AI crawler reference
Google-CloudVertexBot: Vertex AI agent crawls
Google-CloudVertexBot crawls sites on behalf of site owners who are building Vertex AI Agents. It acts for a customer who asked for that crawl, so unless someone is building an agent on your content it has little bearing on your visibility.
The facts
Taken from Google's own documentation.
- robots.txt token
- Google-CloudVertexBot
- Operator
- What it feeds
- Model training
- Obeys robots.txt
- Yes.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Vertex AI agent builders can index your pages for their own agents, including your own team if you build one.
User-agent: Google-CloudVertexBot Allow: /
If you block it
Those crawls stop, including any you commissioned yourself. Search and AI Overviews are unaffected.
User-agent: Google-CloudVertexBot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- It acts for a Google Cloud customer rather than for Google's own products.
- If your own team builds a Vertex agent over your site, blocking this token blocks your own project.
- It has no connection to Gemini training, which is Google-Extended's job.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Google publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
Who is actually crawling us?
A Google Cloud customer building a Vertex AI Agent, using Google's infrastructure to do it.
Does it affect Search?
No. It is separate from Googlebot and from Search indexing entirely.
Should we block it?
Only if you object to third parties building agents on your content — and check first that your own team is not doing exactly that.
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap