AI crawler reference
Google-Extended: what it really controls
Google-Extended controls whether your content may be used to train Gemini models and to ground responses in Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal. It does not control AI Overviews.
The facts
Taken from Google's own documentation.
- robots.txt token
- Google-Extended
- Operator
- What it feeds
- Model training
- Obeys robots.txt
- Yes. It exists purely as a robots.txt control.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your content may be used for Gemini training and for grounding in Gemini Apps and Vertex AI.
User-agent: Google-Extended Allow: /
If you block it
You opt out of that. Google states your Search inclusion and ranking are unaffected — and, importantly, AI Overviews are unaffected too.
User-agent: Google-Extended Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- The most misunderstood token on this list. Blocking it does not remove you from AI Overviews or AI Mode.
- It is not really a crawler. No separate bot fetches with this name; it is a control token read from robots.txt.
- Google is explicit that it is not a ranking signal, so there is no search penalty for blocking it.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Google publishes no IP list for this agent, so a request carrying this name cannot be confirmed as genuine.
Questions
Does blocking Google-Extended remove us from AI Overviews?
No. AI features in Search are served through Googlebot, so the controls for them are Googlebot's robots.txt rules plus nosnippet, data-nosnippet, max-snippet and noindex.
Will blocking it hurt our rankings?
Google states it does not affect inclusion in Search and is not used as a ranking signal.
Then how do we control AI Overviews?
Through snippet controls. nosnippet removes you from the text shown, and max-snippet limits how much can be used. Both also reduce your normal search snippet.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap