AI crawler reference
Googlebot: the crawler behind AI Overviews
Googlebot builds Google's Search index, and because AI features are built into Search, it is also what makes a page eligible for AI Overviews and AI Mode. There is no separate AI crawler to allow or block — the control is Googlebot's robots.txt rules and the snippet directives.
The facts
Taken from Google's own documentation.
- robots.txt token
- Googlebot
- Operator
- What it feeds
- Search index
- Obeys robots.txt
- Yes.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Standard Search indexing, and with it eligibility for AI Overviews and AI Mode. Google states there are no additional requirements beyond being indexed and snippet-eligible.
User-agent: Googlebot Allow: /
If you block it
You leave Google Search altogether. There is no way to keep organic listings while opting out of AI features through robots.txt alone.
User-agent: Googlebot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- There is no AI Overviews crawler. Teams looking for one to block are looking for something that does not exist.
- The real controls are nosnippet, data-nosnippet, max-snippet and noindex — and each of them costs you normal search snippets too.
- Google states no special schema, chunking or llms.txt file is required or used for AI features.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Google publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
How do we opt out of AI Overviews without leaving Search?
Through snippet controls rather than robots.txt. nosnippet or a low max-snippet limits what can be shown, but the same limit applies to your ordinary search snippet.
Is there special markup that helps us appear?
Google says there is not. A page must be indexed and eligible for a snippet, and normal SEO best practice applies from there.
Does Google use llms.txt?
Google's guidance states llms.txt files are not used by Google Search. Other assistants do fetch it, so it is still worth publishing.
The engine it serves
Guides that cover this
How Allswap measures this
Ready to Dominate AI Search?
Get started now
Allswap