Crawlers and access
robots.txt
robots.txt is a file at the root of a site that tells crawlers which paths they may fetch. For AI search it is the primary control over whether assistants can reach your pages at all.
In more detail
Each AI crawler reads the rules addressed to its own user-agent name. Naming them explicitly, rather than relying on a wildcard, makes your intent unambiguous.
Related terms
AI crawler
An AI crawler is an automated agent that collects web pages for an AI company, either to inform model training or to build the search index an assistant retrieves from. Each is identified by its own user-agent string in robots.txt.
llms.txt
llms.txt is a plain-text file at the root of a site that gives AI systems a curated summary of what the site offers and where its key pages are. It is a proposed convention, not a requirement, and it is inexpensive to publish.
GPTBot
GPTBot is OpenAI's crawler for collecting web pages that may inform future model training. It is controlled in robots.txt by its user-agent name, separately from OpenAI's search and live-fetch agents.
Google-Extended
Google-Extended is the robots.txt control over whether your content may be used for Gemini training and for grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search and is not used as a ranking signal.
Ready to Dominate AI Search?
Get started now
Allswap