Crawlers and access
AI crawler
An AI crawler is an automated agent that collects web pages for an AI company, either to inform model training or to build the search index an assistant retrieves from. Each is identified by its own user-agent string in robots.txt.
In more detail
Crawlers split roughly into three jobs: gathering training material, building a retrieval index, and fetching a page live during a conversation. Allowing the retrieval and live-fetch agents is what lets your pages appear in answers.
How Allswap measures this
Guides that cover this
Related terms
GPTBot
GPTBot is OpenAI's crawler for collecting web pages that may inform future model training. It is controlled in robots.txt by its user-agent name, separately from OpenAI's search and live-fetch agents.
robots.txt
robots.txt is a file at the root of a site that tells crawlers which paths they may fetch. For AI search it is the primary control over whether assistants can reach your pages at all.
llms.txt
llms.txt is a plain-text file at the root of a site that gives AI systems a curated summary of what the site offers and where its key pages are. It is a proposed convention, not a requirement, and it is inexpensive to publish.
Training data
Training data is the body of text a model learned from. It sets the model's default understanding of a category and which brands it associates with what, before any live retrieval happens.
Ready to Dominate AI Search?
Get started now
Allswap