Leitfaden
AI crawlers, explained
AI crawlers do three different jobs: collecting training material, building the search index an assistant retrieves from, and fetching a page live during a conversation. Blocking them is not one decision — allowing the retrieval and live-fetch agents is what keeps you eligible to appear in answers.
Zuletzt aktualisiert
The three jobs
Treating every AI crawler as one thing is the most common and most expensive mistake here. A site that blocks all of them to opt out of training also removes itself from the answers its buyers read.
- Training collection — gathers pages that may inform a future model. Blocking this does not remove you from today's answers.
- Retrieval indexing — builds the index an assistant searches. Blocking this does remove you from answers that rely on search.
- Live fetch — retrieves a specific page during a conversation, usually because someone asked about it. Blocking this stops an assistant reading your page even when a user requests it.
The agents worth naming explicitly
Several of these read a rule addressed to them and ignore a wildcard block, so naming them is clearer than relying on a catch-all.
- GPTBot — OpenAI training collection
- OAI-SearchBot — the index behind ChatGPT's web search
- ChatGPT-User — live fetch during a conversation
- PerplexityBot — Perplexity's index
- ClaudeBot and anthropic-ai — Anthropic
- Google-Extended — governs whether your pages inform Gemini and AI Overviews, separately from ordinary search
- Bingbot — feeds Copilot as well as Bing
- meta-externalagent — Meta AI
Google-Extended deserves its own decision
It is the one people get wrong most often. Google-Extended governs AI answer generation and is separate from the crawling that feeds ordinary search results, so allowing it does not change your rankings and blocking it does not protect them.
Blocking it removes your content from informing Gemini and AI Overviews while leaving you in the index — which is a coherent position, but it should be chosen deliberately rather than inherited from a default.
Fragen
Will blocking AI crawlers hurt my SEO?
Blocking the AI-specific agents does not affect ordinary search indexing, which runs through Googlebot and Bingbot. What it does affect is whether your content can inform AI answers.
Can I allow retrieval but block training?
Yes, and many sites do. Allow OAI-SearchBot and ChatGPT-User while disallowing GPTBot, for example. The agents are separate precisely so this choice exists.
How do I check what my site currently allows?
Read your robots.txt as the crawler would: find the rule block addressed to each agent name, remembering that a named block overrides the wildcard entirely rather than adding to it.
Hier verwendete Begriffe
Weiterlesen
llms.txt, explained
What llms.txt is, what belongs in it, how it differs from robots.txt, and an honest account of what it does and does not do today.
How to get recommended by ChatGPT
What decides whether ChatGPT names your brand, which crawlers to allow, and the work that moves it — in the order worth doing.
How AI Overviews choose what to cite
What decides which pages appear in a Google AI Overview, how it relates to ordinary ranking, and what to do about it.
Bereit, die KI-Suche zu dominieren?
Jetzt loslegen
Allswap