Free tool
Which AI crawlers is your robots.txt blocking?
Most sites blocking AI crawlers are blocking the wrong ones. The list that circulates on blogs mixes training crawlers, which cost nothing to refuse, with the search crawlers that decide whether you can be cited at all — and usually takes the Facebook link-preview agent with it. Paste your file and see what yours is doing.
Rules can differ by path. Leave it as / to check the site as a whole, or enter a page to test that one.
8 of 22 AI crawlers blocked
Checked against /, using Google's matching rules: longest user-agent token wins, longest path pattern wins, and Allow beats Disallow on a tie.
Link previews are broken
facebookexternalhit builds the preview card for every link to your site shared on Facebook, Instagram, WhatsApp and Threads. Blocked, those links render as bare URLs. It is not an AI crawler; it gets swept up in blanket AI rules.
2 search crawlers blocked
These are the crawlers that make pages eligible to be cited and linked in answers: OAI-SearchBot, PerplexityBot. Blocking them gives up citations and referral clicks without opting out of any training.
1 block will not hold
robots.txt is a request, not a barrier, and these are not documented to honour it: facebookexternalhit. Stopping them needs a rule at your CDN or firewall matched against the operator's published IP ranges.
Google-Extended is blocked, which is fine
This opts you out of Gemini training and grounding. Google states it does not affect inclusion in Search and is not a ranking signal — and it does not control AI Overviews, which people often assume it does.
Search crawlers — these lead to citations
OpenAI · matched oai-searchbot, Disallow: /
Your pages stop being eligible to appear in ChatGPT search results. Of the OpenAI bots, this is the expensive one to block.
Anthropic · matched User-agent: *, Allow: /
Perplexity · matched perplexitybot, Disallow: /
You give up citation in a product built around citing sources, and gain nothing on the training side, because this bot does not feed training.
Google · matched User-agent: *, Allow: /
Apple · matched User-agent: *, Allow: /
Microsoft · matched User-agent: *, Allow: /
Fetches a person asked for
OpenAI · matched chatgpt-user, Disallow: /
A person who explicitly asks ChatGPT to read your page gets told it cannot. This is a user-facing refusal rather than an indexing decision.
Anthropic · matched User-agent: *, Allow: /
Perplexity · matched User-agent: *, Allow: /
Meta · matched User-agent: *, Allow: /
Meta · matched facebookexternalhit, Disallow: /
Your shared links can lose their preview and render as bare URLs, which cuts click-through sharply.
Training crawlers — no citation, no traffic
OpenAI · matched gptbot, Disallow: /
Future models may hold less about your brand from your own pages. Your presence in ChatGPT search results is unaffected, because that comes from OAI-SearchBot.
Anthropic · matched claudebot, Disallow: /
Your pages are excluded from that. Claude can still reach you through Claude-SearchBot and Claude-User unless those are blocked too.
Google · matched google-extended, Disallow: /
You opt out of that. Google states your Search inclusion and ranking are unaffected — and, importantly, AI Overviews are unaffected too.
Google · matched User-agent: *, Allow: /
Google · matched User-agent: *, Allow: /
Apple · matched User-agent: *, Allow: /
Meta · matched User-agent: *, Allow: /
Common Crawl · matched ccbot, Disallow: /
You are excluded from that archive going forward. Crawls already published are not retrospectively changed.
Amazon · matched User-agent: *, Allow: /
ByteDance · matched User-agent: *, Allow: /
Ad safety
OpenAI · matched User-agent: *, Allow: /
Want to rewrite it? Build a new robots.txt with the same list, starting from a position rather than a blank file.
How the check works
Your file is parsed in the browser and never sent anywhere. The matching follows Google’s rules rather than reading the file top to bottom, because robots.txt does not work in order:
- The most specific user-agent token wins. A file with a blanket Disallow under User-agent: * and an Allow under User-agent: GPTBot does not block GPTBot — the named token governs, and the wildcard is only a fallback.
- Within a group, the longest matching path pattern wins, not the first one written. Disallow: / followed by Allow: /blog leaves /blog crawlable.
- On an equal-length tie, Allow beats Disallow.
- Anything no rule matches is allowed. robots.txt is an opt-out, so silence means yes.
- Disallow with no value means nothing is disallowed, which is the opposite of Disallow: /.
Each agent, what it feeds and what blocking it costs is set out one page per bot in the AI crawler reference, each with the operator’s own documentation linked.
Questions
Is my robots.txt uploaded anywhere?
No. The parsing runs entirely in your browser. Nothing is sent to a server, and there is nothing to store.
I blocked the list I found on a blog. What did it cost me?
Usually citations. Those lists mix training crawlers, which take content and give nothing back, with search crawlers like OAI-SearchBot and PerplexityBot that decide whether your pages can be cited and linked at all. They also tend to include facebookexternalhit, which is not an AI crawler — it builds the link preview when someone shares you on Facebook, Instagram, WhatsApp or Threads.
Why does it say some blocks will not hold?
Because a few agents are documented as ignoring robots.txt. Perplexity-User and Meta-ExternalFetcher say so in their own docs, and ByteDance publishes nothing either way. A rule for those is a request that may simply be declined, so enforcing it needs a CDN or firewall rule matched against published IP ranges.
Does blocking Google-Extended remove me from AI Overviews?
No, and this is the most common misreading in the category. Google-Extended controls Gemini training and grounding. AI Overviews and AI Mode are served through Googlebot, because AI features are built into Search. Google states Google-Extended does not affect inclusion in Search and is not a ranking signal.
Why test a path rather than the whole site?
Because rules can differ by path. A file may allow crawling generally and block /blog, so a site-wide answer would be wrong for the pages you most want cited. Checking / tells you about the site as a whole; entering a page tells you about that page.
Bereit, die KI-Suche zu dominieren?
Jetzt loslegen
Allswap