Free tool
AI robots.txt generator
Most robots.txt generators hand you a list of bot names and let you tick them. That misses the part that matters: blocking a training crawler costs nothing, blocking a search crawler costs you every citation in that assistant, and three of these agents ignore robots.txt entirely. This one knows the difference and says so.
Start from a position
Keeps the crawlers that lead to citations and referral clicks. Opts out of model training.
Search crawlers — these lead to citations
OpenAI
Anthropic
Perplexity
Apple
Microsoft
Fetches a person asked for
OpenAI
Anthropic
Perplexity · documented to ignore robots.txt
Meta · documented to ignore robots.txt
Meta · may bypass robots.txt
Training crawlers — no citation, no traffic
OpenAI
Anthropic
Apple
Common Crawl
Amazon
ByteDance · no published compliance statement
Ad safety
OpenAI
Your robots.txt
Before you paste it
Put the file at the root of your domain, at /robots.txt. A rule lower down the path tree is not read.
Blocking Googlebot removes you from Google Search altogether, and with it AI Overviews — there is no separate AI crawler to block there. Blocking facebookexternalhit strips the preview from every link to your site shared on Facebook, Instagram, WhatsApp and Threads.
Already have a robots.txt? The checker reads the one you have and says what it is costing you.
Every agent, what it feeds and what each choice costs is set out one page per bot in the AI crawler reference, with the operator’s own documentation linked on each one.
Questions
Should I block AI crawlers?
Rarely all of them. The training crawlers take your content without sending anything back, so blocking those is a defensible position. The search crawlers are what make your pages eligible to be cited and linked in answers, and blocking those removes you from assistants that people now use instead of search. The middle preset here draws that line.
Will blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls Gemini training and grounding. AI features in Search are served through Googlebot, and Google states Google-Extended is not used as a ranking signal and does not affect inclusion in Search.
Why does the tool say some blocks will not hold?
Because robots.txt is a request rather than a barrier, and some agents are documented as ignoring it — Perplexity-User and Meta-ExternalFetcher say so outright. The rule is still written into the file, but stopping those needs a CDN or firewall rule matched against the operator's published IP ranges.
Does this file stop my content being used for training?
For the operators that honour it, going forward, yes. It cannot reach content already collected, and it has no effect on an operator that never crawled you directly — material gathered through a public dataset such as Common Crawl is governed by that crawler's own rule.
Do I still need a sitemap line?
It helps. Adding your site URL puts a Sitemap line in the output, which is a direct pointer for every crawler reading the file and costs nothing.
Ready to Dominate AI Search?
Get started now
Allswap