AI crawler reference
Bingbot: one crawler, search and Copilot
Bingbot builds the Bing index that Microsoft Copilot draws on. Microsoft offers no separate AI training token; instead it uses page-level meta tags. NOCACHE limits Copilot to your URL, title and snippet, and NOARCHIVE goes further still.
The facts
Taken from Microsoft's own documentation.
- robots.txt token
- bingbot
- Operator
- Microsoft
- What it feeds
- Search index
- Obeys robots.txt
- Yes.
What each choice costs
Neither option is free. This is the trade.
If you allow it
Standard Bing indexing, and with it eligibility to be drawn on by Copilot.
User-agent: bingbot Allow: /
If you block it
You leave Bing search and Copilot together. There is no robots.txt way to separate them.
User-agent: bingbot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- Microsoft's controls are meta tags, not robots.txt tokens, which is why a robots.txt written for AI bots does nothing here.
- NOCACHE means only your URL, title and snippet may be used in Copilot answers and in training.
- You can target Microsoft specifically by replacing robots with bingbot in the meta tag name.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Microsoft publishes no IP list for this agent, so a request carrying this name cannot be confirmed as genuine.
Questions
Which token do we block for Copilot training?
There is not one. Microsoft handles this with the NOCACHE and NOARCHIVE meta tags rather than a robots.txt user agent.
What does NOCACHE actually do?
Content marked NOCACHE may still appear in Copilot answers, but only as URL, title and snippet, and only those parts may be used in training.
Can we apply a tag to Bing alone?
Yes. Use bingbot in place of robots in the meta tag name, or as an X-Robots-Tag response header.
The engine it serves
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap