AI crawler reference
Amazonbot: Amazon's crawler
Amazon states Amazonbot is used to improve its products and services, and that the content may be used to train Amazon AI models. It honours robots.txt, and also respects rel=nofollow and page-level robots meta tags, which is broader than most crawlers on this list.
The facts
Taken from Amazon's own documentation.
- robots.txt token
- Amazonbot
- Operator
- Amazon
- What it feeds
- Model training
- Obeys robots.txt
- Yes, and unusually broadly — Amazon states it honours robots.txt, rel=nofollow and page-level robots meta tags.
- User-agent string
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
What each choice costs
Neither option is free. This is the trade.
If you allow it
Your content may be used to improve Amazon products and services, and may be used to train Amazon AI models.
User-agent: Amazonbot Allow: /
If you block it
You opt out of that. Amazon documents no citation or referral path here, so there is no direct visibility cost to weigh.
User-agent: Amazonbot Disallow: /
Deciding this alongside the other bots is easier in one place: build the whole file.
Worth knowing
- Amazon honours more than robots.txt. rel=nofollow and page-level robots meta tags also apply, which gives finer control than a blanket rule.
- Amazon's own wording is broad: improving products and services, and possibly training AI models. It does not name a single product.
- The published IP list is the way to confirm a request, since the user-agent string carries a placeholder version.
Verify it
A user-agent header proves nothing on its own, because anything can send one. Amazon publishes the IP ranges its bots use, so a request is only genuine if the source IP appears there.
Questions
What is our content actually used for?
Amazon says it is used to improve its products and services, and that it may be used to train Amazon AI models. It does not break that down by product, so anything more specific is guesswork.
Can we allow it but limit what it takes?
Yes, more so than with most crawlers. Amazon states it respects rel=nofollow and page-level robots meta tags as well as robots.txt, so you can control it page by page.
How do we verify it?
Check the source IP against the list Amazon publishes. The user-agent string ends in a placeholder version, so it is a poor thing to match on.
Guides that cover this
Ready to Dominate AI Search?
Get started now
Allswap