Crawlers and access
llms.txt
llms.txt is a plain-text file at the root of a site that gives AI systems a curated summary of what the site offers and where its key pages are. It is a proposed convention, not a requirement, and it is inexpensive to publish.
In more detail
A good llms.txt lists the pages that matter with a line of description each, so an assistant can orient quickly rather than inferring structure from navigation.
Some sites also publish a longer llms-full.txt carrying complete product facts, pricing and FAQs for assistants to quote accurately.
Guides that cover this
Related terms
robots.txt
robots.txt is a file at the root of a site that tells crawlers which paths they may fetch. For AI search it is the primary control over whether assistants can reach your pages at all.
AI crawler
An AI crawler is an automated agent that collects web pages for an AI company, either to inform model training or to build the search index an assistant retrieves from. Each is identified by its own user-agent string in robots.txt.
Structured data
Structured data is machine-readable markup that states what a page is and what it contains. It helps search engines show rich results and helps AI systems understand a page before deciding whether to use it.
Ready to Dominate AI Search?
Get started now
Allswap