llms.txt, explained

A crawler can read your navigation, but navigation is designed for people and often buries what matters. llms.txt states it directly: here are the pages that explain this product, here is the pricing, here is the documentation.

Some sites also publish a longer llms-full.txt carrying complete product facts, pricing and FAQs, so an assistant answering a question about you has accurate material to quote rather than assembling it from fragments.

Keep it short and link-led. A heading, a one-line description of the site, then sections of links with a line of description each.

• The pages that explain what you do, one line each
• Pricing, stated plainly, with the actual numbers
• Documentation or reference material, if you have it
• What you are not — a surprisingly useful line, because it heads off wrong associations
• A link to the longer llms-full.txt if you publish one

robots.txt controls access: which agents may fetch which paths. llms.txt offers orientation: given that you can read this site, here is what is worth reading.

They are not alternatives. A site with a perfect llms.txt that blocks the retrieval crawlers is still invisible, and access without orientation still works — llms.txt simply makes the job easier.

No major assistant has committed publicly to reading llms.txt, and you should treat anyone claiming guaranteed effects with caution. It is a low-cost bet on a convention that may become useful.

The stronger reason to write one is the exercise itself: deciding which fifteen pages actually explain your product usually improves the site regardless of who reads the file.