Crawlers and access
Crawl budget
Crawl budget is how much crawling a site receives in a given period. Keeping sitemaps accurate and avoiding duplicate URLs helps crawlers spend that budget on pages you want retrieved.
In more detail
Submitting only the URLs that genuinely exist in a locale, and pointing duplicates at a canonical, both keep the budget pointed at real pages.
Guides that cover this
Related terms
robots.txt
robots.txt is a file at the root of a site that tells crawlers which paths they may fetch. For AI search it is the primary control over whether assistants can reach your pages at all.
Canonical URL
A canonical URL tells search engines which version of a page is the one to index when the same content is reachable at several addresses. It consolidates signals onto a single address instead of splitting them.
Sitemap
A sitemap is an XML file listing the URLs a site wants crawled, with optional hints about change frequency and language alternates. It is how a crawler discovers pages that navigation alone might not surface.
Ready to Dominate AI Search?
Get started now
Allswap