Crawlers and access
Sitemap
A sitemap is an XML file listing the URLs a site wants crawled, with optional hints about change frequency and language alternates. It is how a crawler discovers pages that navigation alone might not surface.
In more detail
For a multilingual site the sitemap should list only the locales a page genuinely exists in, and carry matching hreflang alternates, so the language cluster stays valid.
Guides that cover this
Related terms
Crawl budget
Crawl budget is how much crawling a site receives in a given period. Keeping sitemaps accurate and avoiding duplicate URLs helps crawlers spend that budget on pages you want retrieved.
hreflang
hreflang is markup that tells search engines which language and region each version of a page serves. A valid cluster names every version including itself, so readers are sent to the right one.
Canonical URL
A canonical URL tells search engines which version of a page is the one to index when the same content is reachable at several addresses. It consolidates signals onto a single address instead of splitting them.
Ready to Dominate AI Search?
Get started now
Allswap