Is Your Sitemap Real? Detect HTML Catch-Alls and Canonical Errors

Fetch the canonical endpoint without relying on a browser tab. Record final URL after redirects, status, Content-Type, and body. A real feed should parse as XML and contain a supported urlset or sitemapindex root in the sitemap namespace. text/html, a doctype, an application mount element, or marketing copy indicates a catch-all or middleware problem even when the status is 200. Also test the apex and www forms so a redirect does not send the file through a generic page rule. [AR2][AR11]

Configure explicit sitemap and robots routes before broad application rewrites. Framework-native generators are useful when they use the same content registry as page generation. In a static deployment, produce the XML file during the build and set the correct media type. Add a contract test that fails on HTML markers, XML parse errors, wrong host, and missing expected URLs. [AR1][AR2][AR7]

For each loc, fetch the page and capture final URL, status, media type, canonical tag, robots meta, title, and meaningful body. The URL should not redirect, soft-404, require authentication, or canonicalize elsewhere. Check that the preferred scheme, host, path casing, and trailing-slash policy are consistent. A sitemap should list the canonical result directly rather than relying on crawlers to resolve conflicting signals. [AR2][AR5]

Keep private dashboards, search results, parameter duplicates, preview hosts, error routes, and noindex pages out. Ensure important public pages are not orphaned from navigation merely because they appear in XML. A sitemap helps discovery but does not replace ordinary internal links or useful content. For ten articles and a few trust pages, exhaustive checking is inexpensive and gives much stronger evidence than a sample. [AR1][AR2][AR11]

The optional lastmod value describes the page's last modification, not the time the sitemap was generated. Store publication and material modification dates in the content source and emit them deliberately. Do not stamp every URL with the build time, because unchanged pages then appear newly modified on every deployment. Update the date when the substantive text, data, product behavior, or source review changes; cosmetic layout or unrelated build output may not warrant it. [AR2]

Keep visible article dates and structured data aligned with the sitemap. If an evergreen page has no trustworthy modification record, omit lastmod rather than inventing precision. A deterministic test can compare content metadata with sitemap output. Editorial review should confirm that a changed date corresponds to real work, including rechecking time-sensitive sources and CTA truth. [AR2][AR5][AR7]

Declare the absolute canonical sitemap URL in robots.txt and verify that robots itself returns text rather than the app shell. Review crawl rules affecting listed public paths. Then submit the same canonical feed through the relevant search property only after production fetches prove the XML and pages are live. Submission is a discovery signal, not a promise of indexation, and provider status must be recorded separately from local or preview success. [AR1][AR2]

After deployment, rerun the complete crawl because platform redirects, headers, and middleware can differ from a local server. Preserve the timestamp and response evidence. Watch for new route types and update the generator and test together. A passing sitemap test is strongest when it proves the inventory derives from the same source as the pages rather than comparing two hand-maintained lists. [AR1][AR2][AR7]