robots.txt, Sitemap, llms.txt, and ai-agent.json: What Each File Proves

robots.txt lives at the origin root and expresses crawler-specific allow or disallow rules according to documented parsing behavior. It can also declare one or more sitemap locations. Test the exact canonical origin, final response, text body, and groups that apply to the crawler you care about. A missing file, a broad Disallow rule, and a file served through an HTML catch-all are three different findings. The file is public and advisory; it does not authenticate a requester or protect private content. Sensitive routes need real access control regardless of what robots.txt says. [AR1]

Keep the file small and generated from deliberate policy. Comments are useful for maintainers, but never place tokens, private paths, internal hostnames, or debugging notes there. When you name an agent-specific group, document why it differs from the general policy and review that decision as product and legal requirements change. Confirm that the sitemap line uses the public canonical URL and that deployment routing returns text rather than the application shell. [AR1][AR7]

A sitemap supplies preferred public URLs and optional metadata in a defined XML vocabulary. It helps discovery but does not guarantee crawling or indexing. Every entry should resolve to useful indexable content on the canonical host. Sample the URLs for redirects, soft 404s, login walls, inconsistent canonical tags, and HTML shells. Use lastmod only for material content changes; a generated current timestamp on every build creates noise rather than freshness evidence. [AR2]

For a small site, validate every URL. Larger inventories can combine deterministic generation with representative live checks. Keep search results, duplicate parameter forms, account pages, and noncanonical hosts out. If you split a large inventory, use a sitemap index and keep each child feed internally consistent. Finally, link the sitemap from robots.txt and submit the same canonical endpoint to search tools; conflicting URL sets make diagnosis harder. [AR1][AR2]

An llms-oriented text file can point readers or tools toward concise documentation, while an ai-agent-style JSON document can describe product-specific capabilities. Neither name alone establishes a universal contract. Google says llms.txt is ignored by Google Search and neither helps nor harms visibility or rankings there. Publish the file only for readers or tools that choose to consume it, and only when you can explain the intended consumer, owner, update process, and limitations. [AR7][AR11][AR13]

Validate every referenced URL and make capability language conservative. A manifest should not claim that an operation is safe, available, or unauthenticated unless production enforces exactly that behavior. Do not include secrets or internal implementation details. Prefer generation from the same registry that powers documentation so routes do not drift. A regression test should parse the payload, follow links, and compare the declared version or operation set with the deployed service. [AR6][AR7]

For example.com, /robots.txt can say which public paths a crawler may request and point to https://example.com/sitemap.xml. The sitemap can enumerate canonical product, pricing, documentation, and policy URLs with truthful last modification dates. /llms.txt can provide a concise human-readable index to those maintained pages for voluntary consumers. /.well-known/ai-agent.json can describe product-specific discovery metadata. The four responses should agree on the canonical host, but each has a different parser, maintenance owner, and proof boundary. [AR1][AR2][AR7][AR13]

Test each route as a deployment artifact. A 200 response is insufficient if /llms.txt returns the HTML homepage, if sitemap entries redirect to a different host, or if the manifest advertises an operation production does not support. Then test the customer journey separately: finding pricing, preparing signup, or calling one harmless documented API operation. This distinguishes discoverability from operability and keeps an emerging file from becoming a substitute for product evidence. [AR1][AR2][AR6][AR7]

OpenAPI can define paths, operations, inputs, outputs, security schemes, and servers in a machine-readable document. That makes it more formal than a prose hint, but the document can still be incomplete or stale. Validate it with a standards-aware tool, resolve references, and compare harmless operations against the actual service. Authentication requirements, error responses, idempotency expectations, rate limits, and consequence should be visible to users even when they are not all expressible in one schema. [AR6]

Treat the specification as one layer of an authorization system, not the authorization system itself. The server must validate identity, permission, targets, and state on every request. Exclude secret example values and avoid publishing internal-only servers accidentally. Version changes deliberately, retain a correction path, and rerun contract checks after deployment. Pair the API description with browser-task tests when the product also exposes a graphical interface. [AR6][AR7]

First repair canonical routing and useful HTML. Second publish a real sitemap and a deliberate robots policy. Third add formal API documentation when a public API exists. Only then add experimental orientation files where they serve a named workflow. This order protects against spending time on fashionable files while primary pages remain uncrawlable or primary controls remain ambiguous. Keep a table of artifact, governing standard or convention, owner, generated source, verification command, and last material review. [AR1][AR2][AR6][AR7]

After publication, test user outcomes. Ask an agent or deterministic browser to find a page, cite a fact, navigate to the free scan, and stop before an external consequence. Record failures and repair the underlying information architecture, semantics, or safety boundary. Discovery artifacts should shorten a reliable path; they should never substitute for the path itself. AgentReady's free scan can flag several protocol and crawl signals, but confirm every result against the official specifications and your production behavior. [AR1][AR2][AR11]