Robots.txt Says Yes, Your WAF Says No: Why AI Agents Still Cannot Reach Your Site

Google's robots.txt specification defines how crawlers select user-agent groups and apply allow and disallow rules to URL paths. It is a convention for crawler behavior, not an authentication protocol, authorization system, or substitute for access control. The file is public, user-agent strings can be imitated, and protected content still needs server-side authentication. A rule can express that a cooperative crawler should not fetch a path; it cannot prove who sent a request or stop a non-compliant client from attempting the URL.¹

Robots evaluation is also hostname-specific. Teams commonly update the apex domain while leaving a www host, documentation host, storefront host, or regional host with a different file. Redirects, error responses, HTML fallbacks, and cached versions can make the effective policy differ from the text in a repository. A valid test must fetch the live file on every relevant hostname, preserve status and redirect evidence, parse the matching group, and then test an allowed and disallowed path. Merely finding the string robots.txt in source control is not production evidence.¹

Cloudflare's AI Crawl Control provides visibility into AI-service traffic and lets site operators set granular allow or block policies, monitor robots compliance, and create enforcement rules. Its documentation explicitly separates crawler categories and notes that unsuccessful requests can result from multiple rules or response failures, not only from an AI Crawl Control block. This is why a clean robots policy can coexist with a 403, challenge page, rate limit, or connection failure at the edge. The policy layer and enforcement layer need separate evidence.^{2, 3}

Cloudflare's current bot reference groups relevant traffic as AI Search, AI Assistant, and AI Crawler. That distinction is more useful than a single allow AI toggle because a publisher may welcome user-requested retrieval and citations while declining other crawling, or permit a signed browser agent on public product pages while protecting account and admin paths. The correct rule depends on business intent and verified request characteristics. Blanket exceptions based only on a user-agent substring are fragile because any client can send that text.^{2, 3}

OpenAI documents that ChatGPT Work's cloud browser signs outbound HTTP requests using Web Bot Auth and HTTP Message Signatures. Requests include signature headers and identify the signature agent as chatgpt.com; a site or supporting CDN can retrieve the published keys and verify that the request originated from ChatGPT before applying an allow rule. This is materially stronger than trusting a header that merely says ChatGPT. It lets an operator preserve normal WAF controls while making a narrow exception for authenticated agent traffic.⁴

Do not generalize that mechanism to providers that have not documented an equivalent identity. xAI's official Grok Bot launch says Bots have their own computer, can sign into existing tools, and work across apps and websites, including products without an API or MCP surface. It does not, in the cited launch materials, publish a robots user-agent, address range, or request-signature directory for website operators. Therefore a site should not invent a GrokBot robots group or WAF allowlist from the product name alone. Test the browser behavior you are authorized to observe and wait for provider-specific identity guidance before creating a privileged exception.^{5, 6}

Use four practical classes. An indexing or training crawler performs repeated server-side fetches to build a corpus. A search or user fetcher retrieves public pages in response to a query and may use a distinct identity. A browser agent loads a rendered application, executes JavaScript, stores cookies, and may act in a user's session. A structured site tool exposes a named operation through an API-like contract in the browser or through MCP. One provider may operate more than one class with different identifiers. Policy should attach to the class and purpose, not just the company name.^{2, 7, 3}

Then map paths by consequence. Public articles, product pages, pricing, policies, and documentation may be appropriate for discovery and user-delegated browsing. Login, callback, checkout, and account pages may need browser access but should never be opened to unauthenticated crawlers. Administrative, export, deletion, and secret-bearing endpoints require application authorization regardless of bot policy. This path matrix prevents a common failure: solving a blocked public agent by skipping the WAF for every request carrying a fashionable user-agent string.^{1, 4}

As reviewed on August 30, 2026, AgentReady's current public diagnostic includes stable checks for crawler and agent access policy and for bot access. It fetches live public artifacts, records response truth, and can add browser evidence about edge blocking or agent user-agent probes. It keeps unobservable evidence separate from failure and can recommend a concrete Fix Pack item. That is enough to catch missing or malformed policy, obvious blanket blocks, and disagreement between a normal fetch and a browser pass.⁸

The current scanner does not prove that an unverified request is genuinely from every named provider. It does not operate provider accounts for Grok, Claude, ChatGPT Work, or Gemini, and it does not currently run a signed Web Bot Auth validation exchange on the site owner's edge. Its enhanced browser uses an AgentReady user agent and aborts observed non-GET and non-HEAD requests; that guard cannot guarantee safety on a server incorrectly implemented to mutate on GET. The report should therefore say what AgentReady observed—status, content, challenge, redirect, or browser result—without claiming universal access. The proposed matrix below is the additional evidence needed for provider-specific conclusions.⁸

Run the matrix against a controlled set of public paths: robots.txt, sitemap, one article, one product or pricing page, one policy page, and the login entry point without credentials. Include an ordinary browser control, a plain HTTP control, the documented crawler or signed-agent identity where officially available, and the actual user-facing browser product where authorized. Record DNS and TLS success, status, redirect chain, content type, response body classification, challenge presence, rendered heading, cookie requirements, and time. A 200 challenge page is not success; verify useful content and the expected final URL.^{1, 3, 4}

Change one layer at a time. First preserve current rules and establish the failure. Then adjust robots policy if the intended crawler is disallowed. If policy already allows the path, inspect CDN and WAF logs before creating an exception. Prefer verified signatures or provider detection IDs over user-agent matching. Scope exceptions to relevant hostnames, methods, and public paths; keep application authentication intact. Repeat the same matrix after deployment and retain both results. For broader product comparisons, use the Grok, Claude, ChatGPT, and Gemini test blueprint, which reports each harness separately instead of collapsing them into one AI access result.^{2, 4}

The available AgentReady public diagnostic already provides the first layer: live robots and bot-access observations, browser evidence when available, and prioritized fixes. The available Agent Access Matrix projects those observations into separate rows for declared crawler policy, simulated agent user-agent responses, server-rendered content, browser stability, CAPTCHA friction, and security evidence. It labels the user-agent evidence simulated and leaves provider identity unverified, so a green robots row cannot hide an edge or rendering failure. Hostname discovery, WAF correlation, cryptographic signed-agent verification, and real provider-session results still require separate authorized checks.

The matrix connects naturally to Is your site usable by Grok Bot?, Claude in Chrome website readiness, and agent readiness score versus journey success. Aggregate research could contribute consent-controlled cohorts under the published leaderboard methodology, while exact dated examples belong in the Agentic Customer Journey Index. The output should never publish sensitive WAF rules, raw signatures, account pages, or unconsented company rankings.