Published free diagnostic
AI crawler access checker
Check what robots.txt declares and what the submitted public page actually delivers through redirects, headers, and HTML—without confusing that evidence with named-bot runtime access.
What the result proves
A dated interpretation of the origin's robots.txt for the submitted path plus the status, redirect shape, X-Robots-Tag, and bounded HTML meta robots delivered to the AgentReady checker.
What remains unobservable
Named-bot edge and WAF treatment is deliberately reported as unobservable. This fetch does not prove crawling, indexing, citation, recommendation, JavaScript execution, or future availability.
Questions
Does an allow rule prove an AI crawler can reach my page?
No. robots.txt is declared policy. CDN, WAF, authentication, JavaScript, geography, and runtime failures are separate evidence surfaces, and this release does not impersonate crawler user agents against the page.
Does blocking a training crawler also block AI search?
Not necessarily. Operators publish different tokens for search, assistant retrieval, and model training. This tool reports those purposes separately when the maintained token registry distinguishes them.