Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
-
Updated
May 10, 2026 - Python
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
Does your robots.txt do what you think for AI bots? Finds pages you block for everyone that ChatGPT, Claude or Perplexity can still read.
Prove which AI fetched your page: secret canary links, a Cloudflare Worker log and published-IP checks. Tested on ChatGPT, Claude, Gemini, Grok, Perplexity and Copilot (Oct 2026).
User-agent simulator for major LLM crawlers: measures response code, content length, and JS-dependency
Search Atlas review 2026: run one script and see whether OTTO SEO's changes reach Google and AI crawlers. No account needed. Sourced pricing and limits.
"The AI Discoverability Standard. A structured hub for businesses and products to become understandable, verified, and citable by modern LLM crawlers.
To associate your repository with the llm-crawlers topic, visit your repo's landing page and select "manage topics."