Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
-
Updated
Oct 29, 2026 - TypeScript
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
Eight SEO and GEO agent skills for WordPress: AI crawler log forensics, page citability scoring, brand visibility in AI answers, robots.txt and llms.txt governance, migration redirects, SEO plugin driving, and agency portfolio reporting. Every analysis ends in a self-contained HTML report.
The best AEO/GEO skill for Claude Code. Audit, fix, and monitor your website's visibility across ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews. 100+ optimization techniques, 33 evidence collectors, 4-vector GEO Score. Built on Princeton KDD 2024 paper. Free, MIT.
See which AI crawlers can read your site — GPTBot, ClaudeBot, PerplexityBot & 20 more. Curated, operator-sourced bot list + a zero-dep CLI and GitHub Action to audit robots.txt, test reachability, read access logs, and gate it all in CI.
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
Agent-native multi-bot web crawler simulator. See your site through the eyes of Googlebot, GPTBot, ClaudeBot, and PerplexityBot.
Visual robots.txt manager for the AI era. Block or allow GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers with one click.
Four free MCP tools to check whether AI engines can find, read and recognise a website. No account, no API key, no LLM calls.
Record which AI crawlers read your pages. Forwards GPTBot, ClaudeBot, PerplexityBot and friends to rankveo, and nothing else.
GitHub Action: fail the build when robots.txt or a firewall blocks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot. Checks llms.txt. Zero dependencies.
User-agent simulator for major LLM crawlers: measures response code, content length, and JS-dependency
AI Observatory — verified AI and search crawler retrieval monitoring for business websites
Check whether AI crawlers can access a URL (robots.txt) plus page legibility signals. Zero dependency CLI and library.
Free no-tracking browser utility for drafting and auditing AI crawler robots.txt rules.
Best AEO tracking tools compared by where they measure, plus Python to find AI crawler and citation gaps in your access logs.
robots.txt for AI crawlers: copy-paste templates that allow AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and let you block AI training (GPTBot, ClaudeBot, Google-Extended), a verified AI crawler list, an llms.txt template and access tests. For generative engine optimization (GEO) and AI SEO.
21 AI crawler user-agents we detect, and which 14 of them actually reached one real site over 74 days. First-party detection list plus first-party access logs. Not observed is written "not observed", never 0.
Robots.txt para IA: os robôs do ChatGPT, Claude, Gemini, Perplexity e dos buscadores (GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot) conferidos na documentação oficial, com modelos prontos de robots.txt e llms.txt. Pela TRIWI.
See which search and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot…) your firewall or CDN turns away, even when robots.txt allows them. CLI + GitHub Action. By CMO Owls.
Open directory of AI crawlers and agents, bot verification, and the SenseGap collectors. MIT.
To associate your repository with the perplexitybot topic, visit your repo's landing page and select "manage topics."