Skip to content

About

A maintained, source-cited registry of AI answer-engine crawlers (GPTBot, ClaudeBot, PerplexityBot and more) and whether allowing each helps your AI visibility.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

AI Crawler Registry

A maintained, source-cited list of the bots that read the web on behalf of AI answer engines, and whether allowing each one helps or hurts your AI visibility.

Maintained by Visibility Mesh.

Why this exists

Store owners and developers keep asking the same question: which of these AI bots should I allow, and which are safe to block? The answer is not obvious, because the bots do different jobs:

  • training bots build model training data. Blocking them is a legitimate content choice and does not stop live citation.
  • retrieval bots fetch your page live to answer a user's question. This is the traffic you most want.
  • indexing bots build the search index behind an answer engine. Allow them to be eligible for citation.
  • opt-out-token entries are robots.txt tokens, not crawlers (for example Google-Extended).

This registry records each bot's operator, user-agent strings, job type, whether it respects robots.txt, a plain-language AI-visibility recommendation, and a source URL for every entry.

Use it

The data is one file: crawlers.json.

const registry = await fetch(
  'https://raw.githubusercontent.com/VisibilityMesh/ai-crawler-registry/main/crawlers.json'
).then((r) => r.json());

Each entry:

{
  "name": "PerplexityBot",
  "operator": "Perplexity",
  "userAgents": ["PerplexityBot"],
  "type": "indexing",
  "respectsRobotsTxt": true,
  "aiVisibility": "allow-recommended",
  "rationale": "Indexes pages for Perplexity answers. Allow it to be eligible for Perplexity citations.",
  "source": "https://docs.perplexity.ai/guides/bots",
  "confidence": "confident"
}

confidence is confident when the entry was checked against the operator's official documentation on lastReviewed, or verify when it is lower confidence and welcomes a correction.

Want to run this against your own site?

Use @visibilitymesh/crawler-check, which reads this registry and tells you which AI crawlers your robots.txt allows or blocks.

Contributing

User-agent strings and robots tokens change. If you have a documented correction or a new AI crawler with an official source, open a pull request that updates crawlers.json and includes the source URL. Entries without a verifiable source are not merged.

Part of Visibility Mesh

Maintained by Visibility Mesh, the AI commerce visibility platform for Shopify.

Cite this

Petrovic, Margareta. AI Crawler Registry. Visibility Mesh, 2026. https://github.com/VisibilityMesh/ai-crawler-registry

Machine-readable citation metadata is in CITATION.cff.

License

Data is licensed CC BY 4.0. You may use and redistribute it, including commercially, with attribution to Visibility Mesh (https://visibilitymesh.com).

About

A maintained, source-cited registry of AI answer-engine crawlers (GPTBot, ClaudeBot, PerplexityBot and more) and whether allowing each helps your AI visibility.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages