A maintained, source-cited list of the bots that read the web on behalf of AI answer engines, and whether allowing each one helps or hurts your AI visibility.
Maintained by Visibility Mesh.
Store owners and developers keep asking the same question: which of these AI bots should I allow, and which are safe to block? The answer is not obvious, because the bots do different jobs:
- training bots build model training data. Blocking them is a legitimate content choice and does not stop live citation.
- retrieval bots fetch your page live to answer a user's question. This is the traffic you most want.
- indexing bots build the search index behind an answer engine. Allow them to be eligible for citation.
- opt-out-token entries are robots.txt tokens, not crawlers (for example
Google-Extended).
This registry records each bot's operator, user-agent strings, job type, whether it respects robots.txt, a plain-language AI-visibility recommendation, and a source URL for every entry.
The data is one file: crawlers.json.
const registry = await fetch(
'https://raw.githubusercontent.com/VisibilityMesh/ai-crawler-registry/main/crawlers.json'
).then((r) => r.json());Each entry:
{
"name": "PerplexityBot",
"operator": "Perplexity",
"userAgents": ["PerplexityBot"],
"type": "indexing",
"respectsRobotsTxt": true,
"aiVisibility": "allow-recommended",
"rationale": "Indexes pages for Perplexity answers. Allow it to be eligible for Perplexity citations.",
"source": "https://docs.perplexity.ai/guides/bots",
"confidence": "confident"
}confidence is confident when the entry was checked against the operator's official documentation on lastReviewed, or verify when it is lower confidence and welcomes a correction.
Use @visibilitymesh/crawler-check, which reads this registry and tells you which AI crawlers your robots.txt allows or blocks.
User-agent strings and robots tokens change. If you have a documented correction or a new AI crawler with an official source, open a pull request that updates crawlers.json and includes the source URL. Entries without a verifiable source are not merged.
Maintained by Visibility Mesh, the AI commerce visibility platform for Shopify.
- Website: https://visibilitymesh.com
- Shopify App Store: https://apps.shopify.com/visibility-mesh
- All open tools: https://github.com/VisibilityMesh
- Run it against your site: @visibilitymesh/crawler-check
Petrovic, Margareta. AI Crawler Registry. Visibility Mesh, 2026. https://github.com/VisibilityMesh/ai-crawler-registry
Machine-readable citation metadata is in CITATION.cff.
Data is licensed CC BY 4.0. You may use and redistribute it, including commercially, with attribution to Visibility Mesh (https://visibilitymesh.com).