Google robotstxt: Reference parser/matcher robots.txt, близкий к поведению Googlebot.
-
Updated
Jun 9, 2026
#
robots-checker
Here are 2 public repositories matching this topic...
Validate robots.txt from the command line and test whether URLs are allowed for Googlebot, Bingbot or any user agent, with RFC 9309 matching (wildcards, $, longest match). Flags SEO mistakes like blocked CSS/JS or Disallow: /. Zero-dependency Node.js CLI, library and GitHub Action. Free online version by PromoPilot.
nodejs cli crawler seo web-crawler robots-txt googlebot seo-tools github-action technical-seo robots-txt-parser rfc9309 robots-checker promopilot
-
Updated
Oct 27, 2026 - JavaScript
Add this topic to your repo
To associate your repository with the robots-checker topic, visit your repo's landing page and select "manage topics."