Parse and build robots.txt files in Rust per RFC 9309, with crawl-delay and sitemap extensions.
-
Updated
Oct 28, 2026 - Rust
#
rfc9309
Here are 19 public repositories matching this topic...
An explainable RFC 9309 robots.txt engine for MoonBit
-
Updated
Sep 18, 2026 - JavaScript
Check which AI crawlers a site's robots.txt allows or blocks, and which line decided it. Zero dependencies.
-
Updated
Nov 8, 2026 - JavaScript
Validate robots.txt from the command line and test whether URLs are allowed for Googlebot, Bingbot or any user agent, with RFC 9309 matching (wildcards, $, longest match). Flags SEO mistakes like blocked CSS/JS or Disallow: /. Zero-dependency Node.js CLI, library and GitHub Action. Free online version by PromoPilot.
nodejs cli crawler seo web-crawler robots-txt googlebot seo-tools github-action technical-seo robots-txt-parser rfc9309 robots-checker promopilot
-
Updated
Oct 27, 2026 - JavaScript
Paste a robots.txt and see which AI crawlers can actually read your site, which rule decided it, and whether they fell back to the wildcard group. Implements RFC 9309 precedence.
-
Updated
Aug 5, 2026 - HTML
Free, open-source robots.txt tester — Google's retired tester rebuilt with RFC 9309 matching. Test any URL against Googlebot, GPTBot, ClaudeBot, PerplexityBot and more, with a one-click all-bots matrix.
-
Updated
Oct 16, 2026 - HTML
A small, fast, zero-dependency robots.txt parser and matcher for TypeScript.
-
Updated
Oct 31, 2026 - TypeScript
Pure Standard ML robots.txt parser and matcher (RFC 9309) — dependency-free, MLton + Poly/ML
-
Updated
Jul 11, 2026 - Standard ML
robots.txt (RFC 9309) parser for Julia — a port of robotstxt-spec
-
Updated
Oct 7, 2026 - Julia
Language-neutral spec and conformance fixtures for robots.txt (RFC 9309)
-
Updated
Oct 7, 2026 - Python
robots.txt (RFC 9309) parser for Swift — a port of robotstxt-spec
-
Updated
Oct 7, 2026 - Swift
robots.txt matching the way RFC 9309 and Google define it: most specific rule wins, * and $ wildcards. Fixes urllib.robotparser first-match bug. No dependencies.
-
Updated
Sep 29, 2026 - Python
Check which AI and search crawlers a robots.txt allows for a path (RFC 9309). Browser + CLI, no dependencies.
-
Updated
Oct 8, 2026 - JavaScript
Read-only mirror of gitlab.com/bart-turczynski/robotstxtr
crawler r seo robots-txt rstats googlebot r-package web-crawling robots-exclusion-protocol technical-seo rfc9309
-
Updated
Nov 8, 2026 - R
robots.txt (RFC 9309) parser for Zig — a port of robotstxt-spec
-
Updated
Oct 7, 2026 - Zig
Test your robots.txt like code: RFC 9309 matching, a crawler access matrix, and CI policy checks.
-
Updated
Oct 20, 2026 - Python
robots.txt parser test cases with verdicts sourced from RFC 9309 and Google's docs
-
Updated
Oct 22, 2026
Add this topic to your repo
To associate your repository with the rfc9309 topic, visit your repo's landing page and select "manage topics."