Parse robots.txt and test URLs against it, following RFC 9309. A zero-dependency Python package and command-line tool. Python's built-in urllib.robotparser predates the standard and does not understand the * wildcard or the $ end anchor in paths, or longest-match-wins; this does, and it tells you which rule on which line decided each URL. A linter catches the mistakes that quietly break crawling, such as a stray Disallow: / under User-agent: *.
$ python -m robotscheck test tests/data/example.txt /cart /blog/post /admin/public/a "/page?sessionid=1" --agent SomeBot
agent somebot uses the '*' group
BLOCKED /cart Disallow: /cart (line 4)
ALLOWED /blog/post no rule matches
ALLOWED /admin/public/a Allow: /admin/public/ (line 6)
BLOCKED /page?sessionid=1 Disallow: /*?sessionid= (line 5)
$ python -m robotscheck test tests/data/example.txt /cart /private/x /private/press-kit.pdf --agent "Googlebot/2.1"
agent googlebot uses the 'googlebot' group
ALLOWED /cart no rule matches
BLOCKED /private/x Disallow: /private/ (line 10)
ALLOWED /private/press-kit.pdf Allow: /private/press-kit.pdf (line 11)
$ python -m robotscheck lint tests/data/problems.txt
warning rule-outside-group line 1: Disallow before any User-agent line is ignored
warning path-not-slash line 3: Disallow path 'private' should start with / or *
warning unknown-directive line 4: unknown directive 'Disalow' is ignored
info crawl-delay line 5: Google ignores Crawl-delay (Bing and some others honour it)
info noindex line 6: Google stopped supporting Noindex in robots.txt in 2019; use a meta robots tag or X-Robots-Tag header
warning no-colon line 7: not a 'field: value' line, ignored: 'this line has no colon'
warning sitemap-not-absolute line 8: Sitemap must be a full http(s) URL
warning duplicate-group line 9: user-agent '*' has more than one group (rules are merged)
warning blocks-everything line 10: 'User-agent: *' with 'Disallow: /' blocks the whole site for every crawler
3 group(s), 1 sitemap(s), 7 issue(s) to look at
Notice the second example: Googlebot has its own group, so the User-agent: * rules (including Disallow: /cart) do not apply to it at all. That surprises many people, and it is exactly what test shows you. The sample files describe a synthetic site (example.com).
test exits 1 if any URL is blocked, so it can guard a deploy ("the pages that must be crawlable still are"); lint exits 0 unless there are errors (--strict also fails on warnings); 2 means an unreadable file.
No dependencies; Python 3.9 or newer.
pip install git+https://github.com/drlarzlalaa/py-robotscheck
robotscheck lint robots.txt
robotscheck test robots.txt /pricing /blog/post --agent Googlebot
curl -s https://www.example.com/robots.txt | robotscheck test - /admin/
or run it from a checkout with python -m robotscheck.
| Command | What it does |
|---|---|
test FILE URL... [--agent NAME] |
For each URL or path: ALLOWED or BLOCKED, and the deciding rule with its line number. --agent takes a name (Googlebot) or a full User-agent string; the default is a crawler with no group of its own. |
lint FILE [--strict] |
Report likely mistakes with line numbers. |
sitemaps FILE |
List the Sitemap: lines (exit 1 if there are none). |
bots FILE [--policy SPEC] |
How the file treats known AI crawlers, optionally checked against a policy. See below. |
signals FILE [--expect SPEC] |
The Content-Signal and License lines, and what each crawler ends up with. See below. |
As a library:
import robotscheck as rc
robots = rc.parse(open("robots.txt").read())
allowed, rule = rc.check(robots, "Googlebot", "https://example.com/private/x")Most sites now want a policy for AI crawlers (block training, allow the ones that send readers, decide about the rest), and robots.txt is where it is written. bots reads a robots.txt and reports, for each well-known AI crawler, which group applies to it and whether it may fetch your paths. With --policy it becomes a check that exits 1 when the file does not implement what you meant.
$ python -m robotscheck bots tests/data/ai_policy.txt --policy training=block,search=allow,user=allow
AI crawlers against / (crawler list as of 2026-09-26; it changes, see the README)
TRAINING (collects content to train models)
BLOCKED GPTBot OpenAI group 'gptbot' Disallow: / (line 5)
BLOCKED ClaudeBot Anthropic group 'claudebot' Disallow: / (line 5)
ALLOWED anthropic-ai Anthropic group '*' no rule matches
BLOCKED Google-Extended Google group 'google-extended' Disallow: / (line 8)
ALLOWED Applebot-Extended Apple group '*' no rule matches
ALLOWED meta-externalagent Meta group '*' no rule matches
ALLOWED Bytespider ByteDance group '*' no rule matches
BLOCKED CCBot Common Crawl group 'ccbot' Disallow: / (line 5)
SEARCH (builds an index that AI search products cite)
ALLOWED OAI-SearchBot OpenAI group 'oai-searchbot' Allow: / (line 12)
ALLOWED Claude-SearchBot Anthropic group '*' no rule matches
ALLOWED PerplexityBot Perplexity group 'perplexitybot' Allow: / (line 12)
USER-TRIGGERED (fetches a page because a person asked; may ignore robots.txt)
ALLOWED ChatGPT-User OpenAI group '*' no rule matches
ALLOWED Claude-User Anthropic group '*' no rule matches
ALLOWED Perplexity-User Perplexity group '*' no rule matches
OTHER
ALLOWED OAI-AdsBot OpenAI group '*' no rule matches
ALLOWED Amazonbot Amazon group '*' no rule matches
Blocked on every path: training 4 of 8, search 0 of 3, user 0 of 3, other 0 of 2
Policy: training=block, search=allow, user=allow
MISMATCH anthropic-ai (training) is not blocked on: /
MISMATCH Applebot-Extended (training) is not blocked on: /
MISMATCH meta-externalagent (training) is not blocked on: /
MISMATCH Bytespider (training) is not blocked on: /
policy NOT met
Look at the last block: anthropic-ai, Applebot-Extended, meta-externalagent and Bytespider are not blocked, because the file names some training crawlers but not these, so they fall through to the User-agent: * group, which only blocks /admin/ and /cart. That gap is the usual reason a "we block AI training" robots.txt does not.
| Option | What it does |
|---|---|
--path P |
A path to test (repeatable, default /). With several, a crawler that is blocked on some and not others shows as PARTIAL, and a block policy needs it blocked on all of them. |
--policy SPEC |
Desired treatment, category=block or category=allow, comma-separated: training=block,search=allow,user=allow. Categories are training, search, user, other and all. Explicit items win over all, in either order. |
--bots-file F |
Add or correct crawlers: lines of Token,category[,vendor[,note]], # comments allowed. An entry replaces the built-in one with the same token. |
--json |
Machine-readable output, including the list date, each crawler's group and deciding rule, and the policy violations. |
Exit status: 0 report printed (or policy met), 1 policy not met, 2 bad option, policy or file.
The list (16 tokens in robotscheck/aibots.py, dated AS_OF) is data that goes out of date. Categories follow how vendors and reference sources describe them: training crawlers collect content for models, search crawlers build an index that AI search products cite, and user-triggered fetchers retrieve a page because a person asked. On 2026-09-26 the OpenAI and Anthropic entries were checked against those vendors' own pages (OpenAI's bots page and Anthropic's help-centre article), which name GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot and Claude-User. The other vendors' entries (Perplexity, Google, Apple, Meta, ByteDance, Common Crawl, Amazon) come from independent 2026 crawler references, not from those vendors' pages, and anthropic-ai is kept only because it is widespread in existing files. Correct the list with --bots-file if a vendor changes a name.
Things this can and cannot tell you:
- User-triggered fetchers may ignore robots.txt. OpenAI's page says robots.txt rules "may not apply" to
ChatGPT-User,Perplexity-Useris reported not to follow it (a secondary report), while Anthropic statesClaude-Userdoes. ABLOCKEDverdict for those means "the file asks", not "they will comply". The output says so under that heading. Google-ExtendedandApplebot-Extendedare control tokens, not separate crawlers: they are read by Google's and Apple's own crawlers, and govern AI use, not search indexing.- A verdict is about the file, not behaviour. Some crawlers,
Bytespiderin particular, are widely reported to ignorerobots.txt. Blocking there is a request. To enforce, use server or CDN rules (verify the crawler's published IP ranges, do not trust the User-agent string alone). - Crawlers you did not list are governed by
*. That is the RFC 9309 rule, and it is the point of the report.
Two newer lines now appear in real robots.txt files, and older versions of this tool (and most linters) wrongly reported both as unknown directives:
Content-Signal: ai-train=no, search=yes, ai-input=no, deployed by Cloudflare across millions of domains. The three signals are defined by the IETF Internet-Draftdraft-romm-aipref-contentsignals(now expired):search(building a search index and showing results),ai-input(feeding content to an AI model for retrieval-augmented generation) andai-train(training or fine-tuning models). That draft does not define the line's syntax; the syntax used here is the one seen in real files and described by Cloudflare: comma-separatedname=yesorname=nopairs, placed inside aUser-agentgroup. A signal that is not stated means no preference.License: https://example.com/license.xml, from the Really Simple Licensing (RSL 1.0) standard. The value must be an absolute URI. It may appear before the firstUser-agentline (global) or inside a group, and a group'sLicensetakes precedence over a global one.
lint now understands both. signals shows what each crawler actually ends up with:
$ python -m robotscheck signals tests/data/signals.txt --agent '*' --agent GPTBot --agent ClaudeBot --agent Bytespider --expect ai-train=no
Content-Signal and License lines (signal names from the IETF draft; the line syntax is the one Cloudflare deploys)
License (global, line 2): https://example.com/license.xml
group '*' (line 4): search=yes, ai-input=no, ai-train=no
group 'gptbot' (line 9): no signals
group 'claudebot' (line 12): ai-train=no
Effective for each crawler (a named group replaces the '*' group; a signal not stated means no preference, shown as -):
* group '*' search=yes ai-input=no ai-train=no License global
GPTBot group 'gptbot' search=- ai-input=- ai-train=- License global
ClaudeBot group 'claudebot' search=- ai-input=- ai-train=no License global
Bytespider group '*' search=yes ai-input=no ai-train=no License global
Expected: ai-train=no
MISMATCH GPTBot: ai-train=no expected but not stated (its group 'gptbot' has no Content-Signal for it)
signals do NOT match
Look at GPTBot. The file says ai-train=no for *, but GPTBot has its own group (for a Disallow), and a named group replaces the * group entirely, so under normal robots.txt grouping GPTBot sees no signal at all. --expect ai-train=no catches exactly that, and exits 1 when any shown crawler does not state what you expect. By default signals shows * and every crawler in the AI-crawler list; --agent (repeatable) narrows it. A signal that is not stated is shown as -, and never counts as a match.
A second mistake, seen in a real, widely deployed file, is placing a Content-Signal line after another group's rules: a line joins the group above it, so the signal then belongs to that one crawler and everything else that uses * sees none. lint reports that as content-signal-not-for-all. The same applies to License.
$ python -m robotscheck lint problems.txt
warning content-signal-value line 7: signal search has value 'maybe'; expected yes or no
warning content-signal-syntax line 7: 'ai-input' is not name=value
warning license-not-absolute line 8: License must be an absolute URI (RSL): 'license.xml'
info content-signal-not-for-all line 7: the Content-Signal on line 7 belongs to the group for ccbot; crawlers that use 'User-agent: *' see no signal. A line placed after another group's rules joins that group, so move it under 'User-agent: *' if you meant it for everyone
info license-in-group line 8: the License on line 8 belongs to the group for ccbot only; a License before the first User-agent line applies to every crawler
2 group(s), 0 sitemap(s), 3 issue(s) to look at
| Code | Level | Meaning |
|---|---|---|
content-signal-syntax |
warning | Not name=value pairs, or empty. |
content-signal-value |
warning | A value that is not yes or no. |
content-signal-conflict |
warning | The same signal stated as both yes and no in one group. |
content-signal-outside-group |
warning | A Content-Signal before any User-agent line, which applies to no group. |
content-signal-unknown-signal, content-signal-duplicate |
info | A name other than search, ai-input, ai-train (kept, not interpreted); the same signal stated twice with the same value. |
content-signal-not-for-all |
info | Signals sit only in named groups while User-agent: * has none. |
license-not-absolute |
warning | The value is not an absolute URI. |
license-insecure, license-duplicate, license-in-group |
info | The licence is fetched over http; the same licence twice in one scope; a License inside a group, so it applies to that group's crawlers only. |
Limits. These lines are preferences: they do not stop a crawler, and this tool cannot say who honours them. Group selection for Content-Signal is not specified by the draft, so this tool assumes the usual robots.txt behaviour. The no value and the "not stated means no preference" rule come from Cloudflare's own documentation as reported by secondary sources; the real files sampled for this project all used yes. License follows the RSL 1.0 specification (read in full), but the tool never fetches or validates the licence document.
- Which group. A group that names the crawler's product token (case-insensitive) is used; otherwise the
User-agent: *group. SeveralUser-agentlines in a row share one group, and multiple groups for the same agent are merged. Rules from a different group never leak in. - Which rule. Every
AllowandDisallowpattern that matches the URL's path and query is a candidate; the longest pattern wins, and on a tieAllowwins. An emptyDisallow:allows everything; no matching rule means allowed./robots.txtitself is always fetchable. - Patterns. Matching is case-sensitive prefix matching;
*matches any run of characters and a final$anchors the end.Disallow: /*.php$blocks/a.phpbut not/a.php?x=1. - Encoding. Percent-encoding is normalised, so
/caf%C3%A9,/caf%c3%a9and/cafématch each other.
rule-outside-group, path-not-slash, unknown-directive (including typos like Disalow), no-colon, sitemap-not-absolute, duplicate-group, no-user-agent, empty, too-large (over 500 KiB), blocks-everything (User-agent: * with Disallow: /), and informational notes on Crawl-delay, Noindex, Host, Clean-param and Request-rate, which are non-standard or ignored by Google.
- It implements the standard, not every crawler. Google, Bing, Yandex and others each add quirks. In particular, matching here uses the crawler's own product token; some crawlers (Google's image and news bots, for example) also fall back to a parent group such as
googlebot, so pass--agent googlebotto see that group's verdict. - Robots rules are not access control. They ask well-behaved crawlers to stay away; anything listed in
robots.txtis publicly readable. - Blocking is not de-indexing. A URL blocked here can still appear in search results if other pages link to it; use a
noindexmeta tag or header (on a page that is not blocked) to remove it. - Offline. It never fetches
robots.txtitself; give it a file or standard input. - Only
User-agent,Allow,DisallowandSitemapaffect the result; other directives are reported bylintbut not interpreted.
python -m unittest discover -s tests -v
81 tests cover the documented matching examples (prefixes, wildcards, end anchors, longest match, ties), group selection and merging, percent-encoding, BOM and CRLF files, every lint code with line numbers, and the commands with their exit codes. The signals tests cover each rule with matching and non-matching lines; 400 generated robots.txt files written from a known structure (random groups, signals and licences, in single and split Content-Signal lines) that must parse back to exactly that structure; and, for group selection, 300 more where the effective signals are computed independently and compared. The first generator was wrong, not the parser: a group with no line of its own merges into the next group under RFC 9309, and it now always gives each group a line. Four deliberate breakages of the new logic (group choice, licence precedence, conflict versus duplicate, the not-for-all condition) each made tests fail. Against real files, an opt-in test (ROBOTS_REAL_DIR) checks public robots.txt files that use these lines, fetched on 2026-09-26 and not stored in the repository: none produces a false unknown-directive any more, the observed signals and licences read back as expected, and the misplaced-line case is reported. The bots tests check the crawler list is well formed (unique tokens, valid categories, each token survives product_token unchanged), the group rule that surprises people (a named group replaces * entirely), case-insensitive names, several paths, policy parsing and its error cases, both orders of all= against explicit items, the bots-file format, JSON output and exit codes; two deliberate breakages of the code (a wrong "blocked everywhere" test and a wrong policy precedence) were confirmed to make tests fail. CI runs them on Python 3.9 to 3.13.
MIT (LICENSE).