Support for Content-signal directive

[3x10RaiseTo8]

Feature Request: Support for Content-signal directive

The Content-signal directive works by signaling your preference of AI actions.

Example

robots.txt taken from https://contentsignals.org/

# As a condition of accessing this website, you agree to abide by
# the following content signals:

# (a)  If a content-signal = yes, you may collect content for the
# corresponding use.
# (b)  If a content-signal = no, you may not collect content for
# the corresponding use.
# (c)  If the website operator does not include a content signal
# for a corresponding use, the website operator neither grants nor
# restricts permission via content signal with respect to the
# corresponding use.

# The content signals and their meanings are:

# search: building a search index and providing search results
# (e.g., returning hyperlinks and short excerpts from your
# website's contents).  Search does not include providing
# AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g.,
# retrieval augmented generation, grounding, or other real-time
# taking of content for generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS
# RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION
# DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE
# DIGITAL SINGLE MARKET.

User-Agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yes
Allow: /

Arguments

1. The directive is non-standard. Yes, but it's extremely helpful as a site owner to communicate intent to crawlers. Very relevant to current AI era. Drafting RFC, standardizing in spec and then implementing it might be a safer bet.

[fili]

A few thoughts, in no particular order:

• This repo is for the parser/tester, not the RFC (where this should be first tackled).
• There is no need to add a non-standard directive, the RFC does not exclude non-standard directives but rather includes standard directives. This distinction is important because for example the widely used Crawl-Delay directive is not even considered standard, but OK to use.
• There are so many other similar initiatives out there, e.g. https://github.com/creativecommons/cc-signals and https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ and https://datatracker.ietf.org/doc/draft-madhavan-aipref-displaybasedpref/ and many more. So before adding a non-standard directive which is not even accepted as RFC to this codebase as a standard directive, let's first see which initiative gets the most adoption. And even then, like Crawl-Delay it may not be needed to add it explicitly.

Hope this helps.

[3x10RaiseTo8]

Thanks for helpful comments, @fili. It broaden my perspective. A few more thoughts.

1. Indeed this must be tackled at RFC level first. Gary Illyes committed a copy of draft RFC related to "Robots Exclusion Protocol Extension for URI Level Control".

   While there many ways people have tried to solve the overarching issue of "preferences of AI actions", eventually one or a combination of many will resolve to be a standard. This repo is hotspot for people who can take actions concerning the issue. Since there is a need and no consensus, this Github issue serves as a TODO that an RFC is needed that tackles at robots.txt level.

2. Non-standard directives like Crawl-Delay which Google does not use but is common is listed here in the code.

   robotstxt/reporting_robots.cc

   Lines 11 to 19 in 86d5836

   	// The kUnsupportedTags tags are popular tags in robots.txt files, but Google
   	// doesn't use them for anything. Other search engines may, however, so we
   	// parse them out so users of the library can highlight them for their own
   	// users if they so wish.
   	// These are different from the "unknown" tags, since we know that these may
   	// have some use cases; to the best of our knowledge other tags we find, don't.
   	// (for example, "unicorn" from "unicorn: /value")
   	static const std::vector<std::string> kUnsupportedTags = {
   	"clean-param", "crawl-delay", "host", "noarchive", "noindex", "nofollow"};

   While functionally it makes no difference, the users of this parser (such as Googlebot) interpret Unknown tags/directives and Unsupported tags differently.

   

   While this isn't code for googlebot, the list of Unsupported tags were last updated on May 22, 2024, ~ 2 years ago. There is scope to update the list again.

[fili]

I appreciate your thoughts.

I would argue that the IETF workgroups is where this discussion is held. That having said, I am fine discussing it here as well but this is a much larger discussion than "just adopt this one since it is missing".

Regarding the screenshots, these are from Google Search Console. Although Google uses a close-variation of this particular open sourced tool, it is not exactly the same on how it is used in Google Search Console with the highlighted warning and errors. In addition, the warnings and errors are, as you also stated, specific for Googlebot. So even if your request would be integrated into the RFC for robots.txt and this tool, the warnings and errors would likely still show in Google Search Console for Googlebot. To change that you should submit feedback through the Google Search Console Feedback channel, or Search Central' Google Groups.

As I mentioned before, your request is already supported because it is not explicitly blocked. This particular paragraph highlights that: https://www.rfc-editor.org/rfc/rfc9309.html#name-other-records. We don't need to solely rely on kUnsupportedTags to parse and/or recognize additional directives.

That having said, you could submit a pull request to update kUnsupportedTags.

In my humble opinion: the RFC, this tool and Google Search Console on the mentioned highlights work as intended. No change needed at this point, but feel free to submit a pull request.

Hope this helps.

[garyillyes]

Thanks for bringing this up. It's a little complicated on our side to merge PRs that half-address a larger issue (this library is used in production systems), so just merged a PR that adds this and a couple other popular rules.

Your PR was great @3x10RaiseTo8 and thank you for sending it our way, we just had to expand it such that we're not arbitrarily adding random tags. For this, we made a change to to the HTTPArchive custom metrics to collect the stats that enable us to better track popular rules and add them as necessary.

Thanks again for raising this!