Prettify function

[szepeviktor]

It would be nice to output a nice and valid robots.txt.

[fte378]

This already looks good except that there is no ordering for the length of matching and the match-all clause.

I don't know if that's the case but possibly a parser would match "User-agent: *" if this is not the last entry even if there is a more specific entry like "User-agent: GoogleBot". "User-agent: *" should always be the last entry in the generated file.

The same may hold true for the paths. I doubt that a parser matches "/themes/flowers" if there comes "/themes" first. So the entries within the User-agent blocks should be ordered with the longest matching paths first and the shorter ones later. For example:

User-agent: *
Allow: /themes/flowers/roses
Disallow: /themes/flowers
Allow: /themes

[fte378]

I quickly checked the Google and Yandex specs. They say that more specific paths override more general ones. However there may be other not so smart parsers out there. So the ordering has two use cases: satisfy less smart parsers and make the file better readable for humans.

[szepeviktor]

Implemented.
@fte378 Would you open a PR and solve the disabling of $this->convert('self::prepareRegexRule');?

// Disable
//$this->convert('self::prepareRegexRule');

$robots = $parser->getRules();
$output = array();

// Make match-all clause the last
if ( array_key_exists( '*', $robots ) ) {
    $match_all = $robots['*'];
    unset( $robots['*'] );
    $robots['*'] = $match_all;
}

foreach ( $robots as $ua => $rules ) {
    $output[] = 'User-agent: ' . $ua;
    foreach ( $rules as $name => $value ) {
        // Not multibyte
        $nice_name = ucfirst( $name );
        if ( is_array( $value ) ) {
            // Shorter paths later
            usort( $value, function ( $a, $b ) {
                return mb_strlen( $a ) < mb_strlen( $b );
            } );
            foreach( $value as $elem ) {
                $output[] = $nice_name . ': ' . $elem;
            }
        } else {
            $output[] = $nice_name . ': ' . $value;
        }
    }
    $output[] = '';
}

$sitemaps = $parser->getSitemaps();

foreach ( $sitemaps as $sitemap ) {
    $output[] = 'Sitemap: ' . $sitemap;
}
$output[] = '';

print implode( PHP_EOL, $output );

[fte378]

@szepeviktor I'm only an end user of your Wordpress plugin :) I know what a pull request might be but I don't have anything to do with this library.

[szepeviktor]

@t1gor @JanPetterMG Could you add this as a new function and solve the disabling of $this->convert('self::prepareRegexRule');?

[JanPetterMG]

I'm quite busy right now, but give me a day or two and I'll come back to this. Stay tuned.

[szepeviktor]

OK!

[JanPetterMG]

@fte378 Just want to hear your opinion, why does the user-agents really have to be sorted Z-A (witch normally resulting in * as the last one)?
I know there is a lot of pros/cons here, but if parsed by this library, it shouldn't make any difference anyway. Do you plan to use this library to render an existing robots.txt and then publish it? or in other ways use the rendered result outside of this library?

Only one group of group-member records is valid for a particular crawler. The crawler must determine the correct group of records by finding the group with the most specific user-agent that still matches. All other groups of records are ignored by the crawler. The user-agent is non-case-sensitive. All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot). The order of the groups within the robots.txt file is irrelevant.

Source: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt#order-of-precedence-for-user-agents

I know the UA matching code in this library has a lot of bugs, but at least it's selecting the correct UA in most cases. I'm probably fixing that one within a couple of days...

[szepeviktor]

It is going into my plugin: https://wordpress.org/plugins/multipart-robotstxt-editor/

[szepeviktor]

*Added sitemaps.

[szepeviktor]

@JanPetterMG Could you guide me to disable $this->convert('self::prepareRegexRule');?

[JanPetterMG]

@szepeviktor sorry for the delay, I'll take a look at it

[szepeviktor]

Thank you for your help.
I appreciate it.