[wp-trac] [WordPress Trac] #35819: Robots.txt exclusion rule order reversal

[WordPress Trac]

#35819: Robots.txt exclusion rule order reversal
-------------------------+------------------------------
 Reporter:  rdela        |       Owner:
     Type:  enhancement  |      Status:  new
 Priority:  normal       |   Milestone:  Awaiting Review
Component:  General      |     Version:  4.4.2
 Severity:  normal       |  Resolution:
 Keywords:               |     Focuses:  administration
-------------------------+------------------------------

Comment (by dmchale):

 According to [https://developers.google.com/webmasters/control-crawl-
 index/docs/robots_txt Goolge's robots.txt specifications page] ...
 (emphasis mine)

    At a group-member level, in particular for allow and disallow
 directives, '''the most specific rule based on the length of the [path]
 entry will trump the less specific (shorter) rule'''. The order of
 precedence for rules with wildcards is undefined.

 Therefore `Allow: /wp-admin/admin-ajax.php` will trump `Disallow: /wp-
 admin/`

 Personally, before making any changes I would also like to see other
 evidence which corroborates the
 [https://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive
 Wikipedia article]'s assertion that `by standard implementation the first
 matching robots.txt pattern always wins`. There is no citation on that
 claim and I'd love to see something more concrete than one editor's choice
 of phrasing.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/35819#comment:1>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform