[wp-trac] [WordPress Trac] #35819: Robots.txt exclusion rule order reversal

[WordPress Trac]

#35819: Robots.txt exclusion rule order reversal
----------------------------+-----------------------------
 Reporter:  rdela           |      Owner:
     Type:  enhancement     |     Status:  new
 Priority:  normal          |  Milestone:  Awaiting Review
Component:  General         |    Version:  4.4.2
 Severity:  normal          |   Keywords:
  Focuses:  administration  |
----------------------------+-----------------------------
 This is a follow-up to #33156.

 Replying to
 [https://core.trac.wordpress.org/ticket/33156?replyto=17#comment:17
 comment:17 Hube2]:
 > Didn't know if I should start a new ticket or not and couldn't find one
 covers it. The order that WP is outputting the allow/disallow rules
 > {{{
 > Disallow: /wp-admin/
 > Allow: /wp-admin/admin-ajax.php
 > }}}
 > may not compatible with all crawlers. To be comparable with all crawlers
 the order of these rules needs to be reversed.
 >
 > See:
 [https://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive.
 Wikipedia – Robots exclusion standard: Allow directive]
 >
 > I can't find any information that contradicts what is presented in the
 wiki article.

 [https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961
 Google’s example] is in Allow then Disallow order.
 > Pattern-matching rules to streamline your robots.txt code

 {{{
 User-agent: *
 Allow: /*?$
 Disallow: /*?
 }}}

 Google's [https://www.google.com/webmasters/tools/robots-testing-tool
 robots.txt Tester]
 confirms both orders work for googlebot.

--
Ticket URL: <https://core.trac.wordpress.org/ticket/35819>
WordPress Trac <https://core.trac.wordpress.org/>
WordPress publishing platform