[wp-meta] [Making WordPress.org] #5806: Remove legacy robots.txt disallow directives

[Making WordPress.org]

#5806: Remove legacy robots.txt disallow directives
----------------------------+---------------------
 Reporter:  jonoaldersonwp  |       Owner:  (none)
     Type:  defect          |      Status:  new
 Priority:  low             |   Milestone:
Component:  General         |  Resolution:
 Keywords:  seo             |
----------------------------+---------------------

Comment (by dd32):

 > Remove legacy robots.txt disallow directives

 These are the current core robots.txt directives, please follow up with a
 Core ticket if these are not appropriate for all WordPress sites.

 I'll take a middle-ground approach here and specifically allow access to
 `/wp-admin/load-*.php` because I do know that some WordPress.org themes
 trigger some WordPress functionality that other sites wouldn't on the
 front-end.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/5806#comment:1>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org