[wp-meta] [Making WordPress.org] #4559: Dedicated robots.txt file for translate.wordpress.org

[Making WordPress.org]

#4559: Dedicated robots.txt file for translate.wordpress.org
--------------------------------------+--------------------
 Reporter:  jonoaldersonwp            |      Owner:  (none)
     Type:  defect                    |     Status:  new
 Priority:  high                      |  Milestone:
Component:  Translate Site & Plugins  |   Keywords:  seo
--------------------------------------+--------------------
 This consumes huge amounts of crawl budget, for relatively little return.
 We'd like to block crawling of it entirely, via robots.txt.

 At the moment, it shares a robots.txt with other wordpress.org domains,
 which makes this impossible.

 Can we give it a dedicated robots.txt file, which is separate from other
 sites, with the following contents:

 {{{
 User-agent: *
 Disallow: /*
 Noindex: /*
 Allow: /$
 }}}

 If/when this is complete, the `?filter` rule can be removed from the
 shared/global robots.txt file.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/4559>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org