[wp-meta] [Making WordPress.org] #4559: Dedicated robots.txt file for translate.wordpress.org

[Making WordPress.org]

#4559: Dedicated robots.txt file for translate.wordpress.org
--------------------------------------+---------------------
 Reporter:  jonoaldersonwp            |       Owner:  (none)
     Type:  enhancement               |      Status:  new
 Priority:  low                       |   Milestone:
Component:  Translate Site & Plugins  |  Resolution:
 Keywords:  seo needs-testing         |
--------------------------------------+---------------------
Changes (by dd32):

 * keywords:  seo => seo needs-testing


Comment:

 After some discussion with @tellyworth, I think the best route forward
 here is to switch to using the WordPress generated `robots.txt` and remove
 the static `robots.txt` that's currently in place.

 That has some downsides, one of them being the potential unknowns
 (currently) of there being a domain that we're not thinking of that is
 currently relying upon the static file.. But assuming we can work around
 that, it seems like moving is the best option here.

 The above commits implement the existing robots.txt file and the request
 here into something that is usable.

 Currently:
  - https://*/robots.txt is still using the static file
  - The WordPress generated file can be accessed like so:
 https://wordpress.org/?robots=1 or https://de.wordpress.org/?robots=1
  - For translate.wordpress.org, it can be found here:
 https://translate.wordpress.org/?robots=1&gp_route=robots.txt

 The Robots file for translate.wordpress.org..
  - Drops the `noindex:` declaration as it'll be
 [https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-
 robotstxt.html no longer supported by Google as of September].
  - Adds the Global Stats, Consitency Report, locale index, the Locale
 Indexes and Locale Plugin/Theme stats to the Allow ruleset.

 I couldn't find many other pages worth indexing on
 translate.wordpress.org, we really don't need to index every project in
 every locale or all the strings, so having Search engines request those
 URLs isn't doing us any benefit.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/4559#comment:6>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org