[wp-meta] [Making WordPress.org] #4559: Dedicated robots.txt file for translate.wordpress.org

[Making WordPress.org]

#4559: Dedicated robots.txt file for translate.wordpress.org
--------------------------------------+---------------------
 Reporter:  jonoaldersonwp            |       Owner:  (none)
     Type:  defect                    |      Status:  new
 Priority:  high                      |   Milestone:
Component:  Translate Site & Plugins  |  Resolution:
 Keywords:  seo                       |
--------------------------------------+---------------------
Description changed by jonoaldersonwp:

Old description:

> This consumes huge amounts of crawl budget, for relatively little return.
> We'd like to block crawling of it entirely, via robots.txt.
>
> At the moment, it shares a robots.txt with other wordpress.org domains,
> which makes this impossible.
>
> Can we give it a dedicated robots.txt file, which is separate from other
> sites, with the following contents:
>
> {{{
> User-agent: *
> Disallow: /*
> Noindex: /*
> Allow: /$
> }}}
>
> If/when this is complete, the `?filter` rule can be removed from the
> shared/global robots.txt file.

New description:

 This consumes huge amounts of crawl budget, for relatively little return.
 We'd like to block crawling of it entirely, via robots.txt.

 At the moment, it shares a robots.txt with other wordpress.org domains,
 which makes this impossible.

 Can we give it a dedicated robots.txt file, which is separate from other
 sites, with the following contents:

 {{{
 User-agent: *
 Disallow: /*
 Noindex: /*
 Allow: /$
 }}}

 NB: We'll need to be absolutely certain that this is a standalone file,
 and doesn't bleed through to any other WP domains/contexts, or we'll cause
 the end of the world.

 If/when this is complete, the `?filter` rule can be removed from the
 shared/global robots.txt file.

--

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/4559#comment:1>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org