[wp-meta] [Making WordPress.org] #5668: Add an 'empty' robots.txt file to s.w.org

[Making WordPress.org]

#5668: Add an 'empty' robots.txt file to s.w.org
-----------------------------+-----------------------
 Reporter:  jonoaldersonwp   |       Owner:  dd32
     Type:  defect           |      Status:  accepted
 Priority:  lowest           |   Milestone:
Component:  General          |  Resolution:
 Keywords:  seo performance  |
-----------------------------+-----------------------
Changes (by dd32):

 * owner:  (none) => dd32
 * status:  new => accepted


Comment:

 [10840] has added the robots.txt, but due to how the CDN is setup for
 caching, it could take a bit for the changes to take effect.

 {{{
 # Current cached public response
 $ curl -is https://s.w.org/robots.txt
 HTTP/2 301
 location: https://wordpress.org/
 x-nc: HIT mdw 1

 # New Response, uncached.
 $ curl -is https://s.w.org/robots.txt?1
 HTTP/2 200
 x-nc: MISS mdw 1

 User-agent: *
 Disallow:


 # New Response, cached.
 $ curl -is https://s.w.org/robots.txt?3
 HTTP/2 200
 x-nc: HIT mdw 1

 User-agent: *
 Disallow:
 }}}

 I'm not going to request the CDN be cleared quite yet, as it should self-
 clear within a few days I expect.  I'll keep this open to remind myself.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/5668#comment:2>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org