[wp-meta] [Making WordPress.org] #5938: Add an x-robots-tag header to md5/sha1 etc file URLs

[Making WordPress.org]

#5938: Add an x-robots-tag header to md5/sha1 etc file URLs
----------------------------+--------------------
 Reporter:  jonoaldersonwp  |      Owner:  (none)
     Type:  defect          |     Status:  new
 Priority:  lowest          |  Milestone:
Component:  General         |   Keywords:  seo
----------------------------+--------------------
 We expose lots of URLs like this across wordpress.org:

 https://wordpress.org/wordpress-5.6.3.zip.sha1
 https://wordpress.org/wordpress-4.0.5.tar.gz.md5
 https://br.wordpress.org/wordpress-4.7-pt_BR.zip.md5

 These consume considerable crawl resources and often tripS the 'soft 404'
 warning in Google Search Console.

 We should manage this by adding an `x-robots-tag` header to all responses
 ending in `.sha1`, `.md5` and similar, with a value of `noindex, follow`.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/5938>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org