[wp-meta] [Making WordPress.org] #6766: Update developer.wordpress.org/robots.txt to prevent spam attacks

[Making WordPress.org]

#6766: Update developer.wordpress.org/robots.txt to prevent spam attacks
----------------------------+-----------------------------
 Reporter:  jonoaldersonwp  |      Owner:  (none)
     Type:  enhancement     |     Status:  new
 Priority:  low             |  Milestone:
Component:  General         |   Keywords:  seo performance
----------------------------+-----------------------------
 The developer site is the subject of [[https://yoast.com/internal-site-
 search-spam/|internal site search spam attacks]].

 This impacts our crawl budget, and floods our Google Search Console
 account (potentially blinding us to other issues).

 We can reduce the impact of this by tweaking the site's robots.txt rules
 as follows, to block search patterns (and add some best practices whilst
 we're there).

 {{{
 # Prevent crawling of WP internals
 # --------------------------------
 User-agent: *
 Disallow: /wp-admin/
 Disallow: /?rest_route=
 Disallow: /xmlrpc.php

 # Prevent crawling of search URLs
 # --------------------------------
 Disallow: /?s=
 Disallow: /search/
 }}}

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/6766>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org