[wp-meta] [Making WordPress.org] #6763: Update robots.txt (and rosetta variations)

[Making WordPress.org]

#6763: Update robots.txt (and rosetta variations)
---------------------------------------+----------------------
 Reporter:  jonoaldersonwp             |       Owner:  (none)
     Type:  enhancement                |      Status:  closed
 Priority:  low                        |   Milestone:
Component:  General                    |  Resolution:  invalid
 Keywords:  seo performance has-patch  |
---------------------------------------+----------------------
Changes (by ogumemura):

 * status:  new => closed
 * resolution:   => invalid


Comment:

 Thank you for your correspondence regarding this matter. We have observed
 link spamming activities taking advantage of WordPress' behavior with
 `?s=` , as mentioned in the **# Prevent crawling of search URLs** section.

 Although attackers could be monitoring this discussion, we have noticed
 that in some instances, they add valid parameters before the "s"
 parameter. An example of this tactic is:
 https://example.com/?tq=obejm&s=spam_words

 The current `Disallow` rule in the robots.txt file, as proposed in the
 pull request, does not cover this scenario. Therefore, we suggest adding
 the following rule to address this issue:


 {{{
 Disallow: /?*&s=
 }}}

 This modification ensures that search URLs with crafty parameter
 combinations, like the example provided, will also be prevented from being
 indexed by search engines.

-- 
Ticket URL: <https://meta.trac.wordpress.org/ticket/6763#comment:6>
Making WordPress.org <https://meta.trac.wordpress.org/>
Making WordPress.org