[wp-trac] [WordPress Trac] #18465: Prevent search engines from indexing wp-admin and wp-includes (was: robots.txt should tell Google to not index wp-admin and wp-includes)

[WordPress Trac]

#18465: Prevent search engines from indexing wp-admin and wp-includes
--------------------------+-----------------------
 Reporter:  Viper007Bond  |       Owner:  ryan
     Type:  enhancement   |      Status:  reopened
 Priority:  lowest        |   Milestone:  3.3
Component:  General       |     Version:  3.2.1
 Severity:  trivial       |  Resolution:
 Keywords:  has-patch     |
--------------------------+-----------------------
Changes (by joostdevalk):

 * status:  closed => reopened
 * resolution:  fixed =>


Comment:

 This is a valid problem but the "fix" doesn't actually fix it. While the
 addition to robots.txt blocks the crawler from opening the URL, a URL that
 cannot be opened CAN still be listed in the index if Google finds enough
 links pointing to it, see the note on
 [http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
 this Google help page]. Example of this can be seen on my Dutch domain:

 https://www.google.com/search?q=site%3Ayoast.nl++inurl%3Awp-admin

 The solution is to not exclude the admin directory in robots.txt, but to
 send an X-Robots-Tag HTTP header of value noindex (the HTTP version of a
 robots meta tag) for the files in admin and for admin-ajax.php, will add a
 patch.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/18465#comment:28>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software