[wp-trac] [WordPress Trac] #18465: robots.txt should tell Google to not index wp-admin and wp-includes

[WordPress Trac]

#18465: robots.txt should tell Google to not index wp-admin and wp-includes
--------------------------+--------------------
 Reporter:  Viper007Bond  |       Owner:
     Type:  enhancement   |      Status:  new
 Priority:  lowest        |   Milestone:  3.3
Component:  General       |     Version:  3.2.1
 Severity:  trivial       |  Resolution:
 Keywords:  has-patch     |
--------------------------+--------------------

Comment (by dd32):

 > Well I tested it in Google Webmaster tool and it blocked it from the
 crawler access.

 Seems that Google and a few others support an "Extended" robots.txt
 standard, see the "June 2008 Agreement" here:
 http://www.searchtools.com/robots/robots-exclusion-protocol.html

 Best going for the explicit directory deny for compatibility with other
 crawlers, since the wildcard doesn't bring any dramatic advances over the
 the original standard.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/18465#comment:17>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software