[wp-trac] [WordPress Trac] #16416: Settings -> Privacy "block search engines" option is not accurate

[WordPress Trac]

#16416: Settings -> Privacy "block search engines" option is not accurate
---------------------------------------------+-----------------------------
 Reporter:  designsimply                     |       Owner:  westi
     Type:  defect (bug)                     |      Status:  accepted
 Priority:  normal                           |   Milestone:  Future Release
Component:  Administration                   |     Version:
 Severity:  normal                           |  Resolution:
 Keywords:  has-patch ux-feedback 3.2-early  |
---------------------------------------------+-----------------------------

Comment (by mikeschinkel):

 Replying to [comment:5 westi]:
 > Ask and normal sound fine to me.
 >
 > robots.txt is only a request and only obeyed by good citizen search
 engines - in some cases bad citizens of the web will use them to find
 hidden content.
 >
 > Normal was already in the string for a long time and sounds fine to me.

 FWIW, I've recently had several test domains be indexed by Google after
 implementing analytics on them, even though I had `robots.txt` in full
 disallow mode.  I had to request removal to get the pages out of the
 index, and I had to write a plugin that returned 403 for any page
 Googlebot visited on those sites to keep it out of the index.  YMMV.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/16416#comment:6>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software