[wp-trac] [WordPress Trac] #23070: Generated robots.txt file with privacy on should have Allow: /robots.txt

[WordPress Trac]

#23070: Generated robots.txt file with privacy on should have Allow: /robots.txt
-------------------------+------------------------------
 Reporter:  iamfriendly  |       Owner:
     Type:  enhancement  |      Status:  new
 Priority:  normal       |   Milestone:  Awaiting Review
Component:  General      |     Version:
 Severity:  normal       |  Resolution:
 Keywords:               |
-------------------------+------------------------------

Comment (by iamfriendly):

 Sure, that's definitely an option. But I'm thinking more about people who
 don't really know how to do that. If someone has already enabled privacy
 and google has crawled their site, then that generated robots.txt file
 will be cached for an indeterminate amount of time (up to 48 hours).

 My thoughts were that adding this one line to the robots.txt file
 generated by WordPress could stop some folk being out of google rankings
 for a couple of days. And it's a relatively easy process (and something
 I'd add to the codex) as to how to force webmaster tools to recrawl if the
 line is in the robots.txt file.

 Adding a 'real' robots.txt file after G has already crawled doesn't do you
 any good, it'd need to be something you did immediately after you created
 the instance of WP online.

-- 
Ticket URL: <http://core.trac.wordpress.org/ticket/23070#comment:2>
WordPress Trac <http://core.trac.wordpress.org/>
WordPress blogging software