On Sun, Oct 24, 2010 at 3:56 PM, Jeremy Visser <jeremy at visser.name> wrote: > (Yes, you'll need to hook into do_robotstxt to add that line, but that's > left as an exercise to the reader.) > You only need to do that if you can't create the robots.txt file on your server.