[wp-hackers] Blocking SEO robots

[David Anderson]

Haluk Karamete wrote:
> Could this list help you?http://www.robotstxt.org/db/all.txt
At first this looks potentially useful - since it is in a 
machine-readable format, and can be parsed to find a list of bots that 
match specified criteria.... but on a second glance, it looks not so 
useful. I searched for 3 of the recent bots I've seen most regularly in 
my logs: SEOKicks, AHrefs, Majestic12 - and it doesn't have any of them.

Blue Chives wrote:
> Depending on the web server software you are using you can look at using the htaccess file and block users/bot based on their user agent.
>
> This article should help:
>
> http://www.javascriptkit.com/howto/htaccess13.shtml
The issue's not about how to write blocklist rules; it's about having a 
reliable, maintained, categorised list of bots such that it's easy to 
automate the blocklist. Turning the list into .htaccess rules is the 
easy bit; what I want to avoid is having to spend long churning through 
log files to obtain the source data, because it feels very much like 
something there 'ought' to be pre-existing data out there for, given how 
many watts the world's servers must be wasting on such bots.

Best wishes,
David

-- 
UpdraftPlus - best WordPress backups - http://updraftplus.com
WordShell - WordPress fast from the CLI - http://wordshell.net