[wp-hackers] Blocking SEO robots
david at wordshell.net
Wed Aug 6 12:08:26 UTC 2014
Haluk Karamete wrote:
> Could this list help you?http://www.robotstxt.org/db/all.txt
At first this looks potentially useful - since it is in a
machine-readable format, and can be parsed to find a list of bots that
match specified criteria.... but on a second glance, it looks not so
useful. I searched for 3 of the recent bots I've seen most regularly in
my logs: SEOKicks, AHrefs, Majestic12 - and it doesn't have any of them.
Blue Chives wrote:
> Depending on the web server software you are using you can look at using the htaccess file and block users/bot based on their user agent.
> This article should help:
The issue's not about how to write blocklist rules; it's about having a
reliable, maintained, categorised list of bots such that it's easy to
automate the blocklist. Turning the list into .htaccess rules is the
easy bit; what I want to avoid is having to spend long churning through
log files to obtain the source data, because it feels very much like
something there 'ought' to be pre-existing data out there for, given how
many watts the world's servers must be wasting on such bots.
UpdraftPlus - best WordPress backups - http://updraftplus.com
WordShell - WordPress fast from the CLI - http://wordshell.net
More information about the wp-hackers