Blocking bots (such as ahrefs)

[universal4]

Thanks for posting this...

I am actually considering blocking their useragent, and possibly the entire slash 24.

Rick
Universal4

[Pmig]

exist more than one /24,

I already block:

173.199.114.0/24
173.199.115.0/24
173.199.116.0/24
173.199.117.0/24
173.199.118.0/24
173.199.119.0/24
173.199.120.0/24

and some other random ips...

[universal4]

pmig, I would be interested how you came up with that list....I am not disputing it but am intrigued.

I have been looking at this a little today and found this
http://www.webmasterworld.com/search...rs/4352766.htm

My research leads me to think that they also work from the 173.86 classes so it is why I asked.

Rick
Universal4

[Pmig]

Directly from my access_logs, for coincidence 2 days ago I spend my day parsing my logs and banning bad bots, like the ones who make 400 error requests to / and ending also blocking all ips with ahrefs user agents.

Wait 2 or 3 minutes and I will show you some entries

[Pmig]

173.199.114.211 - - [02/Nov/2012:19:40:45 +0100] "GET /************/ HTTP/1.1" 200 8613 "-" "Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)"
173.199.114.163 - - [02/Nov/2012:19:49:54 +0100] "GET /************ HTTP/1.1" 200 8334 "-" "Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)"
173.199.114.243 - - [02/Nov/2012:19:59:13 +0100] "GET /************s/ HTTP/1.1" 200 8698 "-" "Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)"
173.199.114.51 - - [02/Nov/2012:20:08:34 +0100] "GET /************/ HTTP/1.1" 200 8181 "-" "Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)"
173.199.120.59 - - [02/Nov/2012:20:18:13 +0100] "GET /************/ HTTP/1.1" 200 6754 "-" "Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)"
213.186.119.136 - - [04/Nov/2012:22:58:41 +0100] "GET /************/ HTTP/1.1" 200 10139 "-" "Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/)"

the log have a lot more ips but the main ones are that 173.199
You can also see that 213.186

looking now I see the ranges I post is the new bot version, the old one use the 213.186 one.

[universal4]

Thanks,

I would love to see what ip ranges you ban. I'd be happy to provide an email address where you could send it in text.

Since I redid my rack and infrastructure this past spring/summer I have fallen a bit behind on my blocking etc and rather than try to block all the bad bots indivdually in htaccess (although doing so by useragent catches them on new ip's) blocking at the switch or load balance level stops them from getting at any of the servers or sites from a single point.

Sorry for hijacking the thread lowrisk, I may split this off.

Rick
Universal4

[Pmig]

The problem with the user agents is the easy spoof, right now I don't have a big list because I rebuild my infra days ago (and dont backup fw tables and logs) and I'm banning as I see abuse patterns. We can open a topic on private members section where people can share bad ip's and networks.

Feel free to send me a pm with your email and I will send what I get so far (Its not much but is a (re)start.)

[universal4]

I will do that....

Rick
Universal4

[Muppet]

AHrefs honours robots.txt directives. More trouble are the scum sites like Seomoz that don't, and aggregate data from loads of different sources making it almost impossible to completely block them.

The problem with AHrefs is that it doesn't need to spider your site to get the data - it only needs to spider the sites that link to you and they can construct an inbound linking profile. By blocking them you are only stopping them from spidering links from your site to others.

Anyway put this in robots.txt

# AHrefs
User-agent: AhrefsBot
Disallow: /

[lots0]

I try to block all the information/bandwidth stealing sites like seomoz and ahrefs.

I use a combination of, banning IP's and UserAgents along with spider traps.

However as this keeps becoming more and more work to keep these thieves off my site by just trying to block the bad bots...
I keep thinking that I may go from the current model of just trying to block bad bots to a model of only allowing a few select bots access(googlebot, slurp,msn etc.) and blocking all others.

Added... I get my IP block lists from my spider traps.

[lowrisk]

This is hilarious, here's me thinking it was a good tool, seems a lot here aren't so keen.

[universal4]

You see the thing is as webmasters we build sites for visitors.

The obvious fallout from that is that we also build sites for the search engines, but the ultimate goal is to gain more visitors.

Ahrefs, or any other site scraper spidering my sites to provide others with information on my linking or content will do nothing to advance that. And while the bots are doing this they are eating up MY server resources, and MY bandwith which is an associated cost for my hosting that does nothing to further the ultimate goal of visitors.

I found an interesting article the other day while reasearching, and what struck me as the most interesting part of the discussion(in the comments) is the justification that a copyscape rep used to justify what their bot does in order to gain continued access to sites in the comments.(instead of just honestly disclosing to the webmaster community as a whole the user-agent and ip's used)
http://searchenginewatch.com/article...-the-Parasites

Rick
Universal4