Many seo sites suck

/begin rant

I am soooo fed up with all these so called seo sites that spider your site and eat up all kinds of bandwith, and then they cache links pointing at your site to pages that have never existed in the entire history of your domain....

There is no justification for EITHER of these issues.

Spidering a site as a search engine is fine, but a real search engine has the potential of sending you traffic, these seo sites will never do that. (they are spidering site A to provide results for the owner of site B, providing no value for site A whatsoever except eating up bandwith)

And the BS of caching a page that never existed....that is more than disgusting it downright criminal.

I have a number of sites this happens to on a frequent basis and I see the 404's pile up in GWT.

/end rant

Rick
Universal4

caching pages that don't exist, sounds a little 'blackhatish', if you know what I mean.
Sounds like whoever is doing the caching is trying to make your site look like a neglected/low quality site to google.

I try to block all these so called SEO sites. Not an easy task, some like majestic seem to go out of their way to make it difficult to block their bandwith eating bots.

Paying for bandwidth so my competitors(or any dirtbag that wants to pay these SEO sites for my information) can have my site information on how I set up and run a website... this is a bit much for me. So I go out of my way to block them or feed them bogus info as soon as they pop their spying little bot heads up. Most of these SEO bots I just consider low life scrapers.

I think this will become a much larger issue over time as more and more people have to start paying more for bandwith, ip's and hosting in general in the coming months and years.

Rick
Universal4

I keep flirting with the idea of going to a "deny all, except" model.

The more I keep getting hit with bots/scrapers, the more I have to add to my deny files.

I keep thinking it would be simpler to just deny all bot traffic except for the main search engine bots. Some scrapers/SEO bots would still get through, pretending to be browsers, but you can usually identify them anyway by their behavior and block them.

Quote:

---

Originally Posted by lots0

I keep flirting with the idea of going to a "deny all, except" model.

The more I keep getting hit with bots/scrapers, the more I have to add to my deny files.

I keep thinking it would be simpler to just deny all bot traffic except for the main search engine bots. Some scrapers/SEO bots would still get through, pretending to be browsers, but you can usually identify them anyway by their behavior and block them.

---

That's a very good idea. Many bots keep trying to access a website causing you problems with your bandwith. You can't keep them all out but you can block most of them.

Personally, I find AHrefs and similar SEO tools pretty useful from a spy-on-my-competitors point of view, but I agree that they provide no value to the owners of the sites they are scraping. If they provided a recognisable user-agent they would be easy to block but it's not in their interests to do that.

who did you use?

CasinoCoupons,

Read the first post again, almost ALL seo spiders do tyhe same thing.

They spider sites, often ignore robots.txt and do so pretending to provide value to the owner of the site being spidered when all they are doing is sucking up bandwith and scraping content with NO VALUE for the site being scraped/spidered.

Rick
Universal4

Isn't there any way around this problem?

Yes, I'm sick of them too.Is better to do the hard work by yourself :( because they only take the money and that's all :)

I am not complaining about hiring one....I am complaining about my resources being used for the benefit of someone else's website.

Like Lots0 stated, you can block them at the server level, either in a firewall, ipsec, load balancer or other nat device, or even htaccess.

Rick
Universal4

On Wordpress I am using the plugin Wordfense. It has a tab showing your Crawlers, Google Crawlers, Humans. I just click on Crawlers and start banning hosts that I do not like. In fact I don't really need anything other than Google Yahoo Bing perhaps. I am wondering about the Chinese Baidu .... France, Nigeria, Kenya whatever country I see I just perma block that crawler.

The thing is.... new ones come up every day :) What's interesting is MSN is the biggest consumer, even before Google! Yandex comes third, I wonder what would be the impact of blocking MSN....

I have been playing with an "Exclude all, Except" code for htaccess.

So far, the biggest problem is deciding which bots are useful to my site and which are just bandwidth sucking parasites.
By far most are just parasites or thieves, but there are the occasional Bots from a new start up that I want to let in.
So this is not just a set it and forget it type of solution, it takes constant monitoring... :(

I block Baidu, simply because 99.9% of all their users are from China.
We don't ever get punters from China, so I see no reason to let them use my Bandwidth. Even though Baidu is the 2nd largest Search Engine in the world.
On a side note: once I started blocking Baidu, the hacking attempts/attacks from China dropped by at least 80%.

I take a radical choice... I Block all CN, JP, TW and others. 99% of the traffic coming from this are spam or harvesters.

I also never had any player for asian countries so there is no point of waste resources...

I have been considering the China ban also.

Rick
Universal4

I am totally fed up with this. In my 5 years time as casino affiliate I think I have gotten 5000+ comment posts, 1000+ junk mails and 500+ so called business proposals from SEO companies. I DON'T NEED YOUR CRAPPY PARISITAL SEO HELP!!! Is my message to all these guys; is there a way to block this? Many Wordpress Plugins claim to disable spamming but they simply cannot. I know perfectly well a lot of webmasters just ignore these spam posts, but this is the reason they still exist! Let us unite!!! Let's make a special topic: blacklisted SEO spammers and just add every single url that comes up.

*** edit*** they actually come with very good url's , the more reason toi blacklist them so they lose these valuable domains/URL's.

Quote:

---

Originally Posted by universal4

I have been considering the China ban also.

Rick
Universal4

---

I don't think you would regret it.
IMO, It is much easier to just block every Chinese class B IP you find rather than trying to sort out the malicious traffic from China.