Semalt.com

[edgarf76]

I was googling and came across an Seo site that looks decent. Does anyone have any experience with semalt.com?

[universal4]

Caution might be needed if any of these comments are true.
http://clicky.com/forums/?id=16360
http://www.straight.com/blogra/59320...y-flags-semalt
http://thegrinningskull.wordpress.co...what-fk-is-it/

It seems they started being seen as early as January, and a number of people are not impressed.

Rick
Universal4

[universal4]

I recommend staying far away.
https://twitter.com/hashtag/semalt

Rick
Universal4

[universal4]

What I find interesting, and disgusting at the same time, is that they do NOT use a clearly defined user agent for their bot. If they did we could easily block the user agent....that says a lot about them. Most respectable crawlers will have an about crawler page, and will define the user agent so that anyone that wants to block them can.

I also found it quite interesting that no one has been able to pin down a set of ip's or subnets that they always use, again would make blocking easier. They must jump around and bounce off proxies or??

Rick
Universal4

[universal4]

I thought I had finally found someone who had pinned down their ip
http://logorrhoea.net/2014/01/how-to...sing-htaccess/
217.23.11.15 but it turns out that is only the ip of their website, and although they may run the crawler from there, from what I am reading I don't think it will help.

The absolute BEST comment I have read so far is

Hi Seth,
I would not recommend going to their website and submitting anything. I have read numerous times that this does not work and some believe they are using that as a way to confirm websites and people that have done that have found the crawling to intensify. The people running this bot have no respect for webmasters. I read on Twitter where someone had managed to redirect the semalt crawlers back to the semalt server and then the owner of the bot accused this person of launching a DDOS attack against them, when it was actually their own bots coming back at them, #semalt on twitter will bring it up.

(bolding mine)

that had me laughing a fair amount so I will be sure to visit the http://logorrhoea.net again to see what other entertaining information they might have.

What I have read so far on other sites in comments etc, many claimed the ip's were all over the place different countries etc....

Rick
Universal4

[edgarf76]

I only did the trial but I did run my site through. I'm not going to bug out about it, there isn't anything I can do now.

[sportsfreak]

Their bot is actually on my sites daily and I have been wondering about them just the other day...

[RacingJim]

Their bot is actually on my sites daily and I have been wondering about them just the other day...

Same mate, I was wondering what it was but just ignored it.

[universal4]

It may be referral spam, or the bot may also be scraping and databasing content for later.

I have read a number of reports that it ignores robots.txt and also changes the user agent after being blocked by such methods as htaccess. These actions are bad behavior by a bot.

They do have a remove database, but I have read many reports of them returning, (sometime with more activity), after asking for removal and that also should be considered bad behavior.

Rick
Universal4

[jarowe]

ah think my blog is affected by this! What should I do??

[Sergej_AGR]

I blocked semalt referrer spam with adding this to htaccess

RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [OR]
RewriteCond %{HTTP_REFERER} semalt\.semalt\.com [NC]
RewriteRule .* - [F]

[sportsfreak]

http://semalt.com/project_crawler.php - says you can have your site removed from their crawl list, just tried, will see if it works.

[sweetbet]

I blocked semalt referrer spam with adding this to htaccess

RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt\.com [OR]
RewriteCond %{HTTP_REFERER} semalt\.semalt\.com [NC]
RewriteRule .* - [F]

Thanks for that. I'm going to add it to one of my sites that has been getting hit quite a bit by semalt.

[edgarf76]

I just added my site to the list, hopefully no harm was done in the first place.

[sportsfreak]

looks like it worked, no semalt bot on my sites in the last hours

[universal4]

I am interested if this continues for a number of weeks, especially after someone runs a compare with your site.

Rick
Universal4

[kfau]

I am interested if this continues for a number of weeks, especially after someone runs a compare with your site.

Rick
Universal4

Hi folks,

Full disclosure - I work for an SEO agency and have been tracking semalt on both personal and work fronts.
I received an update from google about this thread and wanted to drop in with some details.

I tried their crawler removal tool on a few sites, it does stop briefly, then comes back - blocking it is the best method.
Here is a post I have done with a lot of research and information, with 2 options to block them at the bottom.
kymodo(dot)com(dot)au what-is-semalt-doing-on-your-website/

If you have any questions, please feel free to ask either on the post or on twitter - I'm KarenDFrancis
Hope that sheds some light.

[edgarf76]

How did you receive a update from google about this thread kfau ?

[kfau]

How did you receive a update from google about this thread kfau ?

Thanks for asking, sorry I didn't explain that previously.
I use google alerts - google(dot)com/alerts and one of the terms I have been tracking recently is semalt.

[edgarf76]

Thanks for asking, sorry I didn't explain that previously.
I use google alerts - google(dot)com/alerts and one of the terms I have been tracking recently is semalt.

I gotcha