Hosting Companies Blocking sites like Ahref and SEMrush crawling

Printable View

26 April 2023, 8:13 pm
Jamie - RateMyCasino

Hosting Companies Blocking sites like Ahref and SEMrush crawling

Not sure if anyone else is coming across this, But Ahref + Semrush crawling bots are starting to be blocked by hosting companies. Is this new or something they have been doing for a while.

Quoted from our hosting company below:

Quote:

We have implemented a mod security rule where in this will block bots that are known to heavily crawl website and consume unnecessary resources. This includes ahrefs and semrush.

I only noticed this last night as I went to check our site on Ahref and it cant crawl our site anymore.
26 April 2023, 9:59 pm
TheGooner

I block those bots for the very same reason. I'd love a host that would give me the option to actively monitor for bots and block them.
26 April 2023, 10:56 pm
mickyfu

Surely it should be optional, if not, just switch hosts.
Possibly just another company angry with Ahrefs ridiculous credit system. This seems common on PBN style hosting, I've never seen it on legitimate hosting companies.
26 April 2023, 11:53 pm
PaulEchere

I do also block spammy crawler bots, but I do it with cloudflare
27 April 2023, 2:58 am
CasinosBroker

Quote:

Originally Posted by mickyfu

Surely it should be optional, if not, just switch hosts.
Possibly just another company angry with Ahrefs ridiculous credit system. This seems common on PBN style hosting, I've never seen it on legitimate hosting companies.

Couldn't agree more.

I assume it is a PBN hosting company...
27 April 2023, 3:05 am
xecutable

Quote:

Originally Posted by PaulEchere

I do also block spammy crawler bots, but I do it with cloudflare

Cloudflare's Bot Fight is so aggressive that it blocks the cron jobs from your own host, so you'd have to whitelist that.
27 April 2023, 4:24 am
Jamie - RateMyCasino

Quote:

Originally Posted by TheGooner

I block those bots for the very same reason. I'd love a host that would give me the option to actively monitor for bots and block them.

So if its good to do that, how do you utilize sites like Ahref or Semrush with the suggestions and ideas? Or don't people use their info to help improve your site anymore.
27 April 2023, 5:53 am
baldidiot

Quote:

Originally Posted by LuckyBettors - Jamie

So if its good to do that, how do you utilize sites like Ahref or Semrush with the suggestions and ideas? Or don't people use their info to help improve your site anymore.

Just to play devils advocate (I don't do this)... if you wanted to block these tools but still use them for your own purposes you could unblock the bots and run a manual audit, then block them again.

Also bear in mind that a lot of the keyword suggestions come from the search rankings, which you can't block as they come from the SERPs, not your site. The it's only really for internal audits that you'd need it (ie: broken links, internal link suggestions, page issues and so on).