Hosting Companies Blocking sites like Ahref and SEMrush crawling

[Jamie - RateMyCasino]

Not sure if anyone else is coming across this, But Ahref + Semrush crawling bots are starting to be blocked by hosting companies. Is this new or something they have been doing for a while.

Quoted from our hosting company below:

We have implemented a mod security rule where in this will block bots that are known to heavily crawl website and consume unnecessary resources. This includes ahrefs and semrush.

I only noticed this last night as I went to check our site on Ahref and it cant crawl our site anymore.

[TheGooner]

I block those bots for the very same reason. I'd love a host that would give me the option to actively monitor for bots and block them.

[mickyfu]

Surely it should be optional, if not, just switch hosts.
Possibly just another company angry with Ahrefs ridiculous credit system. This seems common on PBN style hosting, I've never seen it on legitimate hosting companies.

[PaulEchere]

I do also block spammy crawler bots, but I do it with cloudflare

[CasinosBroker]

Surely it should be optional, if not, just switch hosts.
Possibly just another company angry with Ahrefs ridiculous credit system. This seems common on PBN style hosting, I've never seen it on legitimate hosting companies.

Couldn't agree more.

I assume it is a PBN hosting company...

[xecutable]

I do also block spammy crawler bots, but I do it with cloudflare

Cloudflare's Bot Fight is so aggressive that it blocks the cron jobs from your own host, so you'd have to whitelist that.

[Jamie - RateMyCasino]

I block those bots for the very same reason. I'd love a host that would give me the option to actively monitor for bots and block them.

So if its good to do that, how do you utilize sites like Ahref or Semrush with the suggestions and ideas? Or don't people use their info to help improve your site anymore.

[baldidiot]

So if its good to do that, how do you utilize sites like Ahref or Semrush with the suggestions and ideas? Or don't people use their info to help improve your site anymore.

Just to play devils advocate (I don't do this)... if you wanted to block these tools but still use them for your own purposes you could unblock the bots and run a manual audit, then block them again.

Also bear in mind that a lot of the keyword suggestions come from the search rankings, which you can't block as they come from the SERPs, not your site. The it's only really for internal audits that you'd need it (ie: broken links, internal link suggestions, page issues and so on).