Hello everyone...
i want ask about how to block robots ahrefs/majestic/semrush/moz ?
My questions is what script to hide/block in .htaccess ?
thankyou
Printable View
Hello everyone...
i want ask about how to block robots ahrefs/majestic/semrush/moz ?
My questions is what script to hide/block in .htaccess ?
thankyou
try something like this in robots.txt:
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
Or
via htaccess:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot [OR]
RewriteRule . - [F,L]
I don't know what bot use Moz, but i am pretty sure that you can find it :)
Hope that helps a little.
Hope this will help - https://gorilla.marketing/seo-news/h...m-competitors/
What I find ironic is that gorilla marketing claims to do something they do not themselves do.
What they claimed they did in their own robots text file is not what is is in their own robots text file.
Rick
Universal4
Here is a little trick I've tried in .htaccess
RewriteCond %{HTTP_USER_AGENT} !(Google|msnbot|bingbot|facebook|duckduck) [NC]
RewriteCond %{HTTP_USER_AGENT} (Bot|Spider|Crawl) [NC]
RewriteCond %{REQUEST_URI} !^bots\.htm
RewriteRule .* /bots.htm [L]
That basically says :
- if it's not Google, msn, bing etc
- and if it has the words bot or spider or crawl
- and it's not already looking for a file called bots.htm
- then send them to a file called bots.htm
This doesn't give them an error code - just all they ever see is a file called bots.htm no matter what they ask for.
That hopefully screws up their plans and they don't notice that they've been directed to a dummy file.
Ha....and if you wanna be mean, make the bots.htm for certain useragents an infinite loop....or at least one that goes on for a few days....lol
Rick
Universal4
Normally I wouldn't mind sending rogue bots on a wild crawl, but an infinite loop would probably cash your server or max out your shared hosting account. But for that, I'd give your half-joking idea a thumbs up (if I could).Quote:
Originally Posted by universal4