Results 1 to 9 of 9
  1. #1
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,034
    Thanks
    2,228
    Thanked 7,923 Times in 4,996 Posts

    Default Bad bots and other ip's you block

    What sites, user agents or ip's do you block?

    For most people this would predominantly be bad bots that refuse to follow robots.txt

    Personally, I am leaning toward blocking of seo scanners since in essence all they are is scrapers, and I have witnessed a lot of them that refuse to follow robots.txt files. (in addition to the fact that they eat up YOUR bandwith, will NOT be sending you traffic, and their entire function is to help your competition)

    If there is enough interest we can also discuss bad ip's that attempt mail hacks, terminal or ssh attacks, or general port scans and syn floods.

    We have discussed this a few times here, but often we do not publish lists, but often will make them available to those who are interested in them.

    I do not currently do any country wide IP range blocking, but I am getting a little closer to that all the time...lol

    Rick
    Universal4

  2. #2
    Roulette Zeitung is offline Public Member
    Join Date
    July 2012
    Location
    Germany
    Posts
    4,446
    Blog Entries
    5
    Thanks
    6,015
    Thanked 6,685 Times in 2,950 Posts

    Default

    Every response is a gift.
    A gift!
    One response.
    Moved.


    Leopold

  3. #3
    TravG's Avatar
    TravG is offline Private Member
    Join Date
    September 2008
    Posts
    2,050
    Thanks
    34
    Thanked 178 Times in 130 Posts

    Default

    I personally do not block any but your comment about the seo scanners is right on. I really never gave this much thought but I do think it is a good idea. I would be willing to contribute as much as I can which to be honest may not be much just do to lack of knowledge.

    How would you block a bot other than the robots file?
    Live Casino USA - the best USA live online casinos. Play USA online slots and other casino games like USA online blackjack. Like poker? See www.usalegalpoker.com for the best USA poker sites and USA online casinos for the best USA online casino. https://mobileslotslisting.com
    www.usalegalslots.com - www.playusaonlinepoker.com - www.4deucespoker.com

  4. #4
    sweetbet's Avatar
    sweetbet is offline Public Member
    Join Date
    November 2012
    Posts
    2,819
    Blog Entries
    5
    Thanks
    898
    Thanked 1,576 Times in 1,087 Posts

    Default

    Quote Originally Posted by UltimatePokerBonus View Post
    How would you block a bot other than the robots file?
    It'd be via the .htaccess file, but I'm not sure how. I'd have to read about it.

  5. #5
    slotplayer is offline Private Member
    Join Date
    September 2006
    Posts
    1,040
    Thanks
    195
    Thanked 323 Times in 252 Posts

    Default

    I'm seeing these askives back links in WMT lately.
    Looks like askives automatically creates an html page when some asks a ?

    as far as these SEO scanners they're a pain.
    Last edited by slotplayer; 9 October 2013 at 11:11 am.

  6. #6
    Sergej_AGR is offline Private Member
    Join Date
    March 2012
    Location
    Ruhrpott - Germany
    Posts
    215
    Thanks
    48
    Thanked 85 Times in 60 Posts

    Default

    I block libwww-perl agents via htaccess and use anti spambot scripts like honeypot.

    I also block visitors from india, bangladesh etc. I think they all want to grab my pictures and rewrite my content only

  7. The Following User Says Thank You to Sergej_AGR For This Useful Post:

    -Shay- (9 October 2013)

  8. #7
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,034
    Thanks
    2,228
    Thanked 7,923 Times in 4,996 Posts

    Default

    Yes blocking by way of htaccess is probably the easiest way for the average webmaster.

    I also block some ip's and some classes directly at the server level, as well as on some of the NAT devices. I block a lot more on the Mail Server than other servers.

    As far as blocking the average spiders etc, I am currently using the htaccess method, but it is a pain since you have to make the entries in each file for each site.

    If/when I see an ip totally going apesh** I try to block it as close to the front door as I can in order to reduce the amount of bandwith it eats up. If the attempts do not subside at the point, I start capturing packets and contact the service provider if it is possible.

    I admit I am kinda torn about the best way to handle the seo scanners and whether to just start blocking them all or not.

    Rick
    Universal4

  9. #8
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,034
    Thanks
    2,228
    Thanked 7,923 Times in 4,996 Posts

    Default

    A perfect example of this just took place the last half hour.

    While I was reading through the threads I got a warning that something was up on one of my servers. I chacked and the site that triggered the warning was fine and I could browse it so I logged in to the monitoring server and although the site was up, there was something going on.

    So I logged in to the server in question and started capturing some packets and sure enough there were a few ip's that were throwing packets at it with bad checksums. Now some of these were due to checksum offload but searching the ips I find that one was German and two were russian.

    One of the russian IP's has a really bad reputation at stop forum spam so my guess is that one was scanning all the sites looking for emails and login type pages for spamming purposes.

    I did not block these ip's immediately, but will track them and if they start showing up a lot I will take further action.

    Rick
    Universal4

  10. #9
    Sergej_AGR is offline Private Member
    Join Date
    March 2012
    Location
    Ruhrpott - Germany
    Posts
    215
    Thanks
    48
    Thanked 85 Times in 60 Posts

    Default

    The forum members always help me, so I try to do so too.

    I checked a htaccess from one of my websites and this is the actual blocking list.
    Code:
    Options -Indexes
    RewriteEngine on
    #Block comment spammers, bad bots and some proxies
    RewriteCond %{REMOTE_HOST} 12.226.240.248 [OR]
    RewriteCond %{REMOTE_HOST} 24.111.102.26 [OR]
    RewriteCond %{REMOTE_HOST} 24.117.121.113 [OR]
    RewriteCond %{REMOTE_HOST} 65.30.216.140 [OR]
    RewriteCond %{REMOTE_HOST} 67.87.64.23 [OR]
    RewriteCond %{REMOTE_HOST} 68.12.149.198 [OR]
    RewriteCond %{REMOTE_HOST} 69.139.167.203 [OR]
    RewriteCond %{REMOTE_HOST} 74.95.182.57 [OR]
    RewriteCond %{REMOTE_HOST} 91.121.3.29 [OR]
    RewriteCond %{REMOTE_HOST} 203.94.229.227 [OR]
    RewriteCond %{REMOTE_HOST} 208.96.122.142 [OR]
    RewriteCond %{REMOTE_HOST} 210.0.141.247 [OR]
    RewriteCond %{REMOTE_HOST} 210.197.97.67 [OR]
    RewriteCond %{REMOTE_HOST} 212.179.127.188 [OR]
    RewriteCond %{REMOTE_HOST} 216.246.60.183 [OR]
    RewriteCond %{REMOTE_HOST} 220.156.189.233 [OR]
    RewriteCond %{REMOTE_HOST} 222.36.12.42 [OR]
    # Abuse Agent Blocking
    RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Bolt\ 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot\@yahoo\.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} CazoodleBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Custo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Default\ Browser\ 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^DIIbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^DISCo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} discobot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^eCatch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ecxi [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^FlashGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GetRight [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^GrabNet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Grafula [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} GT::WWW [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} heritrix [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^HMView [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} HTTP::Lite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} IDBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} id-search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} id-search\.org [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InterGET [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^InternetSeer\.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} IRLbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ISC\ Systems\ iRc\ Search\ 2\.1 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Java [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^JetCar [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^larbin [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Link [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} LinksManager.com_bot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} linkwalker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} lwp-trivial [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Maxthon$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} MFC_Tear_Sample [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^microsoft\.url [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Microsoft\ URL\ Control [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Missigua\ Locator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mozilla\.*Indy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Mozilla\.*NEWT [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Navroad [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NearSite [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetAnts [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^NetZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Nutch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Octopus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} panscient.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^pavuk [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} PECL::HTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^PeoplePal [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} PHPCrawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} PleaseCrawl [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^psbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^RealDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^ReGet [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Rippers\ 0 [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} SBIder [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SeaMonkey$ [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^sitecheck\.internetseer\.com [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Snoopy [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Steeler [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SuperBot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Surfbot [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Toata\ dragostea\ mea\ pentru\ diavola [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} URI::Fetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} urllib [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} User-Agent [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Web\ Sucker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} webalta [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebAuto [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} WebCollage [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebCopier [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebFetch [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebReaper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebSauger [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebStripper [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WebZIP [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Wells\ Search\ II [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} WEP\ Search [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Wget [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Widow [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WWW-Mechanize [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Yandex [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} zermelo [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Zeus [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ^Zeus\.*Webster [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} ZyBorg [NC]
    RewriteRule ^.* - [F,L]
    # Bluecoat
    deny from 8.21.4.254
    deny from 65.46.48.192/30
    deny from 65.160.238.176/28
    deny from 85.92.222.0/24
    deny from 206.51.36.0/22
    deny from 216.52.23.0/24
    # cyveillance (already blocked 38.*)
    deny from 38.100.19.8/29
    deny from 38.100.21.0/24
    deny from 38.100.41.64/26
    deny from 38.105.71.0/25
    deny from 38.105.83.0/27
    deny from 38.112.21.140/30
    deny from 38.118.42.32/29
    deny from 65.213.208.128/27
    deny from 65.222.176.96/27
    deny from 65.222.185.72/29
    # Cyberpatrol
    deny from 38.103.17.160/27
    # Internet Identity - Anti-Phishing
    deny from 66.113.96.0/20
    deny from 70.35.113.192/27
    # Ironport
    deny from 204.15.80.0/22
    # Lightspeed Systems Security
    deny from 66.17.15.128/26
    deny from 69.84.207.32/27
    deny from 69.84.207.128/25
    # Layered Technologies
    deny from 72.36.128.0/17
    deny from 72.232.0.0/16
    deny from 72.233.0.0/17
    deny from 216.32.0.0/14
    # M86
    deny from 67.192.231.224/29
    deny from 208.90.236.0/22
    # McAfee-Secure-Computing
    deny from 69.48.241.64/26
    deny from 80.66.0.0/19
    deny from 192.55.214.0/24
    deny from 207.67.117.0/24
    # Phish-Inspector.com
    deny from 209.147.127.208/28
    # Prescient Software, Inc. Phishmongers
    deny from 198.186.190.0/23
    deny from 198.186.192.0/23
    deny from 198.186.194.0/24
    # urlfilterdb
    deny from 207.210.99.32/29
    # websense-in.car1.sandiego1.level3.net
    deny from 4.53.120.22
    # Websense
    deny from 66.194.6.0/24
    deny from 67.117.201.128/28
    deny from 69.67.32.0/20
    deny from 131.191.87.0/24
    deny from 204.15.64.0/21
    deny from 208.80.192.0/21
    deny from 212.62.26.64/27
    deny from 213.168.226.0/24
    deny from 213.168.241.0/30
    deny from 213.168.242.0/30
    deny from 213.236.150.16/28
    # Yandex
    deny from 77.88.0.0/18
    deny from 77.88.22.0/23
    deny from 77.88.24.0/21
    deny from 77.88.24.0/22
    deny from 77.88.28.0/22
    deny from 77.88.36.0/23
    deny from 77.88.42.0/23
    deny from 77.88.44.0/24
    deny from 77.88.50.0/23
    deny from 87.250.224.0/19
    deny from 87.250.230.0/23
    deny from 87.250.252.0/22
    deny from 93.158.128.0/18
    deny from 93.158.137.0/24
    deny from 93.158.144.0/21
    deny from 93.158.144.0/23
    deny from 93.158.146.0/23
    deny from 93.158.148.0/22
    deny from 95.108.128.0/17
    deny from 95.108.128.0/24
    deny from 95.108.152.0/22
    deny from 95.108.216.0/23
    deny from 95.108.240.0/21
    deny from 95.108.248.0/23
    deny from 178.154.128.0/17
    deny from 178.154.160.0/22
    deny from 178.154.164.0/23
    deny from 199.36.240.0/22
    deny from 213.180.192.0/19
    deny from 213.180.204.0/24
    deny from 213.180.206.0/23
    deny from 213.180.209.0/24
    deny from 213.180.218.0/23
    deny from 213.180.220.0/23

  11. The Following User Says Thank You to Sergej_AGR For This Useful Post:

    -Shay- (16 October 2013)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •