Page 1 of 2 12 LastLast
Results 1 to 20 of 37

Thread: Digital Ocean?

  1. #1
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default Digital Ocean?

    I've been going through my stats and I notice a lot of direct visits from "Digital Ocean". The visits "appear" (to me) to be a bot (maybe content or image scraper).

    I'm pretty sure I want to block them from visiting. Any ideas on how to do so (efficiently via .htaccess) are appreciated.

  2. The Following 2 Users Say Thank You to -Shay- For This Useful Post:

    Roulette Zeitung (21 March 2016), Vrindavan (24 March 2016)

  3. #2
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    I am sure a few others may ring in here on this, does anyone know how much ip space they have?

    If it is only a handful of ip's or subnets, you could use ip based blocking.

    I am not sure how effective user agent blocking would be.

    Rick
    Universal4

  4. The Following 3 Users Say Thank You to universal4 For This Useful Post:

    -Shay- (18 March 2016), Roulette Zeitung (21 March 2016), Vrindavan (24 March 2016)

  5. #3
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    Actually I have been hammered by a few of their ip's recently as well.

    The way I understand it, some have managed to set up vpn's on some of their ip space however I have no conformation of the subnets supposedly used for that.

    I have also read tidbits here and there about others having seen some scraping and unusually high bandwith with activity from their ip space.

    I do not have any contacts there, but will be keeping an eye out specifically for their subnets.

    Rick
    Universal4

  6. The Following 2 Users Say Thank You to universal4 For This Useful Post:

    -Shay- (20 March 2016), Roulette Zeitung (21 March 2016)

  7. #4
    TheGooner's Avatar
    TheGooner is offline Private Member
    Join Date
    March 2007
    Location
    New Zealand
    Posts
    4,264
    Thanks
    1,952
    Thanked 4,213 Times in 2,005 Posts

    Default

    Digital Ocean is a server farm ... very little real human traffic execept maybe from baddies using open proxies?
    I see no reason to allow them access.

  8. #5
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default

    Quote Originally Posted by universal4 View Post
    Actually I have been hammered by a few of their ip's recently as well.

    The way I understand it, some have managed to set up vpn's on some of their ip space however I have no conformation of the subnets supposedly used for that.

    I have also read tidbits here and there about others having seen some scraping and unusually high bandwith with activity from their ip space.

    I do not have any contacts there, but will be keeping an eye out specifically for their subnets.

    Rick
    Universal4
    This is where I suspect the main source of the content scraping has come from on the site of mine.

    Quote Originally Posted by TheGooner View Post
    Digital Ocean is a server farm ... very little real human traffic execept maybe from baddies using open proxies?
    I see no reason to allow them access.
    I definitely want to block them, as they have no valid reason to be on my site.

  9. The Following User Says Thank You to -Shay- For This Useful Post:

    Roulette Zeitung (21 March 2016)

  10. #6
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default

    Amazon web servers is probably another one I don't want (or need) visiting my sites.

  11. The Following User Says Thank You to -Shay- For This Useful Post:

    Roulette Zeitung (21 March 2016)

  12. #7
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    After further discussions with others I too am planning on blocking the rest of their subnets that I have not previously blocked.

    Shay, you can either block them at the server level by ip address or with htaccess. Doing so at the server level avoids having to block on various sites if you host multiple sites on the same server.

    I would be happy to provide you with the known subnets.

    Rick
    Universal4

  13. The Following 2 Users Say Thank You to universal4 For This Useful Post:

    -Shay- (20 March 2016), Roulette Zeitung (21 March 2016)

  14. #8
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default

    Quote Originally Posted by universal4 View Post
    After further discussions with others I too am planning on blocking the rest of their subnets that I have not previously blocked.

    Shay, you can either block them at the server level by ip address or with htaccess. Doing so at the server level avoids having to block on various sites if you host multiple sites on the same server.

    I would be happy to provide you with the known subnets.

    Rick
    Universal4
    Thank you Rick. That would be very much appreciated.

  15. The Following User Says Thank You to -Shay- For This Useful Post:

    Roulette Zeitung (21 March 2016)

  16. #9
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    I will send this in pm straight away...

    Rick
    Universal4

  17. The Following 2 Users Say Thank You to universal4 For This Useful Post:

    -Shay- (21 March 2016), Roulette Zeitung (21 March 2016)

  18. #10
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default

    Thank you very much, Rick.

  19. The Following User Says Thank You to -Shay- For This Useful Post:

    Roulette Zeitung (21 March 2016)

  20. #11
    Progger's Avatar
    Progger is offline Public Member
    Join Date
    November 2014
    Location
    New Zealand
    Posts
    1,422
    Thanks
    282
    Thanked 930 Times in 583 Posts

    Default

    is digital ocean not a cloud hostng company ?

  21. #12
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default

    Could hosting, cloud computing - yes.

  22. The Following User Says Thank You to -Shay- For This Useful Post:

    Roulette Zeitung (21 March 2016)

  23. #13
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    Yes they are a cloud hosting company, but it does NOT mean that the web clients using that ip space are doing good things, and when a large number of subnets are caught being bad actors, such as scraping sites and hitting web servers with thousands of requests per second, all with ZERO BENEFIT to the site owners, blocking the subnets is often a good thing.

    The same can be said for Amazon, CloudFlare and other hosts that allow this kind of activity.

    I find it ironic that almost all of these cloud hosts (as well as a long list of standard web hosts) will block incoming traffic that trigger rate limiting, but they think it's ok to allow their clients to slam other hosts.

    Rick
    Universal4

  24. The Following User Says Thank You to universal4 For This Useful Post:

    -Shay- (21 March 2016)

  25. #14
    TheGooner's Avatar
    TheGooner is offline Private Member
    Join Date
    March 2007
    Location
    New Zealand
    Posts
    4,264
    Thanks
    1,952
    Thanked 4,213 Times in 2,005 Posts

    Default

    You can call it cloud hosting - I call it server farm - what's the difference?

    What Digital Ocean are NOT is an ISP for human surfers ... so any access from Digital Ocean IP addresses is unlikely to be valuable.
    In fact any access from Digital Ocean is likely to be spamming / scraping / SEO monitoring or any of a hundred other unbeneficial activities.
    Last edited by TheGooner; 21 March 2016 at 8:58 pm. Reason: added UN to the word beneficial - I don't think scrapers are a good thing ;-)

  26. The Following 2 Users Say Thank You to TheGooner For This Useful Post:

    -Shay- (21 March 2016), universal4 (21 March 2016)

  27. #15
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    I agree, and I tend to use the term "cloud" because I have been conditioned to that since it has kinda been pounded into my brain due to reading too darn many "geek" articles....lol The industry came up with that "buzz word" some years ago since they were looking for another way to term server farm since it seemed to fit into a term used at a Google conference from 2006 although there have been other references to the term as far back as 1996.

    I think that you make a really important point that services such as this (no matter what term is used) is there is very few if any, instances where traffic from those ip's would be real users, and certainly not users sitting at home firing up their browsers on a cable, fiber or phone connection.

    Now it could be said that some of these ip's could be used by those writing apps that users might use to access content on the web, but if so there certainly should be no reason to hit a site thousands of times per second.

    Like TheGooner stated, I honestly think that most of these will be scrapers, pulling web content for THEIR benefit, whether that be to copy the content, or republish it for their own benefit, or as some form of seo benefit, to benefit their users NOT the site owner that are scraping the content of.

    Rick
    Universal4

  28. The Following User Says Thank You to universal4 For This Useful Post:

    -Shay- (21 March 2016)

  29. #16
    matthewt's Avatar
    matthewt is offline On Vacation
    Join Date
    July 2006
    Location
    US
    Posts
    526
    Blog Entries
    2
    Thanks
    1,003
    Thanked 141 Times in 112 Posts

    Default

    ORDER ALLOW,DENY
    DENY FROM XXX.XXX.XXX.XXX


    XXX... being their IP address.


    Once you have their IP address, that is all that is required, hopefully.

  30. #17
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    Yes matthewt, that works for htaccess if you are willing to add that to each website.

    For those that run multiple websites on a dedicated server or vps, you might want to consider a single entry in the server firewall.

    For those taking the time to do this, consider how often you may have to continue to add more ip's since they have slightly more than a half million ip's.

    Rick
    Universal4

  31. #18
    matthewt's Avatar
    matthewt is offline On Vacation
    Join Date
    July 2006
    Location
    US
    Posts
    526
    Blog Entries
    2
    Thanks
    1,003
    Thanked 141 Times in 112 Posts

    Default

    Maybe it is one of your pay-per-click accounts.

  32. #19
    universal4's Avatar
    universal4 is offline Forum Administrator
    Join Date
    July 2003
    Location
    Courage is being scared to death...and saddling up anyway. John Wayne
    Posts
    28,096
    Thanks
    2,251
    Thanked 7,943 Times in 5,010 Posts

    Default

    Pay-per-click traffic would never generate thousands of requests per second.....never.

    Rick
    Universal4

  33. The Following User Says Thank You to universal4 For This Useful Post:

    -Shay- (22 March 2016)

  34. #20
    -Shay- is offline Public Member
    Join Date
    November 2012
    Posts
    3,062
    Thanks
    12,211
    Thanked 3,133 Times in 1,686 Posts

    Default

    Quote Originally Posted by universal4 View Post
    Yes they are a cloud hosting company, but it does NOT mean that the web clients using that ip space are doing good things, and when a large number of subnets are caught being bad actors, such as scraping sites and hitting web servers with thousands of requests per second, all with ZERO BENEFIT to the site owners, blocking the subnets is often a good thing.

    The same can be said for Amazon, CloudFlare and other hosts that allow this kind of activity.

    I find it ironic that almost all of these cloud hosts (as well as a long list of standard web hosts) will block incoming traffic that trigger rate limiting, but they think it's ok to allow their clients to slam other hosts.

    Rick
    Universal4
    Exactly. Add Google Cloud to this list (Amazon, Cloudflare, & Digital Ocean). All of them have the same (numerous, less than 1 second visits multiple times daily in groups).

    Thank you to everyone who has provided input & specific advice on this both in the thread and via PM.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •