Digital Ocean?

**-Shay-** · 18 March 2016, 10:40 am

I've been going through my stats and I notice a lot of direct visits from "Digital Ocean". The visits "appear" (to me) to be a bot (maybe content or image scraper).

I'm pretty sure I want to block them from visiting. Any ideas on how to do so (efficiently via .htaccess) are appreciated.

**universal4** · 18 March 2016, 4:22 pm

I am sure a few others may ring in here on this, does anyone know how much ip space they have?

If it is only a handful of ip's or subnets, you could use ip based blocking.

I am not sure how effective user agent blocking would be.

Rick
Universal4

**universal4** · 20 March 2016, 1:36 am

Actually I have been hammered by a few of their ip's recently as well.

The way I understand it, some have managed to set up vpn's on some of their ip space however I have no conformation of the subnets supposedly used for that.

I have also read tidbits here and there about others having seen some scraping and unusually high bandwith with activity from their ip space.

I do not have any contacts there, but will be keeping an eye out specifically for their subnets.

Rick
Universal4

**TheGooner** · 20 March 2016, 3:29 am

Digital Ocean is a server farm ... very little real human traffic execept maybe from baddies using open proxies?
I see no reason to allow them access.

**-Shay-** · 20 March 2016, 5:17 am

Originally Posted by universal4

Actually I have been hammered by a few of their ip's recently as well.

The way I understand it, some have managed to set up vpn's on some of their ip space however I have no conformation of the subnets supposedly used for that.

I have also read tidbits here and there about others having seen some scraping and unusually high bandwith with activity from their ip space.

I do not have any contacts there, but will be keeping an eye out specifically for their subnets.

Rick
Universal4

This is where I suspect the main source of the content scraping has come from on the site of mine.

Originally Posted by TheGooner

Digital Ocean is a server farm ... very little real human traffic execept maybe from baddies using open proxies?
I see no reason to allow them access.

I definitely want to block them, as they have no valid reason to be on my site.

**-Shay-** · 20 March 2016, 5:24 am

Amazon web servers is probably another one I don't want (or need) visiting my sites.

**universal4** · 20 March 2016, 2:44 pm

After further discussions with others I too am planning on blocking the rest of their subnets that I have not previously blocked.

Shay, you can either block them at the server level by ip address or with htaccess. Doing so at the server level avoids having to block on various sites if you host multiple sites on the same server.

I would be happy to provide you with the known subnets.

Rick
Universal4

**-Shay-** · 20 March 2016, 9:35 pm

Originally Posted by universal4

After further discussions with others I too am planning on blocking the rest of their subnets that I have not previously blocked.

Shay, you can either block them at the server level by ip address or with htaccess. Doing so at the server level avoids having to block on various sites if you host multiple sites on the same server.

I would be happy to provide you with the known subnets.

Rick
Universal4

Thank you Rick. That would be very much appreciated.

**universal4** · 21 March 2016, 1:31 am

I will send this in pm straight away...

Rick
Universal4

**-Shay-** · 21 March 2016, 7:05 am

Thank you very much, Rick.

**Progger** · 21 March 2016, 8:09 am

is digital ocean not a cloud hostng company ?

**-Shay-** · 21 March 2016, 8:20 am

Could hosting, cloud computing - yes.

**universal4** · 21 March 2016, 6:28 pm

Yes they are a cloud hosting company, but it does NOT mean that the web clients using that ip space are doing good things, and when a large number of subnets are caught being bad actors, such as scraping sites and hitting web servers with thousands of requests per second, all with ZERO BENEFIT to the site owners, blocking the subnets is often a good thing.

The same can be said for Amazon, CloudFlare and other hosts that allow this kind of activity.

I find it ironic that almost all of these cloud hosts (as well as a long list of standard web hosts) will block incoming traffic that trigger rate limiting, but they think it's ok to allow their clients to slam other hosts.

Rick
Universal4

**TheGooner** · 21 March 2016, 6:57 pm

You can call it cloud hosting - I call it server farm - what's the difference?

What Digital Ocean are NOT is an ISP for human surfers ... so any access from Digital Ocean IP addresses is unlikely to be valuable.
In fact any access from Digital Ocean is likely to be spamming / scraping / SEO monitoring or any of a hundred other unbeneficial activities.

**universal4** · 21 March 2016, 7:32 pm

I agree, and I tend to use the term "cloud" because I have been conditioned to that since it has kinda been pounded into my brain due to reading too darn many "geek" articles....lol The industry came up with that "buzz word" some years ago since they were looking for another way to term server farm since it seemed to fit into a term used at a Google conference from 2006 although there have been other references to the term as far back as 1996.

I think that you make a really important point that services such as this (no matter what term is used) is there is very few if any, instances where traffic from those ip's would be real users, and certainly not users sitting at home firing up their browsers on a cable, fiber or phone connection.

Now it could be said that some of these ip's could be used by those writing apps that users might use to access content on the web, but if so there certainly should be no reason to hit a site thousands of times per second.

Like TheGooner stated, I honestly think that most of these will be scrapers, pulling web content for THEIR benefit, whether that be to copy the content, or republish it for their own benefit, or as some form of seo benefit, to benefit their users NOT the site owner that are scraping the content of.

Rick
Universal4

**matthewt** · 21 March 2016, 7:44 pm

ORDER ALLOW,DENY
DENY FROM XXX.XXX.XXX.XXX

XXX... being their IP address.

Once you have their IP address, that is all that is required, hopefully.

**universal4** · 21 March 2016, 7:49 pm

Yes matthewt, that works for htaccess if you are willing to add that to each website.

For those that run multiple websites on a dedicated server or vps, you might want to consider a single entry in the server firewall.

For those taking the time to do this, consider how often you may have to continue to add more ip's since they have slightly more than a half million ip's.

Rick
Universal4

**matthewt** · 21 March 2016, 8:00 pm

Maybe it is one of your pay-per-click accounts.

**universal4** · 21 March 2016, 8:52 pm

Pay-per-click traffic would never generate thousands of requests per second.....never.

Rick
Universal4

**-Shay-** · 22 March 2016, 5:37 am

Originally Posted by universal4

Yes they are a cloud hosting company, but it does NOT mean that the web clients using that ip space are doing good things, and when a large number of subnets are caught being bad actors, such as scraping sites and hitting web servers with thousands of requests per second, all with ZERO BENEFIT to the site owners, blocking the subnets is often a good thing.

The same can be said for Amazon, CloudFlare and other hosts that allow this kind of activity.

I find it ironic that almost all of these cloud hosts (as well as a long list of standard web hosts) will block incoming traffic that trigger rate limiting, but they think it's ok to allow their clients to slam other hosts.

Rick
Universal4

Exactly. Add Google Cloud to this list (Amazon, Cloudflare, & Digital Ocean). All of them have the same (numerous, less than 1 second visits multiple times daily in groups).

Thank you to everyone who has provided input & specific advice on this both in the thread and via PM.

Thread: Digital Ocean?

LinkBack

Thread Tools

Search Thread

Display

Digital Ocean?

The Following 2 Users Say Thank You to -Shay- For This Useful Post:

The Following 3 Users Say Thank You to universal4 For This Useful Post:

The Following 2 Users Say Thank You to universal4 For This Useful Post:

The Following User Says Thank You to -Shay- For This Useful Post:

The Following User Says Thank You to -Shay- For This Useful Post:

The Following 2 Users Say Thank You to universal4 For This Useful Post:

The Following User Says Thank You to -Shay- For This Useful Post:

The Following 2 Users Say Thank You to universal4 For This Useful Post:

The Following User Says Thank You to -Shay- For This Useful Post:

The Following User Says Thank You to -Shay- For This Useful Post:

The Following User Says Thank You to universal4 For This Useful Post:

The Following 2 Users Say Thank You to TheGooner For This Useful Post:

The Following User Says Thank You to universal4 For This Useful Post:

The Following User Says Thank You to universal4 For This Useful Post:

Posting Permissions