A thread to discuss the current state of scraping and spamming.
I'm actively trying to stop scraping, spamming and scamming of our website by blocking bots and hosted servers ... and making some good progress by blocking bots by user agent, and also the IP address ranges of high profile server farms.
I have a few logging systems on my site - and a few honey pots - that humans would be highly unlikely to reach - but that seem to be good at catching bots. I then log their details and block the ip address, sometimes the server range, and sometimes the user agent.
Interestingly though when you block these guys, you will then get an entry on the webserver error log each time that they hit ...
Currently I am seeing :
- Amazon AWS trying 500 times an hour
- DediPower (UK) trying 30 times an hour
- XL Host 10-20 times an hour
- Various script kiddies looking for Xmlrpc or wp-login
All up it's over 1000 attempts an hour from unwanted bots / hackers / spammers / scrapers.
Some of it is possible human traffic using open proxies ...
Questions :
1/ Does anyone else monitor this stuff?
2/ What do you do to combat them?
3/ What are they actually trying to do?
4/ Do you block open proxies? They could be people - but usually they're not.
Main Question :
What is the rest of the GPWA doing out there to monitor this "bad" traffic and restrict access?
