One of the many hats I wear, is network security. I still "tinker" a bit and I still see many gapping holes out there in the real world. I do not consider what I am about to publish here as hacks, but I think is a good thing to know about, as these are things some webmaster might want to know.
Lets start with good old robots.txt
1. Robots.txt
Is a great file, all I do is add my directories I do not want indexed there. Look maybe something like this:
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
Hmm, what do we see...An administrator page!! Yay!! now depending on what we want to accomplish we, now know where to start to look. Note this is a real file off of another forum popular in our vertical!!
What else about robots.txt?
Many product sellers / service sellers use robots.txt to protect downloadable material. Next time you see something for sale, for fun check it out, you might find it free, as the robots.txt file contains:
User-agent: *
Disallow: /securedownload/
This is not how you protect things online, so simply changing directories and you have it.
Lets talk more about bots
2. SEO and BOTS
Many forum owners think it a great idea to allow bots to search forums, and this way when a prospect comes by, they have to sign up. Sounds good yes? Sure, But of course, a simple Firefox plugin called "user agent switcher". Add something like this…
Googlebot/2.1 (+http://www.google.com/bot.html)
Or for those using Adsense, and HAVING to allow google in:
Mediapartners-Google/2.1
Either way, you now have free access to these forums, and you lost a customer. Ahhh but it gets better.
3. Selling / Free ebook offers
Haven’t seen it here a lot but the “How to win in roulette” or whatever ebook work great still. Usually people such as myself want this prospects email or whatever, so they have to sign up. Or maybe as a non casino type business we sell clickbank products? Whatever the case, we need to know if we do Not do the robots.txt, what can happen?
Try this in google
inurl:cbreceipt
many people have fixed this but here are some more that still work:
site:*.c om intitle:"Thank You For Your Order" intext:Click Here to Download
site:*.com intitle:"Thank You For Your Purchase" intext:Click Here to Download
intitle:Thank you for your Purchase! intext:PLR OR MRR OR Package OR Bonus
intitle:Thank you for your order! intext:PLR OR MRR
intitle:Thank you for your Purchase! intext:PLR OR MRR
inurl:/thankyou*.html intitle:Thank you for your order! intext:Click Here to Download
inurl:thanks intext:"Thank You For Your Order!" "Click Here" filetype:html
intitle:Thank You For Your Order! intext:Private Label
intitle:Thank You For Your Purchased! intext:Private Label
intext:"Thank You For Your Order" intext:PLR
"Thank You For Your Order!" intext:Master Resell filetype:html
"Thank You For Your Order! Your Credit Card Will Show A Charge From"
intitle:"Thank You For Your Order!" intext:download
intitle:"Thank You For Your Order" intext:Click Here To Download Now
intitle:Thank you for your purchase! intext:Click Here to Download
and the list goes on. So in other words NOT putting the directories in is also a bad idea.
So bottom line? Use some third party package is my suggestion. I use a software for about 150 bucks that I love, but I am not here to promote software, just give a heads up. Is good to know these things to protect yourselves. Good luck to you all !!



LinkBack URL
About LinkBacks
Reply With Quote

