Do you use robots exclusion standard on your sites?

[MichaelCorfman]

Quoting from Wikipedia,

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

I've never asked about the use of robots.txt files before, so this week I decided it was time to do a poll on the use of robots.txt files by affiliates. Do you use them on some of your sites, all of your sites, or none of your sites?

Besides voting in the poll share your thoughts about robots.txt files. If you do use them, what are your objectives? For example, I know some affiliates use them to explicitly block links to urls that redirect as affiliate links in addition to indicating nofollow for those links.

Michael

[Cash Bonus]

I use them on some of my websites. I intend to use them on all of my sites as soon as possible.

[TheGooner]

yes - but most scummy robots don't respect them - and you need to block them via .htaccess and user agent.

[casinoduende]

I use the All in One SEO Pack plugin. The plugin blocks bad bots. And you can also modify the robots.txt file

Enviado desde mi SM-J510FN mediante Tapatalk

[good4bonus]

I use them to really allow or disallow some bots of spider the site, however, bad ones do not respec it

[Tony]

Yes I use robots.text files on all of my sites.

[Strider1973]

I use robots.txt on all of my sites to
- block aff links
- point spiders to the sitemap

[MMM]

Use as well:

block aff links
point to sitemap
and also noindex some empty tags and such.

[Jamson]

I use on all my sites)))

[wonderpunter]

use a modified robot txt.. mainly to stop search results being indexed

[PROFRBcom]

use a modified robot txt.. mainly to stop search results being indexed

Can you expound upon this? I'm not sure I understand exactly what you mean...

[sweetbet]

I use them mainly for google and bing, but I notice that many dodgy robots simply ignore them.