View Poll Results: Do you use robots exclusion standard on your sites?

Voters
13. You may not vote on this poll
  • I use robots.text files on all of my sites.

    11 84.62%
  • I use robots.txt files on some of my sites but not on others.

    2 15.38%
  • I don't use robots.txt files on any of my sites.

    0 0%
Results 1 to 12 of 12
  1. #1
    MichaelCorfman's Avatar
    MichaelCorfman is offline GPWA Executive Director
    Join Date
    June 2004
    Location
    Newton, MA
    Posts
    3,972
    Thanks
    797
    Thanked 4,786 Times in 1,589 Posts

    Question Do you use robots exclusion standard on your sites?

    Quoting from Wikipedia,

    Quote Originally Posted by Wikipedia
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.


    I've never asked about the use of robots.txt files before, so this week I decided it was time to do a poll on the use of robots.txt files by affiliates. Do you use them on some of your sites, all of your sites, or none of your sites?

    Besides voting in the poll share your thoughts about robots.txt files. If you do use them, what are your objectives? For example, I know some affiliates use them to explicitly block links to urls that redirect as affiliate links in addition to indicating nofollow for those links.

    Michael
    GPWA Executive Director, Casino City CEO, Friend to the Village Idiot

    Resources for Affiliates: iGamingDirectory.com, iGamingAffiliatePrograms.com, GamingMeets.com

  2. The Following User Says Thank You to MichaelCorfman For This Useful Post:

    Cash Bonus (9 May 2018)

  3. #2
    Cash Bonus's Avatar
    Cash Bonus is offline Private Member
    Join Date
    May 2014
    Posts
    3,014
    Thanks
    8,348
    Thanked 2,088 Times in 1,429 Posts

    Default

    I use them on some of my websites. I intend to use them on all of my sites as soon as possible.

  4. #3
    TheGooner's Avatar
    TheGooner is offline Private Member
    Join Date
    March 2007
    Location
    New Zealand
    Posts
    4,038
    Thanks
    1,797
    Thanked 3,871 Times in 1,859 Posts

    Default

    yes - but most scummy robots don't respect them - and you need to block them via .htaccess and user agent.

  5. The Following User Says Thank You to TheGooner For This Useful Post:

    RMRkT (14 May 2018)

  6. #4
    casinoduende's Avatar
    casinoduende is offline Public Member
    Join Date
    August 2017
    Location
    Spain
    Posts
    65
    Thanks
    24
    Thanked 15 Times in 13 Posts

    Default

    I use the All in One SEO Pack plugin. The plugin blocks bad bots. And you can also modify the robots.txt file

    Enviado desde mi SM-J510FN mediante Tapatalk

  7. #5
    good4bonus's Avatar
    good4bonus is offline Private Member
    Join Date
    May 2013
    Location
    portugal
    Posts
    167
    Thanks
    28
    Thanked 105 Times in 66 Posts

    Default

    I use them to really allow or disallow some bots of spider the site, however, bad ones do not respec it

  8. #6
    Tony is offline Private Member
    Join Date
    March 2005
    Posts
    500
    Thanks
    27
    Thanked 131 Times in 89 Posts

    Default

    Yes I use robots.text files on all of my sites.

  9. #7
    Strider1973's Avatar
    Strider1973 is online now Private Member
    Join Date
    November 2012
    Posts
    194
    Thanks
    132
    Thanked 125 Times in 81 Posts

    Default

    I use robots.txt on all of my sites to
    - block aff links
    - point spiders to the sitemap

  10. #8
    MMM
    MMM is offline Private Member
    Join Date
    October 2014
    Posts
    1,087
    Thanks
    269
    Thanked 493 Times in 342 Posts

    Default

    Use as well:

    block aff links
    point to sitemap
    and also noindex some empty tags and such.
    Best casinos to play slot machines online for real money. Reviews of best Real Money Casinos online.
    Check OnlineBlackjackExplorer for ratings of the best casinos to play blackjack online. Which games offer the lowest house edge, as well as free blackjack games, live dealer and mobile blackjack sites.

  11. #9
    Jamson's Avatar
    Jamson is offline Public Member
    Join Date
    April 2018
    Location
    Helsinki
    Posts
    10
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    I use on all my sites)))

  12. #10
    wonderpunter's Avatar
    wonderpunter is offline Private Member
    Join Date
    August 2013
    Posts
    2,117
    Blog Entries
    5
    Thanks
    382
    Thanked 1,516 Times in 895 Posts

    Default

    use a modified robot txt.. mainly to stop search results being indexed

  13. #11
    PROFRBcom's Avatar
    PROFRBcom is offline Private Member
    Join Date
    April 2013
    Posts
    1,366
    Thanks
    1,215
    Thanked 819 Times in 530 Posts

    Default

    Quote Originally Posted by wonderpunter View Post
    use a modified robot txt.. mainly to stop search results being indexed
    Can you expound upon this? I'm not sure I understand exactly what you mean...

  14. #12
    sweetbet's Avatar
    sweetbet is offline Private Member
    Join Date
    November 2012
    Posts
    2,693
    Blog Entries
    5
    Thanks
    840
    Thanked 1,504 Times in 1,041 Posts

    Default

    I use them mainly for google and bing, but I notice that many dodgy robots simply ignore them.
    Sweet Bet - Reviews of reputable online casinos, poker sites, sportsbooks & bingo halls
    USA Online Casinos | Canadian Online Casinos | Bitcoin Casinos | Live Dealer Casinos | Free Spin Casinos | US Online Casinos | Just HODL Crypto

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •