Results 1 to 8 of 8

Thread: robots.txt

  1. #1
    TravG's Avatar
    TravG is offline Private Member
    Join Date
    September 2008
    Posts
    2,069
    Thanks
    34
    Thanked 179 Times in 131 Posts

    Default robots.txt

    Anyone have any idea why in webmaster tools it shows files that I do not have restricted by my robots.txt file show as restricted?

    For example my playersonly review is said to be restricted but nowhere on the actual robots.txt file do I even mention this folder?
    Live Casino USA - the best USA live online casinos. Play USA online slots and other casino games like USA online blackjack. Play See USAlegalcasinos.com to find the best USA online casino. Want to play USA online poker? Find the best poker sites at 4DeucesPoker.com.

  2. #2
    GaryTheScubaGuy's Avatar
    GaryTheScubaGuy is offline In Memorium, 1966-2018
    Join Date
    December 2008
    Location
    Malta
    Posts
    745
    Blog Entries
    22
    Thanks
    223
    Thanked 430 Times in 205 Posts

    Default

    There is only 2 places you can do this: webmaster tools and the actual robots.txt file.

    If you didn't manually exclude google its the file. Can you post your robots.txt file here without your actual url?

  3. #3
    TravG's Avatar
    TravG is offline Private Member
    Join Date
    September 2008
    Posts
    2,069
    Thanks
    34
    Thanked 179 Times in 131 Posts

    Default

    Here is my robots file copied directly. Does anything look amiss?

    User-Agent: *
    Disallow: /banners/
    Disallow: /out
    Disallow: /out/
    Disallow: /play
    Disallow: /play/
    Allow: /
    Live Casino USA - the best USA live online casinos. Play USA online slots and other casino games like USA online blackjack. Play See USAlegalcasinos.com to find the best USA online casino. Want to play USA online poker? Find the best poker sites at 4DeucesPoker.com.

  4. #4
    Squigley is offline Private Member
    Join Date
    April 2006
    Location
    Spain
    Posts
    101
    Thanks
    25
    Thanked 19 Times in 13 Posts

    Default

    When I run your robots.txt file through an online checker it throws an error on the user-agent.

  5. #5
    GaryTheScubaGuy's Avatar
    GaryTheScubaGuy is offline In Memorium, 1966-2018
    Join Date
    December 2008
    Location
    Malta
    Posts
    745
    Blog Entries
    22
    Thanks
    223
    Thanked 430 Times in 205 Posts

  6. #6
    GaryTheScubaGuy's Avatar
    GaryTheScubaGuy is offline In Memorium, 1966-2018
    Join Date
    December 2008
    Location
    Malta
    Posts
    745
    Blog Entries
    22
    Thanks
    223
    Thanked 430 Times in 205 Posts

    Default

    You may find it easier to create a new robots.txt file here; http://www.searchenginepromotionhelp...ts-creator.php

  7. The Following 2 Users Say Thank You to GaryTheScubaGuy For This Useful Post:


  8. #7
    TravG's Avatar
    TravG is offline Private Member
    Join Date
    September 2008
    Posts
    2,069
    Thanks
    34
    Thanked 179 Times in 131 Posts

    Default

    Thanks Gary - That worked - when I run it through the tool now I receive no errors! Thanks again
    Live Casino USA - the best USA live online casinos. Play USA online slots and other casino games like USA online blackjack. Play See USAlegalcasinos.com to find the best USA online casino. Want to play USA online poker? Find the best poker sites at 4DeucesPoker.com.

  9. #8
    GaryTheScubaGuy's Avatar
    GaryTheScubaGuy is offline In Memorium, 1966-2018
    Join Date
    December 2008
    Location
    Malta
    Posts
    745
    Blog Entries
    22
    Thanks
    223
    Thanked 430 Times in 205 Posts

    Default

    Now go to webmaster central and request a crawl so it pushes everything out quicker.

    You may even want to ping it here

    Then post it on a couple social networks like Facebook and Myspace and ping those as well.

    You'll be back up to par in no time.

    Do a site:yoursite search on google to see if its there or assign a google alert to the page you are targeting to see when Google reindexes it.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •