Printable View
I have a site that's using software throwing out some extra pages that I want to get rid of, but they're two folders deep so I wanted to double check the correct syntax.
So, if the site is in this format:
example.com/folder1/folder2/
And I want to block folder2 but not folder1 would it be:
User-agent: *
Disallow: */folder2/
Note: /folder2/ appears at the end of many URLs not just folder 1.
According to Google's Help Files located right here: http://support.google.com/webmasters...rom=40364&rd=1
therefore I believe the * will match any URL that contains folder2 and that your syntax is correct. BTW blocking directly example.com/folder1/folder2/ will also block only folder2, but since folder 2 is at the end of many URLs you wouldn't want to put them all in the robot.txt I guess.Quote:
User-Agent: Googlebot Disallow: /folder2/ In this example only the URLs matching /folder2/ would be disallowed for Googlebot.
Awesome, thanks for the reassurance! :)
One way you can test it is if you have phpsitemapng, you can scan the site, even if its from another server, and then check the files it would add to the sitemap.
Since it's crawler listens to robots.txt directives, it's a pretty good test.
Rick
Universal4
Yes..... the robots.txt is very sophisticated and lifelike :DQuote:
Originally Posted by halliadam