Robots.txt syntax question

[baldidiot]

I have a site that's using software throwing out some extra pages that I want to get rid of, but they're two folders deep so I wanted to double check the correct syntax.

So, if the site is in this format:

example.com/folder1/folder2/

And I want to block folder2 but not folder1 would it be:

User-agent: *
Disallow: */folder2/

Note: /folder2/ appears at the end of many URLs not just folder 1.

[xecutable]

According to Google's Help Files located right here: http://support.google.com/webmasters...rom=40364&rd=1

User-Agent: Googlebot Disallow: /folder2/ In this example only the URLs matching /folder2/ would be disallowed for Googlebot.

therefore I believe the * will match any URL that contains folder2 and that your syntax is correct. BTW blocking directly example.com/folder1/folder2/ will also block only folder2, but since folder 2 is at the end of many URLs you wouldn't want to put them all in the robot.txt I guess.

[baldidiot]

Awesome, thanks for the reassurance!

[universal4]

One way you can test it is if you have phpsitemapng, you can scan the site, even if its from another server, and then check the files it would add to the sitemap.

Since it's crawler listens to robots.txt directives, it's a pretty good test.

Rick
Universal4

[xecutable]

Robots are becoming increasingly sophisticated and lifelike. Learn about robots, the latest robot technology and how close we are to artificial intelligence.

thanks for sharing.

Yes..... the robots.txt is very sophisticated and lifelike