Robots.txt Question

**Doolally** · 7 April 2012, 9:15 am

I probably should know this but having a brain drain today.

My robots.txt file currently looks like this:

User-Agent: *
Allow: /

Now I want to disallow one directory on the server. Does the code now look like :

User-Agent: *
Allow: /

User-agent: *
Disallow: /go/

or should it look like this:

User-Agent: *
Allow: /
Disallow: /go/

**Chips** · 7 April 2012, 10:21 am

Originally Posted by Doolally

or should it look like this:

User-Agent: *
Allow: /
Disallow: /go/

The 2nd one is the winner. You can also test it in big G's webmaster tools, there is a handy app to see if it is written correctly.

**Doolally** · 7 April 2012, 1:14 pm

Thanks Chips!

**michael1981** · 7 April 2012, 5:55 pm

You should also include a link to your sitemap in the robots.txt file. It will help with faster indexing of new and old pages.

Sitemap: http://www.yourwebsite.com

**bboyspyder** · 8 April 2012, 10:54 am

If you use Wordpress you can get the free plugin PC robots.txt

Very handy for fast editing your robots.txt and has included some options already.

**Doolally** · 8 April 2012, 4:25 pm

Thanks Guys!

Nice tip Michael. I wasn't aware you could link to your sitemap from Robots.txt.

Thread: Robots.txt Question

LinkBack

Thread Tools

Search Thread

Display

Robots.txt Question

The Following User Says Thank You to Chips For This Useful Post:

The Following User Says Thank You to michael1981 For This Useful Post:

The Following User Says Thank You to bboyspyder For This Useful Post:

Posting Permissions