Google ignores robots.txt?

[vmlinuz]

I have a domain, wettpedia.net which has exactly the same content as our main site sportfogadas.org. It will be translated at some point, that's the reason it's uploaded and visible online.
I didn't want Google to index it, so I created a robots.txt disallowing everything, right after uploading.

My question is, how did Google ignore it? In webmaster tools it shows everything is blocked correctly, but even so if I do a site: xxwww.wettpedia.net search it shows up indexed pages. It kinda sucks, as it is duplicate content.

Now I'm thinking about blocking Googlebot completely, based on it's user-agent string, giving a 404 error for each page it tries to index, but it will probably take some time until it disappears from the index.

[Chips]

That is strange, I am in the process of re-designing one of my sites and am doing it all in one folder off the root. I blocked robots from that folder and none of the new pages are in the index. Also, when I begin a new review, I usually include that page in the robots.txt as Disallow and it never gets crawled.

Did you have a mis-spelling in robots.txt?

I am sure that the Disallow: /xx.xxx works as I forgot to delete one page from the list and included it in the xml map and it generated an error as being blocked.

I list it as "Disallow: /newpagetitle.html" or for the folder "Disallow: /folder"

Maybe you can do a page removal request in Webmaster tools to get them off the index sooner.

[vmlinuz]

Here's the content of robots.txt:

User-agent: *
Disallow: /

If i do a removal request I think it could be harder to get it back in the index later.

[Chips]

Yes I think you are correct, it would be hard to get "re-indexed". I thought that the pages you were doing were to be moved at a later date. My bad, sorry to offer incorrect advise, (oops). The robot.txt is on the money too. very odd that it was ignored by the spiders.

[vmlinuz]

I guess I'll have to "ban" Googlebot in some other way, by showing a 404 page insted of the real content until the site gets finished.

[vmlinuz]

This should do it:

if(eregi("bot",$_SERVER['HTTP_USER_AGENT'])){
	header('HTTP/1.1 403 Forbidden');
	die("This page shouldn't be indexed yet!");
}

[pgaming]

Should be able to block googlebot via Robot.txt by using the following syntax:

User-agent: Googlebot
Disallow: /

This is intended for Google only.

greek39

[vmlinuz]

I know that, but using * as User-agent should block it, too.
As I said earlier, it shows as blocked in webmaster tools, but still has indexed pages. That's why from now on all bots get a 403 Forbidden message insted of the content.

[zuhrunezz]

I have a domain, wettpedia.net which has exactly the same content as our main site sportfogadas.org. It will be translated at some point, that's the reason it's uploaded and visible online.
I didn't want Google to index it, so I created a robots.txt disallowing everything, right after uploading.

My question is, how did Google ignore it? In webmaster tools it shows everything is blocked correctly, but even so if I do a site: xxwww.wettpedia.net search it shows up indexed pages. It kinda sucks, as it is duplicate content.

Now I'm thinking about blocking Googlebot completely, based on it's user-agent string, giving a 404 error for each page it tries to index, but it will probably take some time until it disappears from the index.

The robots.txt should look like this:
User-Agent: *
Allow: /

Sitemap: http://yourdomainname/sitemap.xml

[vmlinuz]

The robots.txt should look like this:
User-Agent: *
Allow: /

Sitemap: http://yourdomainname/sitemap.xml

Yes, if you want your pages included. I DON'T want that yet.

[zuhrunezz]

then tha easiest way is NOT to have robots.txt

[vmlinuz]

Not true, read some more about it.

[zuhrunezz]

then use it like this

User-agent: *
Disallow: /path_1/
Disallow: /example/
Disallow: /~joe/

[bbonline]

This is pretty much relevant to this issue. http://www.davidnaylor.co.uk/google-...robotstxt.html

[seoinxs]

I have seen that Google ignore robots.txt exclusions on more than one occasioned when the target is heavily linked.

But the code that vmlinuz used I think it is better to avoid ignorance of robots.txt in Google.