Duplicate content from test sub domain

[casionmark]

Can anyone suggest the best way to resolve this:

I am working on a site with a test version on a sub domain - test.xxx.com. The test site is set to no index and blocking crawlers. Somehow some pages from the test sites have been crawled and indexed and are now showing instead of the real site. Might be the hosting company or developer switching the no index and crawler block off temporarily during an upload.

Whatever the reason, I need to fix the issue and get the pages from the test sub domain removed from the index. Would you delete the test site entirely or remove the block on crawlers temporarily to allow google to go in and see that all pages on the sub domain are now no-index? Or is there another solution?

Thanks,

[chaumi]

My first instinct would be to remove the block from the test site pages (if you are able to do it on a page by page basis), 301 those pages to the appropriate real site pages, then try to force a crawl of the test site pages. If you don't have GSC on the test site, you might need to run them through an external indexer (if you don't have access to one, I have spare credits on Omega, happy to try that just send URLs).

This is all fwiw, ofc. I've never encountered this problem. But seems a logical approach, though even if it works, how long it'll take is a big unknown.

[casionmark]

Thanks, its not in GSC but I can get google to crawl pages using another tool. Currently crawlers are blocked from the whole site through robots.txt so would need unblock the whole site i think.

Interested to know why you think that rather than simply delete the sub domain entirely? Once google has checked and seen its not there a few times wouldn't it just remove any URLs from the index?

[universal4]

If you have yoast or other be sure to set it to no index. (which you may have done already in the wp settings)

Consider adding a robots.txt to disallow all. (which you may have done already)

Consider an htaccess to disallow all IP's except yours.

If it is a dedicated or virtual host or even a regular cpanel or cwp panel you should also be able to block at the server level.

When working on a test site there should be no reason to allow any access except from an ip that is actually working on the site.

In order to have a subdomain that answers, there is in fact an a record in dns for that sub, and likely the sitemap is being read, telling a bot to go awy is never enough, you must stop the bot from reading the content at all. Since the site must answer in order to effectively work on it, clearly it will answer on the index page.

Rick
Universal4

[chaumi]

Interested to know why you think that rather than simply delete the sub domain entirely? Once google has checked and seen its not there a few times wouldn't it just remove any URLs from the index?

Hmmm. Well I'd be thinking...

Why did G put these pages into the index instead of my originals (irrelevant of how they knew about them)
And subsequent to that, what does that mean for those old wanted pages? Has the algorithm sandboxed them, marked them as crap for some reason? A N other?


The risk being you delete the unwanted test pages/site...but the original pages don't go back in.

So my thoughts were to specifically point out what you want done ie Here Google, these pages are 301d to pages you used to rank, you can safely and cheaply put them back where they were, because the pages you chose to rank don't exist any more anyway

Instead of doing it passively ie find out for yourself Google that these pages you ranked don't exist any more, drop them...and while you're at it, please can you put back the ones you dropped.

It seems that with the 301 option, you're controlling and directing (to an extent). Feels like the least risky option to me.

[universal4]

Hmmm. Well I'd be thinking...

Why did G put these pages into the index instead of my originals (irrelevant of how they knew about them)

Simple, 1st published.

If you write the page on the test site, then later add it to the live site, the page on test site has an older timestamp, so it is likely considered the original (because it is) with the page on the live site is in fact copied.

Keep google out of the test site.

Rick
Universal4

[allfreechips]

Why not add the test site to searchconsole and remove it from the index?

[slotsheaven]

I would also recommend using GSC, might be the best solution

[casionmark]

Simple, 1st published.

If you write the page on the test site, then later add it to the live site, the page on test site has an older timestamp, so it is likely considered the original (because it is) with the page on the live site is in fact copied.

Keep google out of the test site.

Rick
Universal4

In this case the published date on the article in the test sub-domain and the main domain is the same. The site was first published in 2021 and articles have been added since. We copied the site to the test sub-domain to make changes - design / WP theme. Then, when changes had been made, we pushed it live. Seems that when you copy a site like that it retains the original published dates.

I've noticed now that google has indexed the test site pages and de-indexed the main site equivalents. I had thought they were competing in the index. But no.

[casionmark]

Why not add the test site to searchconsole and remove it from the index?

You mean using Removals? I was reluctant to add the sub domain to search console - was thinking that would raise its profile to google. But maybe its best solution.

[casionmark]

Hmmm. Well I'd be thinking...

Why did G put these pages into the index instead of my originals (irrelevant of how they knew about them)
And subsequent to that, what does that mean for those old wanted pages? Has the algorithm sandboxed them, marked them as crap for some reason? A N other?


The risk being you delete the unwanted test pages/site...but the original pages don't go back in.

So my thoughts were to specifically point out what you want done ie Here Google, these pages are 301d to pages you used to rank, you can safely and cheaply put them back where they were, because the pages you chose to rank don't exist any more anyway

Instead of doing it passively ie find out for yourself Google that these pages you ranked don't exist any more, drop them...and while you're at it, please can you put back the ones you dropped.

It seems that with the 301 option, you're controlling and directing (to an extent). Feels like the least risky option to me.

This does make sense.

To do this would i need to remove the crawler block of all site from robots as well as no-index from the specific pages?

[baldidiot]

You mean using Removals? I was reluctant to add the sub domain to search console - was thinking that would raise its profile to google. But maybe its best solution.

Not if you're using it to remove it from the index. You'll need to also need to noindex the pages or block google in robots.txt for it to comply though (although google has a habit of ignoring robots.txt).

Moving forward the safest thing to do would be to restrict access to the test domain. Either limit by IP or stick it behind a log in.

[casionmark]

Not if you're using it to remove it from the index. You'll need to also need to noindex the pages or block google in robots.txt for it to comply though (although google has a habit of ignoring robots.txt).

Moving forward the safest thing to do would be to restrict access to the test domain. Either limit by IP or stick it behind a log in.

OK so,

- remove URLs using search console. Can this be done even when a page is no-indexed and the site is blocked from crawlers? And can you remove an entire site in one go? Would be easier.

Then ...

- limit sub domain by IP in htaccesss
- keep robots block and no-index across all pages

Thanks

[baldidiot]

- remove URLs using search console. Can this be done even when a page is no-indexed and the site is blocked from crawlers? And can you remove an entire site in one go? Would be easier.

You can remove based on path, so you're basically just removing everything with the path "/". Although I think for the whole site you'll need to enter the domain as well:

"To block an entire site, use a URL like this: https://example.com/ - see:https://support.google.com/webmaster...-with-a-prefix

And yes, it is fine if you've noindexed it - to keep the urls out of the index long term you'll need to stop serving the content to google anyway as the removal request is only temporary.

Then ...

- limit sub domain by IP in htaccesss
- keep robots block and no-index across all pages

If you're restricting via IP then you don't need to also block in robots.txt or noindex - google won't even be able to see that you've done this as it won't be able to load the site.

[casionmark]

You can remove based on path, so you're basically just removing everything with the path "/". Although I think for the whole site you'll need to enter the domain as well:

"To block an entire site, use a URL like this: https://example.com/ - see:https://support.google.com/webmaster...-with-a-prefix

And yes, it is fine if you've noindexed it - to keep the urls out of the index long term you'll need to stop serving the content to google anyway as the removal request is only temporary.





If you're restricting via IP then you don't need to also block in robots.txt or noindex - google won't even be able to see that you've done this as it won't be able to load the site.

Thats great, thanks!

[baldidiot]

The only other thing you may want to take into account is whether the URL's that are indexed on the test site are getting any traffic. If this is the case then you may want to 301 them to the main site so that you don't lose that traffic. It could also help google show the correct URL in the search.

So the other option is to use a wildcard redirect from the test subdomain to the main subdomain and just leave it in place. Then set up a new test subdomain that you restrict access to (eg: dev.yoursite.com or whatever).

If you're doing this then you may not want to remove the URLs in GSC and instead just let googlebot see the redirects and hope it updates the indexed URLs to be the ones from the main site.

That's probably what I would do now that I think about it a bit more.

[casionmark]

Yes, that was another suggestion.

I don't know if the URLs on test are getting traffic because I don't have GSC or analytics set up. But based on what I know about the orignal URLs from the main site, some will be.

So, to do this, I need to switch off the block on the site (in robots.txt) so that google can go in and see the re-direct?

[Lee1]

Just to add on what BaldIdiot has mentioned. If the pages from the sub-domain don´t get traffic, I´d recommend setting those pages to 410 status, and depending on the crawl rate of the sub-domain pages, you could use the GSC removal tool as that is only a temporary removal tool in tandem (temporary remove them until G has re-crawled and seen the 410)

Just saves adding another redirect.

Lee