Indexing errors involving non-existent translated pages and how to deal with them

[chaumi]

So I'm working on a (English) site which at one point had many translated pages (using Weglot).

Wasn't working suitably, so all the pages and the folders they were in (/it. /es, /fr etc etc) were deleted.

But....those pages had already been indexed before deletion. They're not reachable in any way. Not just the pages/posts of course, but all the usual rubbish URLs that get generated alongside (/?sid, ?/action etc etc etc)

They're showing up in various error/warning reports in GSC (thousands overall).

I'm assuming that overall they may not be hurting (though could in theory be contributing to an overall thin content percentage measure??). But equally, since G appears to think they still exist, they could be interfering with crawl budget and, resulting from that, having an overall negative effect.

The question is how to get rid of them so they're not a problem either way.

I think (assuming that none are linked to by other pages/sites and the sitemap(s) don't contain them either) that instructing robots.txt to return a 410 on every offending URL will do the trick. Still might take time, though.

Anyone done this before? (successfully or not)
Any other approach I'm missing?
Is it possible I'm overthinking it anyway, and just ignoring it is a viable way to proceed?


If the 410 approach is the right one, what code do you actually put in the robots.txt? (I can probably get this from AI, but would be nice to know if someone has used in anger and seen it work).

[baldidiot]

If you're using sub folders for the lanaguges (eg: /it/ ) and you've removed the whole sub folder, then you can request removal in GSC.

https://support.google.com/webmasters/answer/9689846

Specifically this bit:

"Block URLs that start with a prefix"

[chaumi]

Yeak OK. So that page seems to be more about temporary instructions (up to 180 days). I guess by that time G may have got its arse in gear and actually worked out they're all gone forever. But that might be foolish to trust!

But there is a link to how to remove permanently...


https://support.google.com/webmaster...make_permanent

Remove or update the content on your site (images, pages, directories) and make sure that your web server returns either a 404 (Not Found) or 410 (Gone) HTTP status code.

Interestingly it says "Do not use robots.txt as a blocking mechanism"


Think I was right that a 410 is the most suitable, but it doesn't say how to actually do it on that page given I can't get to the original.

[chaumi]

...and using trusty old ChatGPT...

To make an old or deleted webpage return a 410 (Gone) HTTP status code, you’ll need to configure your server or use specific code to signal to search engines that the page is permanently removed. Here’s how to do it based on common server types and setups:
1. Using an .htaccess File (Apache Server)

• If your site is on an Apache server, you can use the .htaccess file to serve a 410 status for specific URLs. Add the following lines to your .htaccess file:apache
  
  Copy code
  
  
  Redirect 410 /old-page-url
  
  
  
• Replace /old-page-url with the path of the URL you want to return a 410 status for. This change tells the server to respond with a 410 for that URL.

2. Using Nginx Configuration

• For sites hosted on Nginx, you need to add a specific rule to your Nginx configuration file:nginx
  
  Copy code
  
  
  location = /old-page-url {
  return 410;
  }
  
  
  
• Replace /old-page-url with the correct path. After saving the configuration, reload Nginx to apply the changes.

3. Using PHP (or Other Server-Side Scripts)

• If you prefer handling this at the application level (e.g., in PHP), add the following code to the deleted page’s PHP filehp
  
  Copy code
  
  
  <?php
  header("HTTP/1.1 410 Gone");
  echo "This page has been permanently removed.";
  exit();
  
  
  
• This will ensure that when the URL is accessed, it returns a 410 status.

4. Using a Content Management System (CMS)

• Some CMS platforms, like WordPress, offer plugins (e.g., Redirection plugins) that let you set up 410 responses for specific URLs. You can use these plugins to easily configure a 410 status without editing server files directly.

5. Check and Confirm the 410 Status

• After implementing, test the URL using a tool like httpstatus.io or browser dev tools to confirm it’s returning a 410 status.

Setting the 410 status code tells search engines definitively that the page is permanently removed, which helps them stop crawling it more quickly than with a 404 status.

[chaumi]

.htaccess looks favorite. But unless I can work some magic, it could be a lot of work compiling the list...and tons of work checking it unless I just go for a handful and trust the rest

[baldidiot]

Yeak OK. So that page seems to be more about temporary instructions (up to 180 days). I guess by that time G may have got its arse in gear and actually worked out they're all gone forever. But that might be foolish to trust!

But there is a link to how to remove permanently...

Yeah it's basically the same thing, it's just saying that during the temporary removal you need to ensure the URLs don't serve pages or they'll get reindexed after the temp period ends.

I've done it this way before and it does work. It's also the fastest way to do it - google will very quickly remove the whole directory this way, if you rely on 410 it needs to crawl each URL individually.

Plus that way you don't need to identify the URLs, which can be tricky if parameters are involved.


Personally I would submit the removal request even if you're planning on returning a 410...


On a side note, if you want to 410 the URLs then you don't need to identify all of them, just use a regex for the path (eg: /it/ ).

[chaumi]

Thx that makes sense. I will have another q when I get my head round it, to follow


PS when I kicked this thread off, in my head I was thinking 'Baldidiot is the most likely to answer this first'. Man, you are so predictable

[universal4]

For the list of ways to make a page or folder show 410, on an IIS server it would need to be done in the web.config, unless you are using third party isapi filter such as Helicon.

And whether using htaccess, web config, cpanel or other server method you should be able to 410 the entire folder. Seems logical to use the google console in addition to one of the other methods, as content within the folder may have been linked to and that should help clear some of that up as well.


Rick
Universal4

[baldidiot]

PS when I kicked this thread off, in my head I was thinking 'Baldidiot is the most likely to answer this first'. Man, you are so predictable

Ha, clearly betting too predictable in my old age...

Btw if you're using wordpress there is a plug in you can use to return the 410: https://wordpress.org/plugins/ultimate-410/

It allows regular expression so you can just tell it to 410 the whole directory.

You can also do the same thing in htaccess but if you're not familiar with the syntax then this might save some hair pulling.