Can anybody help with this Crawl error?

[wonderpunter]

I have hundreds of these, I have added all versions to robots.txt but still getting problems, I also changed the way my site handles the search and changed it to using cleaner words on search.. im stumped

[allfreechips]

you can just tell google to ignore url vars for that url, ie dont look at ?s=xxxx

[TheGooner]

Yep as allfreechips says.

Does that URL use variables at all? If not then one other thing you can consider doing on your site is if the page does not use variables at all then you can add this javascript snippet below - changing the files details to suit your URL of course.


I was having problems with some search bot and google confusing themselves that there was a series of files in subdirectories on my english football page (called english-football.php) ... and I could NOT get it sorted.


So I inserted this code which simply says if this file call does not end at .php, then go to fully defined clean name.

<script type="text/javascript">
if ( location.pathname != "/english-football.php" )
{ window.location = "https://www.goonersguide.com/english-football.php"; }
</script>

It sorted out my google index.

[Michael Martinez]

I think the "disallow /it/" should take care of it for all well-behaved bots but the URLs you shared above would be directly covered by "disallow /it/?s*"

If you don't want anything in the "/it/" folder crawled then you should be good.

That said, you should check to see what is actually crawling your site. Usually when I see these kinds of errors it's someone's poorly managed bot, not Googlebot.

Some people pretend to be Googlebot by running a crawler from a Google Cloud Server account but the IP addresses (and the bad behavior) give them away.

What do your server log files show?

[Malikbhai]

Was this you?

https://www.seroundtable.com/google-...are-29372.html

It answers your question at least for Google.

[wonderpunter]

I have disllowed the /it the /s? etc, its coming from .it so it might be the old translated pages i deleted years ago, I have changed the search parameter to show like this now and see it it works /search/things < hopefully this should work

[fsc]

Not sure if its the case here but i wouldnt blindly trust google to honor robots file directives... I have had issues with this in the past months even though robots file passed the tester I could see kept crawling... even without noscript issues....

My solution was to wherever possible adding directives within the div and ht access blocks

I have hundreds of these, I have added all versions to robots.txt but still getting problems, I also changed the way my site handles the search and changed it to using cleaner words on search.. im stumped

[cantona07]

Yep as allfreechips says.

Does that URL use variables at all? If not then one other thing you can consider doing on your site is if the page does not use variables at all then you can add this javascript snippet below - changing the files details to suit your URL of course.


I was having problems with some search bot and google confusing themselves that there was a series of files in subdirectories on my english football page (called english-football.php) ... and I could NOT get it sorted.


So I inserted this code which simply says if this file call does not end at .php, then go to fully defined clean name.



It sorted out my google index.

i'm loving this little hack!