Found this on my own site last week and I suspect it’s common. I had Disallow: /*?page= in robots.txt to keep faceted crawl space down — but my listing pages link to ?page=2..8 in the pagination control. So I was linking to URLs I’d told Google not to crawl, which is what generates “Indexed, though blocked by robots.txt”, and the deeper pages of my main game listing had no crawl path at all.
First fix was to just delete the disallow. That was worse: measured 116 non-listing URLs then answered ?page=2 with a 200, so every game/article/legal page suddenly had an unbounded parameter space, and it exposed an existing 500 on ?page=-1 to Googlebot.
What I ended up with: block ?page= by default, then re-allow it per paginating route using Google’s longest-match rule (Allow: /games?page= at 14 chars beats Disallow: /*?page= at. Plus a redirect for anything malformed. Curious whether others handle pagination with robots at all, or purely with canonicals.


LinkBack URL
About LinkBacks
Reply With Quote

