Results 1 to 5 of 5
  1. #1
    slotbonus is offline Private Member
    Join Date
    August 2026
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Blocking ?page= in robots.txt while linking to it. How do you handle pagination?

    Found this on my own site last week and I suspect it’s common. I had Disallow: /*?page= in robots.txt to keep faceted crawl space down — but my listing pages link to ?page=2..8 in the pagination control. So I was linking to URLs I’d told Google not to crawl, which is what generates “Indexed, though blocked by robots.txt”, and the deeper pages of my main game listing had no crawl path at all.

    First fix was to just delete the disallow. That was worse: measured 116 non-listing URLs then answered ?page=2 with a 200, so every game/article/legal page suddenly had an unbounded parameter space, and it exposed an existing 500 on ?page=-1 to Googlebot.
    What I ended up with: block ?page= by default, then re-allow it per paginating route using Google’s longest-match rule (Allow: /games?page= at 14 chars beats Disallow: /*?page= at . Plus a redirect for anything malformed. Curious whether others handle pagination with robots at all, or purely with canonicals.

  2. #2
    allfreechips's Avatar
    allfreechips is offline Private Member
    Join Date
    August 2010
    Location
    Ohio - The taxing state
    Posts
    1,632
    Thanks
    220
    Thanked 831 Times in 525 Posts

    Default

    Stop using exposed url params a rewrite to something useable such as page2 ext, and then if its s bogus page then 404 it..
    AFC MEDIA LLC COMPLETE GAMING AFFILIATE SOLUTION
    Statsfetch.com
    secure / powerful inexpensive stats fetch system - Free for 20 provers always
    Allmediamatter
    Interface for casino owners/managers to add their affiliates and brands to feed ezcasinoaff
    Allfreechips -
    My 24 year old SEO casino site

  3. The Following User Says Thank You to allfreechips For This Useful Post:

    universal4 (5 August 2026)

  4. #3
    slotbonus is offline Private Member
    Join Date
    August 2026
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Already 404 anything past the last real page, and 301 malformed values back to the clean listing. page=-1 was throwing a 500 before I looked, hidden behind the robots block.
    Paths would sidestep the robots side, agreed. But my sort/filter states are still query params, so /games/page/2?volatility=high mixes both anyway. Did you move facets to paths too?

  5. #4
    allfreechips's Avatar
    allfreechips is offline Private Member
    Join Date
    August 2010
    Location
    Ohio - The taxing state
    Posts
    1,632
    Thanks
    220
    Thanked 831 Times in 525 Posts

    Default

    really you want to avoid ? in the url. there are some cases it makes sense but usually on completely non indexed items.
    AFC MEDIA LLC COMPLETE GAMING AFFILIATE SOLUTION
    Statsfetch.com
    secure / powerful inexpensive stats fetch system - Free for 20 provers always
    Allmediamatter
    Interface for casino owners/managers to add their affiliates and brands to feed ezcasinoaff
    Allfreechips -
    My 24 year old SEO casino site

  6. #5
    topcaz's Avatar
    topcaz is offline Private Member
    Join Date
    February 2025
    Posts
    39
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    Look on the URL structure, it's just giving a name to your pages https://topcaz.com/compare-slots-arc...no_arch_page=3

    how this will help
    Karen
    General Manager | TopCaz
    karen@topcaz.com
    Compare Slots Online with Zino By Topcaz

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •