Bingbot getting a little bit over zealous

[baldidiot]

One of our sites is getting hammered with requests from bingbot (IP checks out) - talking about 9000 a day.

Weirdly almost all the pages don't exist (ie: the status code for the bot request is 404) so I'm guessing there's some shennigans going on (ton of links to fake pages or something).

Anyone come across this before or have any solutions? I believe you can throttle the bot in robots.txt but the issue there is it could stop it crawling the legit pages as it could end up just crawling through the 404's at a slower rate.

[universal4]

Maybe try to find who has those links. You could try using the "link:" operator in a search.

Then if you can find them you can disavow the domain that contains the links pointing back to you.

I would think if the site containing the 404's will eventually fall out of favor for getting the links followed if they have a widespread problem of pointing at 404's.

Rick
Universal4

[baldidiot]

Maybe try to find who has those links. You could try using the "link:" operator in a search.

Then if you can find them you can disavow the domain that contains the links pointing back to you.

Yeah I tried that but nothing showed up.

Might not be sites linking in, that was just a guess. No idea where it's getting these URLs from - could just be googlebot going bonkers.

Anyone else notice anything weird like this? The URLs it's trying to crawl are things like:

/Kate-Spade-New-York-Isabeli-Alexander-Avenue-Crossbody-Wallet-Bag-Ember-NWT-625062.html
/PEANUTS-SNOOPY-x-YAK-PAK-messenger-bag-BLACK-unisex-GIFT-NEW-927826.html

All 404. Site is totally clean, no malware etc..

It's also only bingbot doing it, no one else tries to loads these urls.

[universal4]

I wonder of it is picclick rendering links wrong somehow, or a bot that is scraping picclick and messing up te links

See
https://picclick.com/Kate-Spade-New-...809921963.html
https://picclick.com/PEANUTS-SNOOPY-...322766630.html

Each of the examples you provided, while not exact, are dynamically generated links (the id changes) from scraping stuff for sale on ebay.

The pattern of pages names you are seeing appear on the surface to match.

To see how I arrived at the snoopy one I searched google for
"PEANUTS-SNOOPY-x-YAK-PAK-messenger-bag-BLACK-unisex-GIFT-NEW"

Results will show an ebay ad (active or not) and their page will always be /itm/"auctionID" but the picclick links are scrapes from ebay rewritten to html pages.

Now I do not know how this is happening and how the html pages seem to be rewritten to your domain somehow, but maybe others seeing the above can help determine what the next step might be.

Rick
Universal4

[baldidiot]

Now I do not know how this is happening and how the html pages seem to be rewritten to your domain somehow, but maybe others seeing the above can help determine what the next step might be.

Yeah that's what I can't figure out - where our domain ties in to all of this.