Why Does Ahrefs/Moz Not Report Some Domain Redirects?

[baldidiot]

I've seen a few sites recently where there's a domain level redirect (eg: domaina.com to domainb.com) but where both moz and ahrefs don't report them. This is including some of ours (ie: so I know how the redirect is set up) and some that have been in place for years, so it's not a time thing.

It's largely curiosity really. Although being able to hide a redirect from the search tools would be interesting if there was a way to do it.

The redirects track correctly both for humans via a redirect checker, eg:

httpx://www.testersite.co.uk/
301 Moved Permanently
httpx://www.endsite.co.uk/
200 OK

I've seen it happen when the redirect is through cloudflare and when done using htaccess. When it's been on our server, nothing is blocking ahrefs/moz and they crawl the other sites on the same server.

[bpmee]

Couple ideas:

1) Unorthodox redirect using PHP, JS, or some other means.

2) Different flags on the Apache Directive (L, S).

3) Others might be using 302 or some other code that wouldn't be readily picked up like a standard 301

4) Some webmasters and hosts block Ahrefs and Moz.

Unrelated regarding #4: I've noticed Ahrefs doesn't have every competitor backlink. Sometimes I'll see sites ranking really well on fairly modest back links and content. When I did some manual detective work in Google, I later found they had a couple big links from authority sites that block Ahrefs. Best to check a couple sources or spend some time Googling, Binging, etc on your own.

[newcustomeroffer]

Unrelated regarding #4: I've noticed Ahrefs doesn't have every competitor backlink. Sometimes I'll see sites ranking really well on fairly modest back links and content. When I did some manual detective work in Google, I later found they had a couple big links from authority sites that block Ahrefs. Best to check a couple sources or spend some time Googling, Binging, etc on your own.

Noticed the same with Majestic, I guess their crawlers all have certain limitations and work in a slightly different fashion so combining services will give you better results as you say.

[baldidiot]

4) Some webmasters and hosts block Ahrefs and Moz.

Unrelated regarding #4: I've noticed Ahrefs doesn't have every competitor backlink. Sometimes I'll see sites ranking really well on fairly modest back links and content. When I did some manual detective work in Google, I later found they had a couple big links from authority sites that block Ahrefs. Best to check a couple sources or spend some time Googling, Binging, etc on your own.

This is happening with a few redirects of our own and we don't block ahrefs. I totally agree that they don't see all links, but the thing I find confusing is that it was a site prior to the redirect that was heavily crawled by moz/ahrefs, but as soon as the redirect was in place it just falls off their map.

They're strong domains with hundreds of links and moz/ahrefs definitely know about them, but they're not being reported. I'm just curious to know why as it's not anything we're doing and I'm surprised they've chosen to exclude them.

This is what ahrefs shows in terms of crawl data:

Crawled pages
HTTP code Pages
200 OK 0 0%
3XX Redirect 0 0%
404 Not found 0 0%
4XX Other client errors 0 0%
5XX Server errors 0 0%

Prior to the redirect it was crawling the site, but after redirect it just went to zero - that's what I'm looking to understand (ie: why).

[BenjiMon]

In ahrefs they are registering very fast, 302 or 301 for me. If the domains get traffic, helps. This week I registered 8 brand new names and redirected them at the best name and put a lander up and linked it from two sites. Within 2 days ahrefs is picking up two of them as backlinks to the main site. FWIW, they are 302.

[bpmee]

Maybe try verifying the site again from your Ahrefs admin panel? E.g. setup endsite.co.uk and see if the 301 redirects appear as links/redirected pages?

Any funky Robots.txt settings?

Check if any firewall settings are blocking Ahrefs bot IPs? (Less likely)

Also possible it's entirely something on their end. It may take time for them to figure out a domain-wide redirect went up before they begin crawling again. Probably a low-priority thing for them that gets reconciled every couple weeks to a month.

[baldidiot]

Maybe try verifying the site again from your Ahrefs admin panel? E.g. setup endsite.co.uk and see if the 301 redirects appear as links/redirected pages?

Yeah we have it in the dashboard as verified, the redirect doesn't show up, neither do any of the links to the redirect.

Any funky Robots.txt settings?

The end site is crawled fine. The redirected site doesn't have a robots.txt file.

Check if any firewall settings are blocking Ahrefs bot IPs? (Less likely)

One of these is on cloudflare using that as a redirect via a page rule, but I've not read anything about cloudflare blocking it.

Another is a htaccess redirect on the same server as the end site and the end site is crawled fine.

Also as I mentioned before the redirected site was crawled fine by ahrefs before the redirect was put in place, so if there was something in the firewall blocking it then it would have been affected before the redirect was added (unless cloudways does something different with redirects?).

Also possible it's entirely something on their end. It may take time for them to figure out a domain-wide redirect went up before they begin crawling again. Probably a low-priority thing for them that gets reconciled every couple weeks to a month.

One has been in place for well over a year.

[baldidiot]

Figured it - I ran the domain through the ahrefs checker and it had recorded a 403 on the robots.txt of the recipient domain, but with a crawl date from a long time ago. I'm guessing they saw the 403 a few times and then stopped trying to crawl it.

I asked them to recrawl and the status was updated, so interesting to see if they do show up now.

[bpmee]

Thanks, noting this as I know others who've had issues with redirect crawls.

[baldidiot]

Thanks, noting this as I know others who've had issues with redirect crawls.

Pop the domain in here and it'll tell you if it's able to crawl it: https://ahrefs.com/robot

The thing that was weird for us is that it could crawl the domain being redirected to and was seeing the correct robots.txt, but for the redirected domain it was showing a 403 for the same robots.txt url. So you have to check the robots.txt status of the redirected domain, not the one being redirected to (even though it ends up being the same url).

[onlinebetdotcom]

There are many strong sites where backlinks are simply not shown with Ahrefs. Don't know why this happens. Same for building links and not seeing them even weeks later show up in reports.