Download Quantcast Top Million.

[kexej28769@nongnue]

Download robots.txt when available from all top million sites.
Parse robots.txt to determine whether the home page and other pages are available.
Collect link data related to blocked sites.
Collect total pages on the site related to blocked sites.
Report the difference between crawlers.
Total sites are blocked.
The first and easiest metric to report is the number of sites that block individual crawlers (Moz, Majestic, Ahrefs) while allowing Google. Most sites that block one major SEO crawler block them all. They simply create robots.txt to allow the major search engines while blocking other bot traffic. Lower is better.

Bar graph showing the number of sites blocking each benin number data tool in robots.txt.
Of the sites analyzed, 27,123 blocked MJ12Bot (Majestic), 32,982 blocked Ahrefs, and 25,427 blocked Moz. This means that of the major crawlers in the industry, Moz is the least likely to turn away from a site that allows Googlebot. But what does this really mean?

Total RLDs are blocked.
As discussed earlier, a major problem with various robots.txt entries is that they block the flow of PageRank. If Google can see a site, they can pass link equity from referring domains on other sites through the site’s outbound domains. If a site is blocked by robots.txt, it’s as if all the outbound lanes of traffic on all roads leading to the site have been blocked. By counting all the inbound lanes of traffic, we can estimate the overall impact on the link graph. Lower is better.