The Importance of Trusted Data
Recently, Google blocked a number of SEO tools from using their AdWords API subject to a “review” which – for many tools – will never be passed, crippling them moving forward. Whilst the AdWords API is designed for PPC tools, SEOs use the data to help plan their organic strategy.
You can trust Majestic SEO to continue to provide a secure level of service. Majestic does not face these threats from third party data suppliers because we do not scrape Google and we collect our data through first principals obeying laws of the land. Where we do use third party data, we take steps to protect ourselves and our customers from possible outages or legal issues. We DO crawl some content [EditAdd 4 Mar 2012: on some Google owned sites] , in the same way Google crawls ours – but only when allowed to do so by Google via Robots.txt.
Because of Google’s decision to withhold its API, some major tools like SEOmoz and our great partners, Raven Internet Marketing Tools, have been affected. Raven Tools has taken what we consider to be a very honest and brave decision to stop rank checking moving forward to be able to continue with the Adwords API from the 1st January. They did so after this clear message from the Search Giant:
Majestic SEO feeds link data to most of the major SEO Tools in the world today and many of these good partners will continue to do rank checking for as long as they are able. The decision by RavenTools, however, marks a clear business decision by Patrick Keeble and his team to build a business which – when it comes to due diligence – will be a business of genuine net worth rather than one potentially built on a flaw which might one day get exploited by lawyers.
One major competitor in legitimate link data collection is SEOmoz itself with their Open Site Explorer product. SEOmoz have in the past looked to avoid conflict with Google – in particular dropping the large scale collection and display of PageRank pulled from or calculated in parallel to Google’s servers. SEOmoz instead developed MozRank and other metrics to analyze URLs from first principals. A similar strategy was also adopted by Majestic (who never pulled PageRank) in the spring of 2012 with the development of Flow Metrics™ . However – the SEOmoz customer base may be more interested in Ranking data and http://www.seomoz.org/q/loss-of-google-adwords-api suggests that at this time they feel that keeping rank checking tools is more important than keeping the Google AdWords API. Any decision, at this point, is brave one for SEO technologies who wish to track rankings.
The Elephant in the Room
As Majestic SEO looks at its competitors in the Link Intelligence space, we do not believe some competitors took their contractual responsibilities seriously out of the gate. We have rarely mentioned AHrefs, for example, who (by our estimation) must be scraping Google at monumental proportions to try to create ranking data on millions of keywords. Presumably, if Raven Tools have taken the step to withdraw SEMRush data, then AHrefs must be seen by Google to be in the same camp – breaking the Terms of Service on Google to a level which must be costing serious money for the search giant to maintain connectivity and bandwidth to real users as they manage the scraping issues. [Editor note] Since this post was published, AHrefs have announced that they will be withdrawing ranking data as well.[/Editor note].
Aren’t there other Link Data suppliers?
Well here’s the thing. Other than scraping a search engine like Google, or buying it from someone else, the honest way to collect data is to crawl the web from first principals. In order to do that ethically, the accepted principal is for crawlers to obey robots.txt standard in the absence of a more explicit agreement with the website owner. To do this, a crawler needs to positively identify itself. We spend time looking at crawlers and their behavior, and very few crawlers are large enough to develop a meaningful link graph. So anyone claiming to have data may be getting the data from sources which – in the final analysis – may be breaking any number of protocols if not laws. Of course, there are crawlers of this scale, which do identify themselves. Yandex, Microsoft and Yahoo all crawl the web independently and can see similar link graphs as Majestic. Our crawl of the web provides independent verification that can be relied upon.
Has this happened before?
Oh yes – and it will happen again. It is not just Google either. In November 2011 Yahoo finally stopped its Site Explorer product. Until that point, the vast majority of low cost SEO Tools used this data to analyze links. The day Yahoo site explorer went offline, huge numbers of tools broke. Only those using data generated from first principals could continue. I can give you countless other situations. But it all boils down to this…
Who Can You Trust?
Some link intelligence and SEO tools are gaining popularity through the black hat community on forums. We have – to this point – chosen not to engage in these communities. But do not assume that this means there are better tools than Majestic SEO for collecting link data. Our reason for not engaging in these communities has been to protect our fledgling brand from being at the forefront of backlashes from corporations with bigger pockets and more lawyers than Majestic SEO. Using black hat tools is your choice – but we will continue to provide you with data you can trust as legitimately crawled and analyzed from first principals. We will continue to try to be the Biggest, Fastest and Freshest source of link data and to provide it at a competitive price point.
[Editor note] Since this post was published, AHrefs have announced that they will be withdrawing ranking data as well.[/Editor note]