Actionable SEO – The Fresh and Historic Indexes
The Internet has no API. The only way to find out how many links there are is by counting them. However, some ways of counting are faster than others and it is unlikely that any two ways of counting will result in the same number. For example, there are many ways for calculating the world’s population. If two people start counting the population of earth in different places, e.g. Ohio and Paris, but they visit each country in a different order, it is unlikely that they will get the same totals for any city due to births and deaths.
In order to get all of our data, we use a combined processing power to crawl the web. Unlike our competitors, our data is stored in two indexes; the Fresh Index and the Historic Index. They contain information on how sites on the web link together. Our tools then access these databases in order to generate reports which help people understand the flow of organic traffic to their own websites, and those of their competitors.
So Why two indexes? From a technical perspective we can update the fresh index far quicker which allows people to see more update information. However whilst it may seem confusing at first this technique also has a huge benefit to you, the user, instantly in with the same tool at a glance you can see the impact that a page has had on the internet (the historic link count) as well as the current state of affairs with regards to a pages link profile (the fresh link count). All of a sudden backlinks tools with only one count start to seem inadequate!
What is it?
Our Historic Index is one of the most comprehensive on the web, and at the time of writing contains over 4 trillion unique URLs and goes back over 5 years. Due to the size of this database and the build time, it can only be updated on a monthly basis.
When should I use it?
The advantage of the Historic Index is that it contains a huge amount of data, so is great for mapping links over time and getting a link profile of a site .The Historic Index can help to give great insights on seeing how links are developed over time, such as the New and Lost Links tabs allowing you to see when links are found or lost. It also helps to find links that have been removed with a very accurate first found date. A very powerful tool to use with the Historic Index is the Backlink History Tool which shows the acquirement of backlinks visualised on a graph over time so you can see how your link building has developed and are able to see spikes which could relate to an event such as a press release, as well as dips which could be a result of problems on your site.
You can also see the Flow Metrics of a site to see how trusted it is, which due to the huge size of the index will give accurate results.
What is it?
Due to the long build time of our Historic Index, we created the Fresh Index which has links added throughout the day and updates on an approximately daily basis. Our Fresh Index is a real game setter and we believe puts us ahead our competitors. One thing to remember with the Fresh Index is the ‘Memory Effect’ – which can lead to occasionally idiosyncrasies, especially on pages with a first seen date close to the start of the index. We suspect this effect influences all site explorers ( but some companies are more open than others ), meaning although the Fresh Index is fantastic for really up-to-date data for data mining you may want to consider using the Historic index.
When should I use it?
The advantage of Majestic’s Fresh Index is how new the data is, meaning it can be used to monitor links as they appear such as with a press release. The Fresh Index is great to see the flow of links as they appear like a ripple through the web.
Tools such as the New links tab allow you to view it as it happens, seeing when links where added as well as the type of links that they are (image, no follow, etc.) and ultimately where the link is from as well as the Anchor Text used.