Actionable SEO – The Fresh and Historic Indexes

By Chloe December 29, 2012

The Internet has no API. The only way to find out how many links there are is by counting them. However, some ways of counting are faster than others and it is unlikely that any two ways of counting will result in the same number. For example, there are many ways for calculating the world’s population. If two people start counting the population of earth in different places, e.g. Ohio and Paris, but they visit each country in a different order, it is unlikely that they will get the same totals for any city due to births and deaths.

Site Explorer Search

In order to get all of our data, we use a combined processing power to crawl the web. Unlike our competitors, our data is stored in two indexes; the Fresh Index and the Historic Index. They contain information on how sites on the web link together.  Our tools then access these databases in order to generate reports which help people understand the flow of organic traffic to their own websites, and those of their competitors.

So Why two indexes? From a technical perspective we can update the fresh index far quicker which allows people to see more update information. However whilst it may seem confusing at first this technique also has a huge benefit to you, the user, instantly in with the same tool at a glance you can see the impact that a page has had on the internet (the historic link count) as well as the current state of affairs with regards to a pages link profile (the fresh link count). All of a sudden backlinks tools with only one count start to seem inadequate!

Historic Index

What is it?

Our Historic Index is one of the most comprehensive on the web, and at the time of writing contains over 4 trillion unique URLs and goes back over 5 years. Due to the size of this database and the build time, it can only be updated on a monthly basis.

When should I use it?

The advantage of the Historic Index is that it contains a huge amount of data, so is great for mapping links over time and getting a link profile of a site .The Historic Index can help to give great insights on seeing how links are developed over time, such as the New and Lost Links tabs allowing you to see when links are found or lost. It also helps to find links that have been removed with a very accurate first found date. A very powerful tool to use with the Historic Index is the Backlink History Tool which shows the acquirement of backlinks visualised on a graph over time so you can see how your link building has developed and are able to see spikes which could relate to an event such as a press release, as well as dips which could be a result of problems on your site.

Backlink History Graph

You can also see the Flow Metrics of a site to see how trusted it is, which due to the huge size of the index will give accurate results.

Fresh Index

What is it?

Due to the long build time of our Historic Index, we created the Fresh Index which has links added throughout the day and updates on an approximately daily basis. Our Fresh Index is a real game setter and we believe puts us ahead our competitors. One thing to remember with the Fresh Index is the ‘Memory Effect’ – which can lead to occasionally idiosyncrasies, especially on pages with a first seen date close to the start of the index. We suspect this effect influences all site explorers ( but some companies are more open than others ), meaning although the Fresh Index is fantastic for really up-to-date data for data mining you may want to consider using the Historic index.

When should I use it?

The advantage of Majestic’s Fresh Index is how new the data is, meaning it can be used to monitor links as they appear such as with a press release. The Fresh Index is great to see the flow of links as they appear like a ripple through the web.

Tab showing new links

Tools such as the New links tab allow you to view it as it happens, seeing when links where added as well as the type of links that they are (image, no follow, etc.) and ultimately where the link is from as well as the Anchor Text used.

Posted In: Training

9 Responses to “Actionable SEO – The Fresh and Historic Indexes”

  1. Jesse Leimgruber said:

    December 30, 2012 at 5:09 am

    The MajesticSEO fresh index is one of the most valuable resources to SEO. Despite their best efforts, competitor backlink analysis can’t match the accuracy and speed of the Majestic fresh index.

    It still baffles me that some SEO Consultants can do their job without an accurate backlink index. I use the Majestic data to get information beyond SEO — To see if a company is legit, to analyze their prominence over time, to gain insight on their marketing strategy, to check how long they’ve actually been using their site, to find press data… I could go on and on.

    Great breakdown. Still waiting for the day when you guys release your own search engine :) I’m heading up a research project centered around LSI at Stanford University, it’s not as hard as you might think ;) – We’re building a small scale link crawler to get our own backlink data so we can show a proof of concept of the accuracy of our indexing matrix. If only we had the server power of majestic. You have everything at your disposal for an incredible search engine.

  2. John Body said:

    December 31, 2012 at 6:09 am

    Thanks for finally let me realy know what the fresh and old index are, and thanks for the great awesome seo service!

  3. Bart said:

    December 31, 2012 at 3:32 pm

    Many thanks for resourceful explanation, from whAt I can see recently Fresh index is gethering information on newly acquired links very quickly even in .pl Many Thanks

  4. Cathie said:

    January 06, 2013 at 4:05 am

    Thanks for explaining the difference between the fresh index and historic one. I not been using the historic index much since many of those links are not longer valid, or so I thought, and had been deleted.
    But if you think it is important then you must be right.

    • Dixon Jones said:

      January 06, 2013 at 11:18 am

      >But if you think it is important then you must be right.

      Dangerous assumption! :)
      We provide data. That’s our expertise. You provide interpretation – that’s the SEO’s expertise.

  5. Gareth said:

    January 08, 2013 at 1:18 pm

    Great explaination, the historical index is useful to see old links which may not be active any more but shows how the SEO link history has changed over time.

  6. Jessica said:

    January 19, 2013 at 12:05 pm

    Thanks for advice, I just joined Majestic SEO to see how I can improve my blog in terms of traffic.

  7. Gautam said:

    January 21, 2013 at 5:15 pm

    This is really helpful guys. Thanks! It’s been interesting as I just ran a comparison of SEOMoz, AHRefs and Majestic for backlinks after purchasing all three.

  8. Trackbacks said:

    July 25, 2014 at 8:34 pm