Archive for the ‘Updates’ Category
Historic Index Update
We have now updated the Historic Index for January, so that it now contains link data all the way from June 2006 through to December 18th 2011 – just a week before Christmas.
The vital statistics as of today are:
Historic Index
360,824,476,574
Pages crawled
3,658,511,547,386
Unique URLs
Historic Index
Backlinks Date range:
06 Jun 2006 to 18 Dec 2011.
Index built on: 09 Jan 2012.
Historic Index Update
Today we have updated the Historic Index. Whilst you should generally use the FRESH index for day to day analysis, because this data contains the most up to date and current links, the historic index remains by far our largest Index as it
- Historic Index
contains historical data going back over more than five years. We don’t remove dead links from this database, which means it can be used to analyze historical data about how a site may have developed links in the past – even if they have since cleaned up their act.
Today’s index contains 3.6 Trillion Unique URLs and we have physically crawled 357.6 billion of these (most of them have been crawled many times).
It is worth explaining why other databases claim to have similar numbers but do not always show so many external inks as us when put to the test. The two reasons are that we include the deleted links in this number but we also ONLY include External links. That is, links coming from another domain or sub-domain to the site being analyzed.
When you look at other index sizes, there is a lot of confusion. Indeed – there is confusion even between our own Fresh index (which only includes links verified as existing within the last 30 days) and out historic index.
In short – use “Fresh Data” for day to day work. use “Historic data” when you want the largest possible numbers or deepest possible investigation provided you accept that this data will not have newly discovered links, as these generally take a month or more to migrate into the historic index.
Fresh Index hits 100 Billion URLs
Screenshot taken on 6th November 2011
I noticed this little milestone just now. It’s a Sunday, so I really should not be looking at the business too closely, but it was a busy week last week winning the best SEO Technology at the Search Awards and we are gearing up for what we hope will be a massive week for Majestic as we go to Pubcon in Las Vegas.
In preparation for that, the Historic index was updated yesterday – but the Fresh Index now updates so often automatically that we forget to look at the numbers, even though they are listed on the home page.
Today, theough, the number stood out for me, at 100,272,695,204 URLs seen by our crawlers within a 30 day period. This does not mean NEW URLs, it means that the links were string enough to get re-crawled or re-seen in the last 30 days by our crawlers. This is why it makes sense to use the FRESH index for normal day-to-day analysis of link data. Frankly, a blog post that deprecates off the home page of a blog without itself getting any external links becomes largely lost on the Internet. We’ll still have the URL in our historic index, but neither ourselves nor the maim search engines will pay much attention to it – because other websites and therefore, presumably, people pay limited attention to it.
We continue onwards and upwards.
Historic Backlinks Index Update
Today we’ve updated our Historic backlinks index, stats are as follows: 355,472,718,166 unique pages crawled 3,588,647,290,128 unique URLs in total. All fresh data crawled in period up to 20 October 2011 was added to this update.
Historic Backlinks Index Update
Today we’ve updated our Historic backlinks index, stats are as follows: 354,143,175,750 unique pages crawled 3,556,328,209,363 unique URLs in total. All fresh data crawled in period up to 24 September 2011 was added to this update.