Fresh Index hits 100 Billion URLs
I noticed this little milestone just now. It’s a Sunday, so I really should not be looking at the business too closely, but it was a busy week last week winning the best SEO Technology at the Search Awards and we are gearing up for what we hope will be a massive week for Majestic as we go to Pubcon in Las Vegas.
In preparation for that, the Historic index was updated yesterday – but the Fresh Index now updates so often automatically that we forget to look at the numbers, even though they are listed on the home page.
Today, theough, the number stood out for me, at 100,272,695,204 URLs seen by our crawlers within a 30 day period. This does not mean NEW URLs, it means that the links were string enough to get re-crawled or re-seen in the last 30 days by our crawlers. This is why it makes sense to use the FRESH index for normal day-to-day analysis of link data. Frankly, a blog post that deprecates off the home page of a blog without itself getting any external links becomes largely lost on the Internet. We’ll still have the URL in our historic index, but neither ourselves nor the maim search engines will pay much attention to it – because other websites and therefore, presumably, people pay limited attention to it.
We continue onwards and upwards.