Historic Index Update

By Alex November 27, 2013

We have now updated our Historic Index with data until end of October 2013. Here are the new stats:

Historic Index –

Unique Pages crawled: 601,375,778,666

Unique URLs: 2,329,017,065,332

Date range: 25 Mar 2008 to 26 Oct 2013

Posted In: Updates

2 Responses to “Historic Index Update”

  1. Dirk said:

    December 02, 2013 at 2:34 pm

    by “we have” you mean who? I am newbie here – you have your own crawling robots that are indexing webpages or what?

    • Dixon Jones said:

      December 02, 2013 at 4:07 pm

      Hi Dirk – welcome.
      Yes – we have our own crawlers. we are the largest Link Intelligence database on the planet. We crawl around 187 billion URLs in the last 90 days, and discovered nearly 700 billion URLs. “We” are a UK based company crawling through an international project. Our bot is called “MJ12 Bot” and it obeys robots.txt.