New Index Record: 3 Trillion URLs Found

By Dixon Jones December 14, 2010

Our new index is live and we have just set a few new records. We now have knowledge of over 3 trillion URLs and we have crawled over 300 billion of them. What’s more, we have dramatically improved our recrawl rate:

Index Statistics

307,174,759,713
Pages crawled

3,003,219,049,123
Unique URLs

The data actually includes a few links crawled in December, but because these few days might obscure December’s graph (which would have been extrapolated in December based on just a day or so of information) we have put set the charts to map to the end of November.

Faster Crawl Rate

Our last record is a new crawl rate. You might have noticed in November we announced a new intelligent crawl methodology. We can now report that of the half a billion URLs that we are crawling EVERY DAY, a whopping 300,000,000 are RE-crawls. This means that links on busy pages will now get updated and checked much more between updates.

Posted In: Updates

8 Responses to “New Index Record: 3 Trillion URLs Found”

  1. Bence said:

    December 15, 2010 at 4:44 pm

    Wow! Keep up the good work!

  2. tcmremedy said:

    December 17, 2010 at 7:50 am

    Come on, good work

  3. Frank Reise said:

    December 17, 2010 at 11:06 pm

    What a number! And I guess half of it isn’t worth a single click ;) *lol*

  4. John said:

    December 19, 2010 at 7:25 pm

    Keep up the good work, SEO Experts need good quality information to assess our marketing strategies.

    John
    SEO Expert and Marketing Strategist

  5. Fraz said:

    December 21, 2010 at 1:15 pm

    well nice to see this website going to explore its tools step by step . looking great tools . Regards
    Fraz
    SEO consultant

  6. wendy tej said:

    January 01, 2011 at 8:00 pm

    Awesome, keep up the good work.

  7. tcmremedy said:

    January 12, 2011 at 3:58 am

    Keep up the good work,go………

  8. Trackbacks said:

    May 21, 2013 at 5:51 pm