Daily Crawl Data Improvement

By Dixon Jones November 27, 2009

Today we can announce some improvements to the “fresh Crawl” reports that users can see in their control panel.


Click to Enlarge

In addition to our now monthly full index updates, we track, daily, any new back-links that we discover to any given domain. The problem was that  many of the flags on these back-links were not being parsed until the full update and the “mentions” of a domain (where a domain might be listed, but with no link) was appearing in these results. Whilst this is not necessarily wrong, it is hardly helpful for SEO.

So from the 26th of November onwards, mentions are removed from the daily updates.

Click to enlarge

Click to Enlarge

Whilst this will reduce the numbers on this report, at least your clients won’t go clicking on all the links and saying “but there’s no link there!”.

Please note that this report lists when we FOUND new links, not when they were actually created.

You can download these reports into CSV files pretty easy, with the link at the bottom left of the page.

We hope this is helpful? It was my suggestion, so if it wasn’t, I’ll have to hang my head in shame and go back to the development team with cap in hand… :)

Posted In: General

3 Responses to “Daily Crawl Data Improvement”

  1. Dave Davis said:

    November 30, 2009 at 12:20 am

    Absolutely AWESOME guys, really good idea Dixon. With SEO, it’s difficult to provide link reports for clients and this can really show progress accurately.

    Don’t hand your head, fantastic addition.

  2. Ivan said:

    January 28, 2010 at 12:06 am

    So you’ve solved the problem of flags on these back-links ?

  3. Trackbacks said:

    September 20, 2014 at 9:58 am