Where’s the fresh crawl data?
Majestic SEO are excited to announce that we are working on a new version of the fresh crawl data to be launched in January 2011.
A number of users have noted that the “fresh crawl” column in the advanced reports is not showing new links each day. When we first introduced this data, back in an upgrade in 2008, it was with a very small user base. we were therefore able to have the crawlers check the urls at the root level against a very small list of domains being watched by users in our system. Nowadays, however, there are tens of thousands of domains being analyzed by users at any one time and the system simply will not scale in its current form. However – as you may have seen in other news – MajesticSEO has now started to roll out a much more intelligent form of crawling. the new crawl will help us build a much more effective fresh index, which will be able to not only give you data at the root domain level, but also at the URL level. The challenge, though, is that we needed to get the new crawler rolled out to our crawler network first, before we can build a new fresh index program – and the new crawl can’t handle the antiquated version of this code.
We know that some of our customers really liked the daily updates and apologize that the system is now just too large to cope with them in their current form, but while the bad news is that you will not get daily updates for a few months, this is already mitigated by giving you and entire index update every two weeks through the improved indexing methodology and is further mitigated by the promise of daily updates in the new year which we feel will be worth the wait.