Archive for the ‘Development’ Category

Increasing query capacity by 400,000 queries a day

By Dixon Jones November 29, 2010

Majestic SEO are restricting bots from scraping the main content, but opening up avenues for these bots through the API system. Continue reading

Posted In: Development

New Intelligent Crawling

By Dixon Jones November 25, 2010

We think we have won the “mine is bigger than yours” debate. Now we want to not just crawl faster, but also smarter. Continue reading

Posted In: Development

Where’s the fresh crawl data?

By Dixon Jones November 25, 2010

we are working on a new version of the fresh crawl data to be launched in January 2011. Continue reading

Posted In: Development