Majestic SEO is a project that has been working over several years, crawling the world wide web with one aim in mind – to map links between web sites. Our index is now considerably larger than Yahoo’s and is the largest commercially available link index on the planet. One of our short term objectives is to have an index comparable to Google’s in size. We think we are now within striking distance of that goal.
How did you build an Index that large?
We built a distributed search crawler. We have many people around the globe using their spare bandwidth on their PCs to help crawl the web. This gave us a level of scale that was comparable to the main search engines. Even so, we’ve been crawling since late 2004.
Isn’t that a lot of data to store then?
Yes, It’s insane.
What will you do with all this data?
It is more a question of what will YOU do with all this data? It is pretty unique and we imagine a number of people building applications to add value to the data. For the rest of us, we have built a sophisticated interface that lets you analyse back links for any site in considerable detail.
We will also be enhancing our system consistently over time.