Getting Bigger. Getting Better.

By Dixon Jones April 9, 2013

Over the last few days, we quietly launched our largest Historic Index ever with nearly 4.2 trillion URLs. It’s impressive, but we have also decided that we should maybe get a little more memory. For that reason, we need to let warn our customers that we may have a few short maintenance upgrades over the next week as our Network Administrator and his team start to take Pallet loads of new computers out of their boxes and into some data racks. Here is one delivery:

Computers by the Pallet

Computers by the Pallet

When the lorry arrived he was somewhat surprised to see a bunch of young faces with an assortment of trolleys. His first question was “What, no forklift?”.

Better Indexing Means More Memory Need

Over the last few months, Majestic has worked hard to improve what we call “Discovery”. We want to find new links, but not at the cost of revisiting and re-crawling good pages which may have changed. The usual trade off for increasing the speed of discovery of URLs is that you have to take away from recrawl time – but we now have implemented several ideas to improve discovery without compromising the re-crawl data. Some of this we are going to keep as part of our “secret sauce” but other ideas – like our bulk URL submitter – we are happy to shout about.

We have built our entire system for scale and thanks to your support we are able to now build on that design.

We’ll try not to disrupt your service – but…

Obviously – installing computers by the pallet-load is not a simple operation, especially in a live environment. We believe we have a way to do this with the minimum of disruption – but hope that you will give us leeway if you find short outages whilst server reboot over the next seven days or so.

What’s the Payback to our customers?

Bigger, Faster and Fresher data – with no increase in prices, of course!


Posted In: General

10 Responses to “Getting Bigger. Getting Better.”

  1. Alex said:

    April 09, 2013 at 10:38 am

    Wow, what DELL servers you are using?

    • Alex said:

      April 09, 2013 at 11:59 am

      It’s a secret…

  2. Jamie Knop said:

    April 09, 2013 at 11:20 am

    Must post a pic of your new data room once its all set up (a little less blurry though!).

  3. mohsin said:

    April 09, 2013 at 11:57 am

    Good Luck to the team :)

  4. Vampire Joe said:

    April 09, 2013 at 1:18 pm

    Do you run all your servers in your own office/datacenter or do you also rent machines from hosters or in the cloud ?

    • Alex said:

      April 09, 2013 at 2:35 pm

      We use all kinds of processing models.

  5. Net66 said:

    April 10, 2013 at 4:42 pm

    That’s a serious amount of hardware.
    Air conditioning must cost you a Bomb! Is your secret “room” just a giant fridge?

    • Dixon Jones said:

      April 10, 2013 at 5:09 pm

      Actually – we have an exceptional “Miniaturization” technology, which reduces all the servers into a mini fridge which we keep in the back of a VW Camper Van. ;)

  6. blog bayu vai said:

    April 19, 2013 at 3:37 pm

    I am a happy MajesticSEO customer for past few years and being in the SEO industry I keep an eye on every upcoming tool. But in my opinion there is no worthy competitor yet. It is my personal opinion but no other backlink data provider comes near the level of MajesticSEO.
    Alex very well concluded the post “regardless of who sits at top spot at any given moment in this market, without constant innovation, it won’t be for long.” I agree with you.