Big Data

Find out why the Hadoop future is going to take a long time

Hadoop interest is real, but it's not going to translate into deployments without two key things happening. Matt Asay explains.

Hadoop

Hadoop has always been a bigger deal with bloggers than CIOs, but there are signs that adoption is picking up, as I've written recently. Still, based on recent Gartner data, the future of Hadoop is clearly going to take a long, long time.

How long?

That depends on how fast Hadoop can make up a $33 billion delta.

Disruption isn't easy

Gartner analyst (and big data afficionado extraordinaire) Merv Adrian has a candid counter to those that ask how he's willing to stand against the hubris of Hadoop fan boys: "The answer is simple: we looked at the data."

That data shows the top three Hadoop vendors pulling in roughly $150 million in 2014 revenue (my own tally is closer to $200 million), which is a drop in the data bucket compared to the database (DBMS) market:

Figure A

While many of these vendors dabble in Hadoop—mostly through partnerships with Cloudera, Hortonworks, and MapR—it's clear that such investments are future-looking and not intended to drive quarterly earnings-defining changes today.

After all, there simply aren't that many enterprises ready to pay for Hadoop today. The big three Hadoop vendors collectively have 1,000 customers or so, whereas the DBMS vendors listed above can point to hundreds of thousands of customers.

And while other, related technologies can claim many more users and customers (MongoDB recently announced over 2,000 paying customers, 10 million downloads, and tens of thousands of users), nothing comes close to the dizzying breadth of the current DBMS market.

Of course, this is not the same as saying Hadoop hype can be discarded.

Oh, the places you'll go!

Though Garter's Adrian "simply [isn't] seeing the breakneck adoption the overall hype indicates," there are still 46% of enterprises that claim to be considering deploying it at some point. That's not trivial.

The trick is to get them there. This, he notes, will require two things:

"Despite investment in training programs, broad availability of Hadoop administration skills (value enablers) and analytics skills (value creators) is 2-3 years out. Don't underestimate this uphill climb. Next, applications taking advantage of the Hadoop ecosystem need to emerge. To this point, the work done on Hadoop has effectively been artisanal data management—everything is built by hand. That's fine if you've got the skills, but most enterprises don't build software—they buy it. Hadoop can be a great place for those applications to reside."

Adrian estimates a two- to three-year ramp to bring enterprises up to speed with Hadoop skills. That may be generous.

And it can't come quickly enough. The demand for Hadoop talent keeps rising, even as interest in more traditional technologies like Oracle falls:

Figure B

That demand matters most not for the JP Morgans of the world seeking Hadoop talent to help them run risk scenarios, but rather the SaaS and other vendors who will bake Hadoop analytics into their products, such that mainstream enterprises will never have to learn Hadoop at all. They'll just benefit from it.

This will take time.

We're still in the "artisanal data management" days of Hadoop, as Adrian points out. We'll be there for several more years, but we're already getting glimpses of the future in Facebook, Uber, and others that have mastered Hadoop to bring its power to the masses. The Hadoop future is happening. It's just not here yet.

Also see

About Matt Asay

Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.

Editor's Picks

Free Newsletters, In your Inbox