Big Data

Hadoop's a matter of innovation, not cost

Hadoop keeps growing in adoption not because it's cheap, but because it drives innovation. Matt Asay explains.

Image: Cloudera via Flickr

Hadoop may have started life as a "poor man's ETL," but cost is the least of its advocates' concerns now. According to a new AtScale survey of 2100 Hadoop-hungry enterprises, just 17% of such enterprises choose Hadoop to save money.

The rest? A big majority (62%) declare Hadoop to be strategic (47%) or game-changing (15%).

Hence, while Hadoop's earliest innovation was arguably to dramatically lower the cost of storing and analyzing large quantities of data, it has since become the industry's preferred platform for big data innovation.

Anemic interest? Hardly...

Though Gartner pegs Hadoop interest as "fairly anemic" over the next two years, AtScale's survey of committed Hadoopers suggests otherwise.

True, AtScale surveyed companies that generally have significant Hadoop clusters (77% of respondents have between 10 and 500+ nodes, and more than 80% have been using Hadoop for more than 6 months), but these are a leading indicator of mainstream adoption.

And the indication is that Hadoop is growing from strength to strength within these enterprises.

As uncovered by the survey, 76% of respondents that use Hadoop plan to do even more with Hadoop over the next three months. A further 21% plan to keep their Hadoop investments somewhat static during this time period.

Figure A
Image: AtScale
A mere 3% expect to cut back.

The reason for this "onward and upward" mentality comes down to results: 49% of the companies surveyed have found success with Hadoop. While another 45% have yet to deliver tangible value with Hadoop, only 6% are pessimistic that they never will.

Ironically, even these "never will" seem to have fallen under Hadoop's spell to some degree, as 0% of those surveyed expect to not use Hadoop. That is, roughly 71% are using Hadoop now, 15% expect to within 12 months, and the remainder are planning to use it at some point beyond a year.

The why of Hadoop

This brings us to why. What's so compelling about this complex, still-too cumbersome data platform that even its most pessimistic adherents can't tear themselves away?

Well, it's not about the money. Not for most companies, anyway.

Figure B
Image: AtScale
When asked why they use or plan to use Hadoop, the need to scale out tops the wish list, followed by a need to build new applications that would be otherwise impossible or improbable with more traditional data infrastructure.

Saving money? That gets just 17% of the votes, with an almost equal number (14%) citing the need to drive revenue, not save money, as their motivation to get smart with Hadoop.

This is particularly interesting, since those companies with the most experience running Hadoop tend to use it for ETL functions (74%), followed by business intelligence (65%) and data science (62%). As mentioned above, in its early days, Hadoop was often dismissed as ETL for companies too cheap to pay for Informatica, IBM, or Oracle.

In a shift, those that have yet to deploy Hadoop now look primarily for its value to transform BI (69%), not ETL (51%). Clearly, word is getting around that Hadoop, if ever it was a cheap way to do ETL, is much more than that.

All of this is good, since those companies that look to Hadoop to cut costs end up discovering less tangible value (44%) than those seeking revenue generation (56%) or to meet scale-out requirements (55%).

You must give to take

All of this serves as a timely reminder that the return on a Hadoop investment is directly proportional to the reason for that investment. Hadoop, and other modern data infrastructure, is generally open source and therefore free to download and try.

But that doesn't mean it's cheap.

On the contrary, getting value from Hadoop requires serious investment. As Redmonk analyst Stephen O'Grady points out, "Never before has so much high quality engineering—tech that would been unimaginably expensive even a decade ago—been available at no cost. And free to modify."

And yet, it's only "organizations that are both capable and view their technical infrastructure as a competitive edge" that will truly benefit from innovations like Hadoop. Because while it may be free to use, to truly make it useful requires a significant investment in people.

In short, Hadoop is a tremendous asset for those companies for which IT matters a great deal. For those looking for cheap ETL or an "unsupervised data landfill," Hadoop could prove to be an expensive, value-less experiment.

Also see

About Matt Asay

Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.

Editor's Picks