For evidence of big data success, look no further than machine learning

To win big in the future, big data vendors don't need to take on the old guard. Instead, they must create new applications that require next-generation infrastructure.

Top 5: Things to know about machine learning

In not answering a question about big data project failure rates, Cloudera co-founder and chief strategy officer Mike Olson actually said something much more interesting. Though Paul Gillin asked about Gartner's contention that 85% of all big data projects fail, Olson responded by pointing to big data success, arguing that the industry has moved much faster than expected, and that "over the next two decades we're going to wrap software around decisions, automating bets people are going to make."

That's a bold proclamation, but one perhaps borne out by the data.

Killing me softly

Underlying Olson's statement is the idea that the future isn't being built by the old guard, but rather by a swelling army of new technologies and new vendors. "We're a high-growth company in the $300 million-plus forecast range and most of our business is still on-prem," he said, indicating that not only is the company cleaning up in big data, it's still doing so on the comparatively old battleground of enterprise data centers. He went on:

People aren't shutting down large football fields of Teradata, but the opportunity was never to displace data warehouses. It was to capture more data than we could before and see what we could do if we had better tools to derive value....The growth rate of machine learning is zero if you compare it against traditional markets because there is no traditional market. But it has created value in huge new ways.

To win, in short, Cloudera doesn't need to topple old-school data warehouse vendors. Instead it needs to enable modern applications (within both new and traditional enterprises) that are a poor fit for old-school data infrastructure.

SEE: IT leader's guide to deep learning (Tech Pro Research)

This corresponds with my own experience. While I worked at MongoDB, we sometimes labored to compare ourselves to Oracle or IBM, because analysts and customers tried to fit us into a convenient box of legacy, relational database-driven applications. Over 5,700 customers and millions of developers later, the company's CEO declared that they are "starting to see the beginning of a real platform shift where customers realize the only way to solve this problem is to modernize the legacy infrastructure by moving to a next-generation data platform."

New world, new applications, new opportunities.

This would sound like empty pomp except for the fact that MongoDB's experience is echoed at Cloudera, Hortonworks, and more. Listen in to the earnings calls of those two vendors, in particular, and it's clear that for all the alleged failed big data projects Gartner sees, the opportunity to get it right is driving lots of dollars into new-school vendors who can help.

We've come a long way

Those dollars are chasing the innovation Olson sees as rife within technology today, and particularly open source:

One area where innovation has moved faster than I would have bet is enterprise adoption of machine learning. That's been driven by two things. One was that Spark made it a lot easier to build training models. More than that, though, there's been enormous progress in machine learning models themselves, and God bless the developers for releasing them in open source. I would not have bet on the innovation we've seen in those platforms.

SEE: Machine learning: The smart person's guide (TechRepublic)

If the 1990s and 2000s were all about driving business processes through software, Olson contended, the next two decades are going to be all about driving business decisions through software, specifically machine learning software. That's a big claim, but the pace that data infrastructure is maturing, opening up, and creating big business would suggest that he might be correct.

Also see

Image: iStockphoto/chombosan