A year ago, Gartner estimated that 60% of big data projects fail. As bad as that sounds, the reality is actually worse. According to Gartner analyst Nick Heudecker this week, Gartner was "too conservative" with its 60% estimate. The real failure rate? "[C]loser to 85 percent." In other words, abandon hope all ye who enter here, especially because "[T]he problem isn't technology," Heudecker said. It's you.
Old dogs, new tricks
Well, not you exactly, but rather the difficulty of grafting modern big data practices onto existing infrastructure and into company cultures that are ill-prepared to embrace big data. After working through scads of botched big data projects, Heudecker told me that the primary causes of failure are the difficulty inherent in integrating with existing business processes and applications, management resistance, internal politics, lack of skills, and security and governance challenges.
It's interesting that management would be a key blocker of big data projects, given that they've been the biggest cheerleaders for big data, touting its power to transform their businesses on earnings calls and to the press. Ask them how those projects have turned out, in fact, and 80% will tell you that they've been super successful and have delivered significant value, according to a NewVantage Partners survey.
SEE: Research: How big data is driving business insights in 2017 (Tech Pro Research)
Get into a postmortem with the teams actually responsible for running these projects, however, and you get that 85% failure rate that Gartner has uncovered. Ironically, this may partly derive from a tendency for executives to trust their gut rather than data, as a Fortune Knowledge Group survey revealed. Probe the NewVantage respondents a bit more and they, too, admit to a cultural mismatch that is nettlesome to overcome:
If there is any sobering trend in these results, it lies in the apparent difficulty of organizational and cultural change around Big Data. More than 85 percent of respondents report that their firms have started programs to create data-driven cultures, but only 37 percent report success thus far. Big Data technology is not the problem; management understanding, organizational alignment, and general organizational resistance are the culprits. If only people were as malleable as data.
People are not this malleable, of course, and executives aren't solely at fault. Look down the org chart and there's plenty of political infighting over who will own the project, how broadly to run it, and who is equipped to staff it (answer: not nearly enough people). All of which trends toward virtually no projects delivering a successful outcome, by Gartner's reckoning.
Not that you should lose hope.
A ray of big data light
The problem, Heudecker went on, is that "Organizations...need a plan to get to production. Most don't plan and treat big data as technology retail therapy." This isn't surprising, since big data vendors, from legacy stalwarts to buccaneering startups, all basically promise the same thing: Buy my expensive and hard-to-use technology stack and you'll magically get [insert the buzzphrase of your choice; I like "360 view of the customer"].
It's this promise of magic that keeps funding big data IPOs, but it's this same promise that almost certainly assures a company of abject big data failure. The key is to better align expectations.
SEE: How your executives will screw up your next analytics project (TechRepublic)
There's also the problem of moving too fast. Hortonworks' Ronda Swaney, for example, urges enterprises to quickly build out small, departmental successes into holistic, company-wide initiatives. While that sounds pragmatic (it's certainly better than going big right from the start), a company's culture may not be able to keep pace with attempts to quickly scale out projects, and the very DNA that made the small-scale project successful would likely prove insufficient to carry the broader project to a successful conclusion.
In general, however much vendors may want their customers to go big with big data, the last few years of rampant big data failure suggest that a far better way is to start small, and build slowly. Indeed, I'd go one step further and suggest that companies seed these projects in a more bottom-up fashion, driven by developers. Let them experiment and grow projects organically. Given the current 15% success rate of big data projects, it's time to try something different.
- How your executives will screw up your next analytics project (TechRepublic)
- What to do when big data gets too big (TechRepublic)
- Video: Why big data is baseball's next big hit (TechRepublic)
- Cheat sheet: How to become a data scientist (TechRepublic)
- Infographic: More than half of companies haven't started a big data project (TechRepublic)
Matt is currently head of the developer ecosystem at Adobe. The views expressed are his own, not those of his employer.
Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.