The big data project failure rate is 60%, according to 2015 research from Gartner. A major contributor to this failure rate is political or people-oriented, rather than the technology. What can big data project managers do to avoid these project failures?
“To succeed, you must develop a viable strategy to deliver business value from a big data initiative,” according to Gartner research director Svetlana Sicular, as stated in a Gartner blog post. “Then map out and acquire or develop the missing and specialized skills that are needed. Once strategy and skill priorities are addressed, then you can move on to big data analytics.”
If you can’t demonstrate immediate value to the business from a project, your end users and managers will be scratching their heads and asking, “Sure, the project got installed and the application is working, but what are we getting from the our investment that is traceable to revenue expansion or the reduction of operating expenses?”
In an interview with insideBIGDATA, tech expert Robin Purohit said he believes that it can be all too easy for IT to get consumed in just completing a technology project.
“The one problem that especially plagued a lot of the early Hadoop projects, and still pops up now and then, is that setting up a big data analytics platform is seen as an end in itself,” said Purohit. “Without any clear understanding of pain in the organization that Hadoop might address, a cluster and its associated technologies are set up as a sort of science experiment…. the likelihood that the project will be declared a success under those conditions is virtually nil. Unless a particularly savvy project manager stumbles on a business use case, there just isn’t any way to call that kind of project a win.”
How to improve your odds of big data project success
1: Never start a big data project without specific cost reduction or revenue enhancement goals
The lifeblood of business is cash, which is earned through revenues and is conserved through operational savings. If a big data project can’t contribute either measurable revenue gains or expense savings, you need take a look at it, because the chances that it will be deemed a success by the company are low.
2: Continuously assess the likelihood of project success
Last year, I visited with Teradata, a major provider of big data and analytics solutions. My Teradata acquaintance said, “Especially for experimental big data and analytics projects, our most experienced project leads intuitively sense when a project isn’t going to succeed, and they pull the plug.”
3: Always keep your project stakeholders engaged
Everyone starts running for cover when they spot a project that has consumed lots of money and effort that isn’t going to pay off in business value. You can avoid being left as the one “holding the bag” if you keep your big data stakeholders on the business side continuously involved with the project through project meetings, status reports, and execution.
IT tends to lose sight of business value as deadlines approach and project pressures grow; IT management can’t afford to ever let this happen.
SEE: How to build a successful data scientist career (free PDF) (TechRepublic)
4: Allow enough time for your project to demonstrate results
If a new big data project is going to collect data for the purposes of trend analysis, it might take six months for the project to gather enough data to analyze. In cases like this, business managers should be informed that the system will need a six-month run timeframe. In the 2015 Gartner blog post, Moore wrote: “Organizations should allow ample time to deliver meaningful results.”
5: Don’t let the big data project run too long
As part of Teradata’s big data work, the company “tries out” a lot of different big data projects in its research lab, understanding up-front that some of these projects will work and others won’t. One key in this environment, according to Teradata, is to know when to “pull the plug” on a project that won’t work and to swiftly move on to the next project.