Beat the odds in the big data productivity battle

Gartner predicts it will take companies five to ten years to achieve productivity with their big data. Learn six ways to avoid delaying productivity.

Gartner published in July 2013 its revised hype cycle on big data. This is a timeline that predicts the phases that companies will go through as they try to make their big data work for them.

In the Gartner model, businesses will first enter a stage where they will innovate and experiment with big data. This will be followed by a period of optimism and very high expectations from IT and business management.

Gartner's next predicted phase for companies is a period of disillusionment. This is when companies are expected to take a "time out" from big data pursuits. Companies will begin to question whether they are getting all of the competitive-advantage information that they expected. It is also during this period that companies will regroup and revisit what they are trying to do with big data, asking whether the business is on the right path. As a consequence of this introspection, companies will make adjustments to their methodologies and expectations and then launch into a new path toward big data productivity where they will begin to see their results of their labor.

The Gartner model is particularly interesting because it downwardly revises the estimated arrival of business productivity from big data. The 2012 Gartner big data hype cycle projected that companies would achieve productivity with their big data within a two to five year timeframe. In today's restated hype cycle model, this attainment of productivity is estimated to be five to ten years away.

Corporate big data project leaders don't want to hear this news, but it's better to think about it now (and to reflect on what you can do to avoid it) than to continue without considering whether a delayed productivity forecast could apply to you.

Gartner says the two main reasons for its downward big data productivity revisions for companies are: (1) Big data tools and techniques are being acquired before businesses have the expertise and maturity needed to use them, and (2) businesses are struggling with how to ask the right questions and spot opportunities where they can harness big data for breakthrough insights.

The natural question is: Do big data champions in organizations have to accept the reality of extended time to big data productivity? The answer is no. You can take steps to avoid sand traps that seem to center more around a lack of organizational know-how about big data than a lack of tools or resources. Here are six recommendations.

1: Build your business case first

Nathan Gnanasambandam, Senior Research Analyst at Xerox, recently spoke spoke about "dream sessions" they had with their clients. In the sessions, clients are asked to come up with questions about their businesses that they'd always wanted to answer but never could with the information they had available.

Building your business cases for big data doesn't have to be that involved, but no one should go shopping for a big data tool and system without a clear strategic direction on what the company expects to learn from the data.

2: Don't get budget happy

Big data shopping is a lot like mobile device and cloud shopping was several years ago -- everyone wants bragging rights in executive summits that their company "has" big data solutions. For IT, which frequently finds itself embroiled in budget battles, there is also the challenge to fund big data systems and tools while the budgetary sun is shining on big data. Try to avoid this until you have those strategic sessions with business leaders on what you want to derive from your big data.

3: Don't "underdesign" pilot projects

The normal course in an IT project using new technology is to design a small pilot project first so you get a quick project success under your belt. The strategy should be no different with big data.

However, you still need to design that first project large enough so the methods you use can be applied to subsequent business cases. If you don't design a proof of concept that you can leverage for future work, you could quickly find yourself (and your management) propelled into the disillusionment phase of big data that Gartner talks about.

4: Seek help if you don't know how to ask the right questions

You should start looking for big data consultants and experts with the help of your big data vendors. If your vendors can't provide assistance in this critical developmental area, you should reconsider partnering with them.

5: Perform a proof of concept before you buy anything

It's important to have your business cases in hand before you sign with a big data solutions provider. You should also plan to execute a proof of concept with one of your business use cases with the vendor before you buy; this gives you and your executive team greater ability to see whether the proposed solution is really a good fit.

6: Consider a batch vs. a real-time big data application first

There is a lot of buzz about "in-memory" technology and the ability to get near-real-time analytics reports from big data. The reality is most companies simply want to expand the realm of their business intelligence.

In most cases, big data intelligence can answer key questions just as easily in a batch reporting mode, and you avoid introducing too much complication into your processes too soon. The added complication comes from simultaneously having to manage real-time processing from both systems of record and your big data analytics in the data center. Unless you work in an industry that must have real-time analytics, such as stock trading or high-volume retail, you can always add the real-time capability later.