With a New Year upon us, and newly allocated IT budgets at many companies, Big Data is an obvious initiative on most IT leaders’ minds. With extensive press coverage and a fair share of success stories, Big Data looks to be the solution to all problems and a competitive tool that’s quickly becoming a basic requirement for most businesses, rather than a “nice to have” capability.

Big data problems

The core promise of Big Data is that it can rapidly produce actionable decisions in near real-time. The technology behind Big Data has certainly evolved to the point where we can now process unfathomable quantities of data, but this new capability hasn’t changed the core problem of data analysis: that data should be one element of a decision about the future, not the sole guide.

Consider the stock market for a moment. Taken at face value, it seems like the perfect Big Data problem. We have over a century of historical data, real-time performance information, and market and ancillary data feeds that could fill a data warehouse in moments. There’s also a compelling motive; if one could model and predict the movements of the market, they would be set for life. Even with all of this data, and some of the best minds and technologies in the world applied to the problem, most people would laugh at the suggestion that you could accurately predict the future of the stock market. Yet, these same executives will not bat an eye when it’s suggested that Big Data can accurately model and predict similarly complex market moves.

Consider another example in basic weather prediction. Despite years of historical data and abundant environmental data, in addition to some of the world’s most powerful supercomputers, the forecast for the next 24 hours, let alone next week, is not much better than an educated guess.

“Fact-based” failings

If anything was demonstrated by the last US presidential election, it was that both sides can produce “facts” and statistics that support their policy objectives, even if that objective is in direct conflict with the “facts” of the opposing side. Just as the opposing sides of a contentious political issue can produce reams of facts that support their position, executives who use Big Data-driven “facts” to support of discredit a single position can usually find data to support their position.

So how do we best leverage Big Data?

Rather than using Big Data, and data in general, to prove or disprove a single hypothesis, executives should come to the table with a variety of hypotheses and opinions and constantly revise them based on what the data are telling them. For example, if you craft a Big Data analysis to determine what demographic is most likely to buy your new product, you may confirm that, as you suspected, the pre-teen market is buying it in droves. However, you might miss signals coming from non-buyers that would cause you to tweak your offering and triple your sales in the teenage market.

Just as predicting the moves of the stock market has been an obsession with investors since the dawn of markets, so too has the promise of information-based tools that provide a clear answer about the future. While it’s clear no one can accurately predict every market move, that doesn’t prevent savvy investors from approaching the market with a variety of hypotheses and letting various analyses guide their investing approach. Similarly, Big Data can guide you toward the right strategy, but it remains incumbent upon management to develop a variety of strategies and maintain the flexibility to switch and modify them when appropriate.