The big data canary in the coal mine

Patrick Gray states that, at its very best, any analytical tool is the proverbial canary in the coal mine. Here's one way you might appropriately use big data.



One interesting story emerging from the 2013 holiday shopping season is that many of the big data analytical tools employed by retailers failed to predict consumer behavior. A notable item is that Amazon had over a million customers sign up for its Prime expedited shipping service in the third week of December, causing the company to throttle back its sign-up acceptance.

Additionally, everything from weather patterns to last-minute competitor offers threw a monkey wrench into analytical models, confounding even the most data-savvy retailers. Even the lowly consumer has learned to game the big data system to some extent, comparing pricing and exploiting price match policies at retailers to get the best deals. In some cases, consumers even skipped pre-Christmas offers that didn't offer compelling price cuts and saved their buying for post-holiday sales.

Big data isn't bidirectional

The power of big data is the massive amount of data points and the ability to analyze structured and unstructured data, but unfortunately the technology is not magical and, like any analytical tool, provides answers based on the past but cannot predict the future. Just as forecasting the weather or predicting stock market moves will always be an inexact science, so too will the results of even the best big data tools be a hypothesis rather than gospel truth.

At its very best, any analytical tool is the proverbial canary in the coal mine. The singing stops when the danger is already real, with the hope that you have enough time to react. The canary can't preemptively predict the future -- it merely warns of an active gas leak so you can seek shelter. Despite occasionally overzealous vendor claims, big data can identify historical trends and alert you to a change that's already occurred, providing valuable reaction time.

Leveraging the canary

Once your organization understands that big data is not a magical crystal ball, you can begin to use it appropriately. Big data was never meant to be a Wizard of Oz-like man behind a curtain, who would opine about the future and issue dictates; rather, it was meant to be a tool to test hypotheses developed by intelligent humans.

In the case of a retailer facing an uncertain holiday season, an obvious concern would be shoppers waiting until the end of the season. Various hypotheses could be developed to gauge this, ranging from traffic counts in various stores, to browsing digital offers but not pulling the trigger. More sophisticated analysis might factor in environmental data like the weather, or issuance of price matches or some other metric. With deep big data capabilities, this retailer might see a high incidence of price matches early on the morning on Black Friday and issue a predetermined flash sale that evening to steer consumers to a higher-margin product. The hypotheses were in place, generated by intelligent humans, and big data provided an alert that led to quick action.

Big data fire drills

In the hypothetical scenario above, rather than waiting for the big data Wizard of Oz to dictate the next tactic, the retailer had a variety of contingency plans and let the data guide the next action. The faster reaction time engendered by big data provided the advantage, not some property of big data itself.

Key to this reaction time is having various hypothetical scenarios as well as appropriate responses determined and ready in advance. Too many organizations wait for the analysis before determining the next action. Like a coal miner staring at a dead canary and wondering where he should go, or if the elevator is waiting at the bottom of the mineshaft, this wastes valuable time. You need a variety of plans to cope with the potential outcomes of your analysis, rather than a vague idea in someone's head, and these should be tested and ready to execute. Just as the miner practices his escape plan, you should rehearse responses to critical business changes. While lives may not be at stake, competitive advantage and the associated revenues surely are.

Does your big data analytics team usually have hypothetical scenarios as well as appropriate responses determined and ready in advance? Let us know in the discussion.