Patrick Gray talks about what to keep in mind when you're talking about Big Data.
Unless you've managed to completely tune out the IT press, you've likely been deluged with gushing articles about "Big Data" and its impact on IT. Traditionally, Big Data was little more than large data sets that might require some specialized tools and techniques for storage, manipulation, and analysis. Like Web 2.0, Cloud Computing, and other overwrought jargon, depending on whom you talk to Big Data will now revolutionize IT, make the CIO into a C-Suite hero, and completely obviate the need for conventional reporting and data analysis. If you're feeling a strange tingling sensation in the "what's old is new" corner of your brain, you might recall the days of Data Warehouses (later Business Warehouses) and Business Intelligence, where large data sets would bring similar promised benefits to IT.The kernel of truth
With most hype-worthy technologies, there's usually something valid beneath the smoke and mirrors, and Big Data is no exception. New technologies have provided the technical tools to perform more rapid analyses on large data sets, and everything from storage to networks have evolved to the point where we can more rapidly move, process, and manipulate these data. While that's exciting and there are certainly some interesting innovations in this area, it's certainly not news to anyone who's spent time in IT that a new year brings bigger, better, and faster technologies. Arguably, the technology is one small portion of the promise of Big Data.
A key promise of IT has always been leveraging information to produce better decisions. From the first spreadsheet to multi-dimensional databases, executives saw computing as a way to make better or more rapid decisions. Like Data Warehouses and similar technologies, Big Data has shifted IT's focus to how it provides timely and accurate reporting. For years most IT organizations have been hoarding information in ever-expanding Data Warehouses, and shifting the spotlight onto actually analyzing these data is a refreshing change.Who's afraid of Big Data?
Where the hype surrounding Big Data has done IT a disservice is the usual symptom of a breakthrough technology: hiding a valid business problem beneath a veneer of shiny new technology. Historically, many IT organizations have been flagged as doing a poor job of leveraging information. We're great at acquiring and storing the information and cranking out report after report, but we can't leverage information into actionable decision making. With Big Data being presented largely as an IT challenge, we risk putting the analysis and decision support efforts in the wrong hands. While IT can certainly build an infrastructure around data and equip our counterparts with tools to manipulate it, I see Big Data as predominately a business problem rather than an IT challenge.
From a larger philosophical perspective, one must wonder if overreliance on historical data is even as relevant as Big Data proponents would imply. Management guru Peter Drucker saw increasingly available IT as a threat to corporate decision making, not due to cost or some perceived evil, but because IT made it so easy for management to focus on the past rather than attempting to determine and react to future trends.Leveraging Big Data
The best aspect of most hype-generating technologies is that they pierce the "knowledge bubble" that surrounds IT and end up being considered by your C-suite peers. Conversations around Big Data are a great time to discuss the information that IT is diligently gathering and storing, and how to design better ways to allow relevant parties outside IT to access, manage, and report on those data. Like all things data, just because you can store it and report on it doesn't mean you should, and conversations around this are one of the best potential outcomes of the big hype surrounding Big Data.