Big Data

Is Big Data big hype?

Patrick Gray talks about what to keep in mind when you're talking about Big Data.

Unless you've managed to completely tune out the IT press, you've likely been deluged with gushing articles about "Big Data" and its impact on IT. Traditionally, Big Data was little more than large data sets that might require some specialized tools and techniques for storage, manipulation, and analysis. Like Web 2.0, Cloud Computing, and other overwrought jargon, depending on whom you talk to Big Data will now revolutionize IT, make the CIO into a C-Suite hero, and completely obviate the need for conventional reporting and data analysis. If you're feeling a strange tingling sensation in the "what's old is new" corner of your brain, you might recall the days of Data Warehouses (later Business Warehouses) and Business Intelligence, where large data sets would bring similar promised benefits to IT.

The kernel of truth

With most hype-worthy technologies, there's usually something valid beneath the smoke and mirrors, and Big Data is no exception. New technologies have provided the technical tools to perform more rapid analyses on large data sets, and everything from storage to networks have evolved to the point where we can more rapidly move, process, and manipulate these data. While that's exciting and there are certainly some interesting innovations in this area, it's certainly not news to anyone who's spent time in IT that a new year brings bigger, better, and faster technologies. Arguably, the technology is one small portion of the promise of Big Data.

A key promise of IT has always been leveraging information to produce better decisions. From the first spreadsheet to multi-dimensional databases, executives saw computing as a way to make better or more rapid decisions. Like Data Warehouses and similar technologies, Big Data has shifted IT's focus to how it provides timely and accurate reporting. For years most IT organizations have been hoarding information in ever-expanding Data Warehouses, and shifting the spotlight onto actually analyzing these data is a refreshing change.

Who's afraid of Big Data?

Where the hype surrounding Big Data has done IT a disservice is the usual symptom of a breakthrough technology: hiding a valid business problem beneath a veneer of shiny new technology. Historically, many IT organizations have been flagged as doing a poor job of leveraging information. We're great at acquiring and storing the information and cranking out report after report, but we can't leverage information into actionable decision making. With Big Data being presented largely as an IT challenge, we risk putting the analysis and decision support efforts in the wrong hands. While IT can certainly build an infrastructure around data and equip our counterparts with tools to manipulate it, I see Big Data as predominately a business problem rather than an IT challenge.

From a larger philosophical perspective, one must wonder if overreliance on historical data is even as relevant as Big Data proponents would imply. Management guru Peter Drucker saw increasingly available IT as a threat to corporate decision making, not due to cost or some perceived evil, but because IT made it so easy for management to focus on the past rather than attempting to determine and react to future trends.

Leveraging Big Data

The best aspect of most hype-generating technologies is that they pierce the "knowledge bubble" that surrounds IT and end up being considered by your C-suite peers. Conversations around Big Data are a great time to discuss the information that IT is diligently gathering and storing, and how to design better ways to allow relevant parties outside IT to access, manage, and report on those data. Like all things data, just because you can store it and report on it doesn't mean you should, and conversations around this are one of the best potential outcomes of the big hype surrounding Big Data.


Patrick Gray works for a global Fortune 500 consulting and IT services company and is the author of Breakthrough IT: Supercharging Organizational Value through Technology as well as the companion e-book The Breakthrough CIO's Companion. He has spent ...


I have hunted through biological information for over 25 years with great tools and great mathematicians. I have also worked as a business consultant, watching the dynamics of large companies from top to bottom. The idea that prospecting through agnostic data for novel insight will be a net profitable endeavor is lunacy. However, the idea that senior managers can work with smart data analysts and come up with useful predictive models that capture organizational intelligence in a meaningful way has been proven for generations. Big data is just an increase in scale of an accepted practice. Getting to that scale is an investment that must be made to stay in competition with the smartest companies.

stupid user name
stupid user name

Big Data certainly can add value depending on how you use it. And the firms using it best aren't sharing their secrets to maintain their competitive advantage.


After deploying a few data warehouses of more than a couple of hundred terabytes over eight years ago I can see where the business problem soon became a serious IT problem. Aside from the serious backup problems it presented the use of Business Analytic and Intelligence technologies placing an enormous strain on existing computer processing, memory, and disk storage performance capabilities. And we were only working with relational data warehouses. NoSQL technologies had to be developed to deal with a whole new data warehouse paradigm which included office document files; publications; diagrams; schematics; audio/music; video/entertainment/gaming; CAD/CAM, digital animation, geospacial mapping and topology, etc. In other words we now had to manage gobs of blobs with the traditional transactional data in some form of accessible yet manageable technology. This required the creation of new file system technologies to build new data warehousing file structures and the hardware technology to support it all. This has put an enormous strain on ETL - data integration and aggregation technologies. Business Analytic and Intelligence technologies have to adapt to the new data warehouse paradigm. All of this with hardware and OS software technologies scrambling to catch up with demand to access hundreds of petabytes of "big" data.


The biggest value for Big data is to get a swag - as long as you are not loking for an integrity greater than 80%, it works. So, it is great to make predictive decisions and one does not have to wait for "clean data" - Big data is instructured.

Editor's Picks