The right big data infrastructure comes down to asking the right kinds of questions, and developing a suite of analytics reports that carry back business intelligence on both historical and real time levels.
Enterprises have been hard at work defining what their IT investments in big data and big data equipment are going to be. In many cases, they are making longer term decisions to consolidate any big data processing now being carried out in individual company departments into a central data center where IT can take charge of scheduling big data jobs and ensuring that clusters of big data servers are being optimized to best advantage.
The good news from this is that organizations are making headway with the task of mapping out a physical infrastructure for their big data processing workloads.
At the same time, however, many CEOs feel that the most looming challenge to their success with big data is understanding how best to capitalize on big data in the business. This comes down to asking the right kinds of questions, and developing a suite of analytics reports that carry back business intelligence on both historical and real time levels.
One approach to determining what to ask and what to report on from big data is to start fresh. You can hire or retain temporary expertise in statistical analysis, heuristics, and big data analytics know-how for the business vertical you are in. But if you opt for an entirely "start new" approach, you are running the risk of not thinking broadly enough about your entire business reporting assets and the value they deliver to the enterprise-whether or not they involve big data.
Here's what I'm getting at:
The average company has online transaction reporting systems already in place that tell it how quickly transactions are getting processed, which systems are running well, and which aren't running well and require corrective action. If the systems involve e-commerce or other customer-facing transactions, there are analytics that tell you about speed to response and even transaction abandonment from frustrated customers.
The same company also has a thirty year history with an assortment of daily, weekly, monthly, quarterly, annual and exception reports that draw upon internal data marts or warehouses, and that give managers, executives and line personnel immediate insights into operations, work orders, amount of goods shipped, customer distribution, etc. New analytics vendors complain that one of the major obstacles for implementing big data and general business analytics is that managers and line staff have become so accustomed to these old reports, that they don't like to give them up-even if the new analytics offers more. However, the flip side of that coin is that there is value in these old reports-and companies shouldn't be too quick to discard them.
IT has a unique vantage point because it sees all of this reporting activity-whether reports come from online transaction systems, traditional data mart and warehouse batch reports, or new big data analytics. Because of this, it makes sense for IT to present this body of work to business management-for the purpose of creating a business reporting infrastructure that will likely be "hybrid" in nature because it will identify all of the most value-added reports from every reporting source-whether they are online transaction systems, traditional batch reports or big data.
A documented hybrid report and information infrastructure positions management to capitalize on present and future knowledge needs. Once this infrastructure is defined, IT can also identify which reports (and even data repositories) have been left off the list-and purge or archive them.
Surprisingly, clearly defining a reporting and information infrastructure is not a project that is on most enterprise schedules. It should be. Because if you don't understand the content and direction of your end to end reporting and information infrastructure, you can't always be sure that you're identifying the right information to go after-whether it comes from big or traditional data.