Many IT organizations do an incredibly poor job of managing and presenting "small data," which should give pause before launching a more complex Big Data project.
There's massive recent interest in Big Data for good reason: An ability to make rapid business decisions based on superior information has an obvious value. Big Data also seems to be all over the press for less noble reasons. From an IT leader's perspective, Big Data is something new and exciting, and the idea that IT can provide a crystal ball of sorts is personally compelling.
Additionally, Big Data requires all manner of new tools and techniques. From different organizational structures to new and bleeding edge hardware and software, Big Data is attractive to technologists and the vendors who support them, for obvious reasons. What's interesting and often ignored when discussing Big Data is that many IT organizations do an incredibly poor job of managing and presenting "small data," which should give pause before launching a more complex Big Data project.
Sweat the small stuff
While Big Data is new and exciting, IT as a profession was launched primarily to gather and provide insight from data, much of it relatively mundane. While Big Data involves massive data sets and near real-time analysis, it's also generally a one-off activity. Big Data might analyze a niche dataset like security or infrastructure analysis, or it might provide insight into a current sales promotion with analysis that is retooled for each subsequent promotion.
More traditional "small data" is generally operational reporting. While a monthly divisional P&L isn't as exciting as doing deep analysis to discover real-time infrastructure attacks, it's equally if not more critical to the health of the company. Despite this, I've seen IT leaders wax poetic about Big Data initiatives, while people managing $500M divisions don't know their profitability until three months after a period closes.
Similarly, while it's exciting to salivate over new technology behind Big Data, few companies have broadly available self-service tools that allow users to quickly and easily create reports from the dusty archives sitting in a forgotten business warehouse application. Instead, users must spend months fighting for a desired report, or hours jockeying Excel spreadsheets to get the data they need. I can only imagine the millions of hours of productive time spent each month by people doing the Excel "hokey pokey" each month to generate a management report that IT has deemed not worthwhile.
Take a consolidated approach to data and reporting
Rather than viewing Big Data as a new and unique concept, look at it in the context of overall data gathering and reporting in your organization. Every company has opportunities to craft more effective reports and delivery mechanisms, whether it's a simple web portal or highly-functional analysis tool that allows users to quickly build the reports they need without engaging IT resources.
Whether the data are big or small, timely, accurate, and easy-to-access information is sure to win IT accolades far more than the cutting-edge technology behind the process.