Big Data is all about technical innovation: high speed networks, in-memory databases, and innovative software that makes it all possible. However, one of the best ways to squander an investment in Big Data is to make it a technology-driven endeavor. Here's why.
From an IT perspective, questioning who owns Big Data seems a bit foolish. After all, Big Data is all about technical innovation: high speed networks, in-memory databases, and innovative software that makes it all possible. However, one of the best ways to squander an investment in Big Data is to make it a technology-driven endeavor. Here's why:
The technology is really just the plumbing
While much of the technology surrounding Big Data is truly innovative, at the end of the day it's a means to an end. An ability to process massive amounts of data in near real-time may be a technical fiat, but it doesn't solve any business problems unless you're producing actionable insights from the data. Technologists certainly play a critical role in Big Data, since its real-time nature demands a higher level of support, but we're simply not equipped to intimately understand the data being analyzed, or the statistical machinations required to produce actionable outputs from those data. Fighting for "ownership" is the wrong tack; instead come to the Big Data party with technical competence, flexibility, and IT's usual talents for project management.
Big Data and hoarding
One of my guilty pleasures is the reality TV show Hoarders, where some poor soul can no longer move about their house since it's crammed with everything from giant collections of porcelain figurines, to decades of unopened mail, to a gaggle of cats, alive and dead. Many of the disappointments with Business Intelligence and Data Warehousing before that were around an inability to actually use the data stored in the systems. When IT has been the primary motivating force behind any data initiative, the repository often ends up looking like the data equivalent of Hoarders: anything that can be captured and stored will be shoved into a repository, with a methodology and organizational schema that only makes sense to the hoarder.
The technology behind Big Data makes it ever easier to gather wide ranging bits of information, and traditionally IT has captured anything and everything. However, having IT be a voice of reason and bring some sanity to the "if we can capture it, we should" argument will prevent your Big Data initiative from turning into the IT equivalent of piles of unopened Beanie Babies and the complete series of Hedgehog Aficionado Monthly for the last 50 years.
Like many business-driven projects, cross-functional teams have the best chance of success. Big Data obviously requires a technical element and strong knowledge from business users who are intimately familiar with their data. Additionally, data analytics experts will likely play a key role in your Big Data initiative, creating a team with a diverse skillset.
Just as this team has a non-traditional makeup, the "rules of engagement" on a Big Data initiative will likely require some tweaking and adjustment from your standard IT management practices. If your Big Data team is analyzing a regional holiday promotion in real time and needs to adjust the model generating the analytics, a call to the help desk to log a ticket likely isn't going to cut it. While IT controls exist for good reason, be aware that adjustments will likely be necessary.
The attention being lavished on Big Data presents a way for IT to shine when it cedes ultimate control of the effort and chooses instead to provide technical expertise and project management and delivery acumen. If nothing else, talk of Big Data may reinvigorate stalled business intelligence efforts, or provide new life for a forgotten data warehouse that provides the analytical capabilities the business desires, without pursuing newer database technologies.