Company must-have: an overarching big data framework

Mary Shacklett explores the importance of having a big data and analytics framework, whether it's in-house, pre-packaged, or a combination of technologies and methods.

Image: Wikimedia Commons/Lusitana

Companies understand that the ultimate responsibility for their big data and analytics rests with them -- and with their ability to adapt the technology to solve critical business needs. One of these needs is to get an analytics program up and running quickly, even if the company is relatively small and lacks the internal resources of its large enterprise counterparts. On top of this, companies should have longer-term strategies and a technology framework from which they can make big data work.

What exactly is a big data framework?

It is a combination of technologies and methodologies that is used to transform big data in its raw form into refined data governed by a mature framework that can continuously be used for virtually any big data application -- whether it is a batch report, a near-real time data stream analysis, a dashboard, a detailed representation of data, or an automated alarm trigger point for a machine to machine process.

A big data and analytics framework that is well adapted to the company's business is the element of big data that a company has to "own." Once the framework is in place, companies have the option to perform their own big data technology applications and methodologies in-house, or to select from a plethora of vendor offerings that can make these processes easier -- with the company sacrificing some of the customization opportunities that come from building from scratch.

Today, utilizing technology "templates" for big data reports and operations exists on two levels: the end reporting products of big data processes, and the streamlining of big data processes. Both can ease IT learning curves and workloads.

End reporting options from outside vendors have enabled many companies to keep pace with unstructured data coming in from the web about their customers. An example is UrbanSitter, which provides home babysitting services. The company decided to use Google Analytics to track its web-based activities instead of building up the capability entirely in-house. "Google Analytics has reduced our customer acquisition costs by 30 percent," said Daisy Downs, UrbanSitter's Chief Marketing Office. "Our media spend goes right to our best-performing channels."

Companies also go to vendors for "pre-packaged" big data technology methodologies that can reduce IT learning curves and time to market.

Cloudera recently made noise in this space by announcing that it had raised $160 million from T. Rowe Price and three other investors for the purpose of expanding beyond its current business, which is selling services around a popular analytic framework. Cloudera's intent is to become an analytics platform company that furnishes software, services -- and a big data and analytics framework -- that companies need.

Cloudera's goal is to sell standard processes for implementing Hadoop, and also a framework that can be complex to install. Adding to its value proposition, Cloudera also offers data and application integration service, along with training for corporate IT. Intel, SAP, IBM, and others also offer solutions in this space.

"Most times, business [users] don't know what they want in the sense that you have to tell them what kind of data is available and all that stuff," said Sri Vemparala in an interview with TechTarget. Sri Vemparala manages reporting and business intelligence at Stanford University. "But when you demonstrate a pre-packaged application from a vendor, they can easily relate to it immediately." But as Vemparala also says, "Nothing works out of the box, as they say it does -- every environment is different....We got some pre-packaged analytics for student data recently, but it took us a while to get [the software] up and running, definitely longer than what [the vendor] said."

This brings us back to the critical importance of an overarching framework for big data, regardless of whether companies choose in-house, pre-packaged, or a combination of technologies and methods. IT managers agree. A recent Intel survey of 200 IT managers revealed that 75% of those surveyed said they had already had their big data strategies in place. What about your company? Let us know in the discussion.