Big Data

Big Data and social media: A match made in heaven?

In the private sector, there's excitement around a relatively simple premise: that Big Data can divine actionable answers from the chaos of social media. Is this true?

With the increased interest in Big Data and the capabilities it provides, suddenly everyone is looking for questions that Big Data can answer. Many of these questions involve another relatively new technology: social media. In the public space, the marriage of these two has been prominently featured during the most recent Olympics and the U.S. Presidential debates. Twitter and Facebook are the two "usual suspects" on the social media front, and news organizations have been attempting to glean some sort of signal from the social media noise, offering analysis based on everything from keyword searches of Facebook posts to "tweets per second" during the debates as a measure of key topics mentioned in the debates.

In the private sector, there's equal excitement around the combination of these two technologies centered around a relatively simple premise: that Big Data can divine actionable answers from the chaos of social media. One example might be Big Data-type analysis quickly gauging the public opinion of your company after an internal scandal or Big Data providing a detailed public impression of a new product moments after launch. The obvious "magic" of this type of solution is that it supposedly takes the millions of banalities excreted on the various social platforms every millisecond and distills them into "Everyone in the tween demographic loves Product X, but women aged 37-43 who own a dog think it should be a lighter shade of magenta."

Like many emerging technologies, theory rarely meets practice all the way, and the combination of social media and Big Data is no exception. Despite recent advances, computers still have a difficult time understanding the nuances of human speech, especially on the informal social media platforms. Combine this with emotion, colloquialisms rampant on the social platforms, and nuance (if my new vacuum "really sucks" that's a good thing, versus if I say the same about my new suit), and you have a recipe for an underwhelming performance.

Save your pennies

Even if Big Data were capable of these overzealous feats, social media may not always be the treasure trove of information they are purported to be. Most of the social platforms provide a built-in filter, as users are generally technology savvy and have enough disposable income and time to share their lives on these platforms. If you're marketing a new video game this might be your perfect audience, but if you're marketing work boots or high-end investment vehicles, only a fraction of your audience may be on social media.

Big Data is also a fairly expensive proposition. The technical infrastructure and human resources required to build this capability are far from cheap, and identifying the right data to gather, questions to ask, and analyses to perform is a detailed and difficult process. Furthermore, there might be easier ways to find the answers to your questions. While traditional techniques like focus groups or calling on customers have fallen out of vogue and are far less exciting than building a massive data infrastructure, a few hours with a handful of key or representative customers might give you far more actionable information than trying to separate the wheat from the abundance of social media chaff.


Patrick Gray works for a global Fortune 500 consulting and IT services company and is the author of Breakthrough IT: Supercharging Organizational Value through Technology as well as the companion e-book The Breakthrough CIO's Companion. He has spent ...


Metavana is able to identify and measure sentiment as expressed through social media (such as Twitter). They're among others able to identify the polarity (positive, neutral or negative) from literal statements, and they're special (the only one's capable) to also: - measure the force of the signals (measures of causation are far more important that observing mere correlation); and - interpret polarity for figurative statements. The Metavana approach (science) operates in real time, and requires to a priori assumption (bias) - so in other words, one does not need to know what one is looking for in order for the platform to identify actionable knowledge.


So you are correct that there are many, many possibilities to merge social media information into retail marketing or stock picks for that matter. However, before people move to using big data technology for processing social media information, we anticipate that most companies (keyword here is "most") will start with using technology like Hadoop to off load their existing "big data" technologies which have tended to be proprietary and expensive, and move some of that data and associated processing to Hadoop. So while Hadoop won't replace existing ETL and DW appliance technology, it does provide additional capabilities at a different price/performance point. I expect that companies will start using Hadoop with what they know already, getting the data they already have to be more cost effectively managed first, then move on to using these new technologies to tackle new problems like the one you mention.

Editor's Picks