Big Data

10 things you shouldn't expect big data to do

Many business leaders have embraced big data initiatives expecting miracles, only to discover that big data introduces new complexities -- and that reaping the benefits requires a lot more effort than they anticipated.

hero
Image: iStockphoto.com/Tashatuvango

Every organization pursues big data with high hopes that it can answer long-standing business questions that will make the company more competitive in its markets and better in the delivery of products and services. Yet in the midst of this enthusiasm, it's easy to build false expectations for big data -- benefits that will never materialize unless you give it the right amount of "help." Here are 10 key things that big data in itself won't do for you unless you take the right steps to optimize its value.

1: Solve your business problems

Big data doesn't solve business problems. People do. Only those organizations that sit down and decide what they want to get out of their big data before they start working with it are going to reap the calibre of business intelligence they're looking for.

2: Help your data management

IBM claims that 2.5 quintillion bytes of data are being generated daily. Most of this is big data. Unsurprisingly, the amount of data under management in companies around the world has grown exponentially, too. As the data piles up without clearcut data retention and usage policies (especially for big data), organizations are struggling to manage it.

3: Ease your security worries

For many companies, determining security access for big data is still an open item. This is because security practices for big data aren't as defined as they are for data that belongs to systems of record. We are at a point where IT should be working with end users to determine who gets access to which levels of big data and its corresponding analytics.

4: Address critical IT skill areas

Big data database management, server management, software development, and business analysis skills are in short supply. They add incremental burden to a core of critical IT skills that many IT departments already lack.

5: Diminish the value of legacy systems

If anything, legacy systems of record are more valuable than ever with big data. Often, it is these legacy systems that offer critical clues as to how to best dissect big data for analytics that can answer important business questions.

6: Simplify your data center

Big data requires parallel processing compute clusters and a different style of system management from traditional IT transaction and data warehouse systems. This means that energy consumption, cooling, software, hardware, and the systems skills needed to run these new systems will also be different.

7: Improve your data quality

The beauty of traditional transaction systems is that there are fixed data field lengths and comprehensive edits and validations on data that help get it into relatively clean form. Not so with big data, which is unstructured and can come in almost any form. This makes big data quality a major headache. Data quality is critical. If you don't have it, you can't trust the results of your data queries.

8: Validate current ROI metrics

The most common way to measure return on investment from systems of record is to monitor speed of transactions and then extrapolate what this means in terms of captured revenues (like how many new hotel reservations you can capture per minute). Speed of transactions is not a good metric for big data processing, which can take hours and even days to crank through a large cache of data and to run analytics. Instead, the best metric for gauging the effectiveness of big data processing is utilization, which should be above 90% on a regular basis (contrast this with transaction systems, which might be only 20% capacitized). It's important to develop these new ROI metrics for big data, because you still have to sell the CFO and other business leaders on big data investments.

9: Create less "noise"

Ninety-five percent of big data is "noise" that contributes little or nothing to business intelligence. Sifting through this data to get to the nuggets of intelligence that will truly help the business can be daunting.

10: Work every time

For years, universities and research centers ran big data experiments to derive elusive answers on genomes, drug research, and whether there was life on other planets. While some of these algorithms and queries yielded results, many more were inconclusive. There is tolerance for inconclusiveness in university and research environments, but not in corporate settings. This is where IT and other key decision makers need to manage expectations.

Also read...

Other expectations?

Have you encountered a situation where your big data expectations fell short? Share your experience with fellow TechRepublic members.


About

Mary E. Shacklett is president of Transworld Data, a technology research and market development firm. Prior to founding the company, Mary was Senior Vice President of Marketing and Technology at TCCU, Inc., a financial services firm; Vice President o...

8 comments
pfretty
pfretty

Great list. Big data equals potential, yet there are a lot of components businesses need to figure out before they can truly realize its full benefit. As you mention, data management is a prime example.  According to a recent IDG CommVault survey, 80 percent say IT spends too much time responding to end-user data retrieval requests while 31 percent of CIOs call their data management practices chaotic.


Peter Fretty 

Shawn Quinn
Shawn Quinn

Your data is only going to be as useful to you as your ability to make meaningful analysis out of it permits.

mina2014
mina2014

Thanks for the piece. I certainly doubt yor 2nd and 4th points. New disruptive solutions such as Datameer (available at http://www.datameer.com) or Hunk (Available at Splunk's site) redefined the management and users of big data. You can pull data from different sources virtually unlimited and both business and IT users can use them . It is like excel for big data...

marcusattwood
marcusattwood

If a big data application can solve or accelerate a business need then great - but it shouldn't start with looking at data.


The first step is agree on what outcome is needed, and then seek guidance on the tools to achieve that goal. 


A bit like Arthur Dent seeking the ultimate answer to life, the universe, and everything: he didn't expect he would need to then find the ultimate question :)


Big data platforms like triggar.com can accelerate sales, marketing and support, but not without some understanding of 'why'.

IsJos Kan
IsJos Kan

They need to go to the cloud, put it all in a cloud somewhere, preferably cumulonimbus type formation for great coverage and stability. Then provision the cloud as a service, something like FWCaaS or Fluffy White Cloud as a Service. Then implementing some enterprise grade business analytic software, using a high-end cloud service provider for the back-end so we can really delve deep into those data-streams, pattern recognition, artificial intelligence and statistical algebra focusing on maximum wank velocity and business verbiage.

Editor's Picks