Big Data

Big data: Cheat Sheet

You know all that information that fits into nice relational databases? It's not that...

...based around retail – by analysing weather and economic conditions, information on local events, supply chain details, real-time sales data, social network sentiment monitoring, information on employee performance and all sorts of other factors, retail chains can adapt their marketing promotions or sales strategy on the fly to maximise the number of units flying off shelves.

Financial services could also put big data to work by examining all sorts of data sources – voice recordings, demographic information, social networking behaviour, credit and transaction history for example - to determine likely sources of financial fraud.

Some industry watchers have even posited that big data might have a future in health: instead of going to the doctor to find out what's wrong with you, big data systems could diagnose illnesses by crunching data from various sources - sensors detecting your temperature, information from your pacemaker or a study of your DNA. These systems could work out what might be going awry, or predict what condition might befall you in the future, enabling you or the health service to act accordingly.

Right, I get you. So the future of big data analysis is bright then?
Well, there are a couple of dark clouds on the big data analysis horizon at the moment – one, according to McKinsey & Company, will be the lack of big data trained staff, "particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from big data", the consultancy said in a recent report. That talent deficit could run to a 190,000 job shortfall in deep statistical analysis by 2018 in the US alone.

The other problem is a more superficial one – literally. The challenge with gathering data from so many sources and crunching it through huge amounts of computing resource is how to present it back to the poor old user at the other end. A bit of work on the front end of big data systems still has to be done before they're as friendly as they might be.

Big data will remain problematic for large corporations over the coming years thanks to issues including how to present data gathered in a useful fashion, according to analyst house Gartner.

"Collecting and analysing the data is not enough - it must be presented in a timely fashion so that decisions are made as a direct consequence that have a material impact on the productivity, profitability or efficiency of the organisation. Most organisations are ill prepared to address both the technical and management challenges posed by big data; as a direct result, few will be able to effectively exploit this trend for competitive advantage," it said recently.

It's not a situation that's likely to change any time soon, either: until at least the end of 2015, more than 85 per cent of Fortune 500 organisations will fail to effectively exploit big data for competitive advantage, Gartner reckons.

About Jo Best

Jo Best has been covering IT for the best part of a decade for publications including, Guardian Government Computing and ZDNet in both London and Sydney.

Editor's Picks

Free Newsletters, In your Inbox