Big Data

Big data bonanza: But is IT capable of mining it?

There are shovelfuls of riches to be had in big data. In fact it could lead to a new gold rush - as long as we have the tools and skills to tap it.

Written in a coffee shop in Vounaki, Greece, and dispatched to TechRepublic via a free 2.5Mbps wi-fi link.

I have just read a consultancy report that says the world is short of at least 150,000 analysts with the skills to take on the upcoming challenge of big data. Seems to me they have the wrong end of the stick. It will be the machines doing the analysis.

If you want to see big data just lift the lid on any mobile operator, retail chain, bank, investment house, insurance company, utility, government department, manufacturer or big business. They're all knee deep in data and generally wondering what to do with it.

So the big talking point is now big data and the money that it can release from an intimate knowledge of markets, logistics, resources, supply and individuals such as you and me.

But the biggest rewards are buried in the metadata defining and describing our habitual nature: activities, preferences, family, friends, contacts, networks, relationships, work, play, travel, purchases, power to convene, influence, and so on.

Image: Peter Cochrane/TechRepublic

Yet how do you get at these all-embracing characteristics of individuals and groups, markets, products and resources? What's certain is that the scale and complexity will be beyond the scope of any human intellect and capability and will demand a different kind of intelligence to tease out the subtleties of hidden relationships and opportunity.

Does this technology already exist, and where are the appropriate people to staff such a new mini-industry? What we are looking at here is behavioural analysis on a massive scale and something more akin to national security than civil analysis.

So we might guess that suitable technology already exists. As for the people, how about all those IT folks about to be displaced by the cloud and the rising bring-your-own-device movement?

The skills required include technical, mathematical, modelling, war-gaming, role-playing, and strategic decision-making. For the most part I think industry and government have all the necessary people, while the biggest challenge and opportunity lies in the direction of machine intelligence development.

Creating the tools necessary is going to be the biggest challenge and opportunity. It is rocket science and involves evolutionary software and adaptive intelligence, and this is where people shortages will show up very quickly.

So my forecast is that we will see the rapid emergence of a raft of new companies destined to profit from big data by providing the outsourced services and customised tools, a customer and sector at a time.

Not only is this a data-rich area, it is opportunity-rich, and will probably be self-funding right off the bat. It may be that a model based on payment by results is ideally suited. It might also be another gold rush where the people making the shovels make the real money - we'll soon see.

About

Peter Cochrane is an engineer, scientist, entrepreneur, futurist and consultant. He is the former CTO and head of research at BT, with a career in telecoms and IT spanning more than 40 years.

Editor's Picks