Big Data

Big data bonanza: But is IT capable of mining it?

There are shovelfuls of riches to be had in big data. In fact it could lead to a new gold rush - as long as we have the tools and skills to tap it.

Written in a coffee shop in Vounaki, Greece, and dispatched to TechRepublic via a free 2.5Mbps wi-fi link.

I have just read a consultancy report that says the world is short of at least 150,000 analysts with the skills to take on the upcoming challenge of big data. Seems to me they have the wrong end of the stick. It will be the machines doing the analysis.

If you want to see big data just lift the lid on any mobile operator, retail chain, bank, investment house, insurance company, utility, government department, manufacturer or big business. They're all knee deep in data and generally wondering what to do with it.

So the big talking point is now big data and the money that it can release from an intimate knowledge of markets, logistics, resources, supply and individuals such as you and me.

But the biggest rewards are buried in the metadata defining and describing our habitual nature: activities, preferences, family, friends, contacts, networks, relationships, work, play, travel, purchases, power to convene, influence, and so on.

Image: Peter Cochrane/TechRepublic

Yet how do you get at these all-embracing characteristics of individuals and groups, markets, products and resources? What's certain is that the scale and complexity will be beyond the scope of any human intellect and capability and will demand a different kind of intelligence to tease out the subtleties of hidden relationships and opportunity.

Does this technology already exist, and where are the appropriate people to staff such a new mini-industry? What we are looking at here is behavioural analysis on a massive scale and something more akin to national security than civil analysis.

So we might guess that suitable technology already exists. As for the people, how about all those IT folks about to be displaced by the cloud and the rising bring-your-own-device movement?

The skills required include technical, mathematical, modelling, war-gaming, role-playing, and strategic decision-making. For the most part I think industry and government have all the necessary people, while the biggest challenge and opportunity lies in the direction of machine intelligence development.

Creating the tools necessary is going to be the biggest challenge and opportunity. It is rocket science and involves evolutionary software and adaptive intelligence, and this is where people shortages will show up very quickly.

So my forecast is that we will see the rapid emergence of a raft of new companies destined to profit from big data by providing the outsourced services and customised tools, a customer and sector at a time.

Not only is this a data-rich area, it is opportunity-rich, and will probably be self-funding right off the bat. It may be that a model based on payment by results is ideally suited. It might also be another gold rush where the people making the shovels make the real money - we'll soon see.


Peter Cochrane is an engineer, scientist, entrepreneur, futurist and consultant. He is the former CTO and head of research at BT, with a career in telecoms and IT spanning more than 40 years.


Thanks Peter, How about the people who invented the Web? People in Physics have a big head start in the Big Data game, as compared to most IT folks. I saw Sau Lan Wu speak a couple of weeks back, about the discovery of the Higgs boson, and she detailed how a whole new big data infrastructure had to be constructed, just so the scientists could compare notes. But unique skills are required to sort things out. I think the Octonion number system yields great big benefits, including allowing the construction of a fully subjective search engine, but virtually nobody outside of the Advanced Physics and Higher Math crowd has a clue how to use them. Multiplication and division are like putting together or taking apart a watch, where all the sub-assemblies have to fit together. But as you point out; computers are very good at some of this stuff. You still need plenty of analysts that talk that language, however. A BIG challenge, if you ask me - finding enough qualified people. But I do know some out of work scientists with the chops needed. Jonathan


Jonathan = You may well be right on many levels. Not only do they have masses of data to analyse, they have a history, tools, and experience ahead of most other disciplines. Not surprising to find them working in banking, insurance, defence and intelligence! What really intrigues me is the play off between the human skill set and that of our machines - the too are formidable - and the machine component is growing rapidly. Please you enjoyed and thanks for the input.