Reply to Message

Great Article Chip, only one more point I'd make:
Big data relies solely on collecting past information and looking for trends in it, then seeing if you can utilise those trends. This is always the analysis of historical activity, nor does it really take into account external factors that are often brought to bear on a situation. Thus the need to doubly emphasis what you say about getting a very clear identification of what they expect to get out of the big data project and for you as the consultant to confirm that it is at least likely to deliver that goal. Too often one can get buried in old fine detail and lose sight of the real end.

A long term and classical use of the uses of the big data concept is the insurance industry as they collect the data on various areas and readjust insurance rates based on how trends are moving in that industry and / or area. This has been going on for centuries without computers, and is improved with computers, just shows the computers are NOT essential to being able to work with big data.

Sometimes the only analysis they need is an experienced professional with knowledge in the field they're studying to look at the 'current' trend and make an educated guess. A large segment of the investment industry works this way.

A couple of examples from history where outside factors had a significant influence on the end results are the way the HD DVD vs Blueray and Beta vs VHS fights came out. All the historical data in the first half to three quarters of those wars trended towards one side winning, but external factors came into play and the winner was the other side. Another example of the historical trends not continuing is the concept of battleship, for centuries building bigger ships with more guns was shown to be the way to go for a navy, but the changes in other military weapons during WW2 have shown that to not work any more. It's these sort of pitfalls the consultant has to be very careful to identify and tread carefully with, as they're very hard to evaluate the effect they'll have.

No amount of historical data collection can help a company with the external factors, and it's important to look for any possible such problems early in the project so an intelligent decisions to go or not go ahead with the project can be made.
Posted by Deadly Ernest
Updated - 3rd Dec