Discussion on:

Big Data: The perils of past performance

Instead of using Big Data analysis to predict future results, perhaps we should try using Big Data analysis to discover what not to do.

I am definitely not an expert on business computing, but I do know something about the social sciences. I would think "Big Data" should be used to detect relationships and correlations that would otherwise not be apparent. It is not so much a question of predicting the future but rather trying to determine what is going on right now. Actually I think business does this all the time although they often neglect to take advantage of the analysis.

This sort of thing was a big deal in the social sciences in the 1960s -- using factor analytic techniques to extract "dimensions" from Big Data. By the time I hit grad school in 1975 it was already a flop.

Why? The data were collected using theories as to what was important. And what, exactly, were those theories? Good luck finding any documentation at all -- the people who defined the data collection probably weren't aware of them. They were there nonetheless. And that's what you found with your factor analysis -- exactly what the unspoken, unchallenged theories of the mediocre trudgers put in. If you asked them why, all they would be able to say was, "It's obvious." So you find out what's obvious to the ignorant and dull.

Harsh but true. Instead, try starting with a sharp, clever theory, and refining it with testing.

The World Ends on December 21 2012 according to the Big Data Annalist so we don't have much time to wait.

We all may as well get on and do what we want instead of what the Boss wants after all what's it going to matter come December 22?

Col

Being unable to predict the future does not make it impossible to derive value from data analytics. Much of it comes down to the scope of the response to the findings of such analysis.

Would I bet my house on a sure-fire investment plan arising from analysis of past data. Certainly not! Would I take a closer look at credit card usage that correlated very strongly with past patterns of fraud. Very probably.

As Elijah said, we already make decisions all the time. The inability to make a decision that is guaranteed to be correct does not preclude the necessity of making your decisions on the best data available.

Many organizations cannot implement little data correctly: no MDM; the same data input into multiple systems manually causing errors; the same field defined differently in different systems; no incremental loading from operational systems to reporting systems; inability to aggregate information from different organizational units. Many organizations have no hope to implement big data in the foreseeable future.

This belief parallels the classical physics view of the universe and how all things could be predicted before the advent of quantum mechanics.

The epistemological issues of predicting social-technological development is taken up in this recent publication:

Wicked Problems Social Messes: Decision support Modelling with Morphological Analysis. Springer, 2011.

You can see a description at:

http://www.springer.com/business+%26+management/technology+management/book/978-3-642-19652-2

Regards,

Tom Ritchey
SweMorph

So there was a whole flock of Turkeys. Every day the farmer would come and feed them corn, but they were basically afraid of him, but one turkey noticed that if he ran up to the farmer instead of running away he got more corn and didn't have to compete with other turkeys for it. This happened every day, and he got bolder and bolder eventually eating out of the farmer's hand. Soon he was the biggest turkey in the flock, and he was able to wait for the farmer and choose his corn from the bucket. Every day learning from yesterday and getting bolder and bigger. Until Christmas eve......

(for Turkeys substitute DEC, Microsoft, Nokia, RIM (and Apple?)