Big Data

Consultants, beware the hype of Big Data

If a client is suffering from Silver Bullet Syndrome over Big Data, Chip Camden advises you to take three cautionary steps.

Our clients expect us to help them compete in their markets by providing skills and experience that they lack. Because we (should) possess a more thorough understanding of the problem domain for which we're employed, our relationship with clients usually involves negotiation between what the client thinks they want and what they really need. It takes a perceptive consultant to ask the right questions to get down to the client's real business requirements. That process becomes even more complicated with the injection of fads and buzzwords.

To make this even more confusing, sometimes a buzzword can be a business requirement in itself. If your client is losing sales because they can't answer "yes" to "Do you use technology XYZ?" then an implementation of some at least token form of XYZ has become a business need for them. Never mind if you have a better alternative -- they don't want to have to argue for it against their potential customers.

In other cases, your client may think they need to implement the latest hot technology because otherwise they're missing out on a real competitive advantage. Usually this means that they don't fully understand the technology, and they're suffering from Silver Bullet Syndrome. A good place to start with these clients is to ask them, specifically, what they hope to gain from the technology. Without a clear goal, your client is likely to spend significant resources and achieve little or nothing -- or worse, get themselves in trouble.

When a prospect asks us, "Can you help me with technology XYZ?" we find ourselves under tremendous pressure to respond enthusiastically in the affirmative. After all, the prospect wouldn't have asked if they hadn't already made up their mind to go in that direction, would they? But adding our voices to the hype chorus isn't helping our client -- it only confirms their opinion that they need to jump on the bandwagon as quickly as possible, and helps them justify omitting careful consideration in the name of agility. So I like to respond, "Yes, I can. But why do you want to do that?"

One of the hottest new buzzwords in the industry is "Big Data" -- the collection and analysis of large amounts of information in order to detect trends, personal preferences, or some other competitive advantage. These data sets are so large that their analysis may require a distributed computing grid containing hundreds or thousands of servers, using a distributed framework such as Hadoop.

The attraction of Big Data is that correlating that much information has already proven to yield surprising and useful results. The potential downside is that it may pose a threat to user privacy, which could lead to legal troubles and a damaged reputation. And as with any fad, it could drain more resources from your client than what they gain in return, only to become obsolete when The Next Big Thing arrives. So, take some cautionary steps with your client:

  1. Calm the hype. Big Data will not magically transform their business. Like any tool, it's how you use it that counts.
  2. Define the goals. What exactly does the client hope to achieve? How can Big Data help? Or is it even the right solution?
  3. Respect user privacy. If you do decide to collect lots of data, treat it more ethically than warranted by the widespread apathy towards privacy we're seeing these days. Tell users what data you collect, guard it from leaks into the wild, and provide users an easy and obvious way to opt out of data collection. Treat users as you would want someone to treat you -- which I doubt is like a data mine.

Also read on TechRepublic


Chip Camden has been programming since 1978, and he's still not done. An independent consultant since 1991, Chip specializes in software development tools, languages, and migration to new technology. Besides writing for TechRepublic's IT Consultant b...


Great Post Chip!!! the points made in this article can be applied to many different scenarios besides Big Data.


How's that for a title? :D I grew up in the world of big data and frankly find the current focus absolutely hilarious. (What people now call big data, we used to call the week's batch of transactions). Unfortunately, IT people in general love to jump on the latest hype-laden fad.Typically, forgetting that other people might just have previously experienced that brand-new super-fantastic solution-to-all-your-problems under a different name. Then the media blows the whole thing out of proportion. (BTW -- many marketing trainers recommend reviewing old "technology", updating and renaming it and then releasing it as BRAND NEW! So the phenomena is neither unique to IT nor unknown). As consultants we have a responsibility to throw the cold light of day on any silver bullet idea. Unfortunately, getting our clients to listen is an entirely different story. Glen Ford, PMP

Deadly Ernest
Deadly Ernest

Big data relies solely on collecting past information and looking for trends in it, then seeing if you can utilise those trends. This is always the analysis of historical activity, nor does it really take into account external factors that are often brought to bear on a situation. Thus the need to doubly emphasis what you say about getting a very clear identification of what they expect to get out of the big data project and for you as the consultant to confirm that it is at least likely to deliver that goal. Too often one can get buried in old fine detail and lose sight of the real end. A long term and classical use of the uses of the big data concept is the insurance industry as they collect the data on various areas and readjust insurance rates based on how trends are moving in that industry and / or area. This has been going on for centuries without computers, and is improved with computers, just shows the computers are NOT essential to being able to work with big data. Sometimes the only analysis they need is an experienced professional with knowledge in the field they're studying to look at the 'current' trend and make an educated guess. A large segment of the investment industry works this way. A couple of examples from history where outside factors had a significant influence on the end results are the way the HD DVD vs Blueray and Beta vs VHS fights came out. All the historical data in the first half to three quarters of those wars trended towards one side winning, but external factors came into play and the winner was the other side. Another example of the historical trends not continuing is the concept of battleship, for centuries building bigger ships with more guns was shown to be the way to go for a navy, but the changes in other military weapons during WW2 have shown that to not work any more. It's these sort of pitfalls the consultant has to be very careful to identify and tread carefully with, as they're very hard to evaluate the effect they'll have. No amount of historical data collection can help a company with the external factors, and it's important to look for any possible such problems early in the project so an intelligent decisions to go or not go ahead with the project can be made.


I couldn't give a darn about Big Data but the rest of the article is the most succinct advice I've seen in a long time.

Editor's Picks