Make no mistake about it. As companies get on board with the idea of mining their big data for business answers, they are also expecting to solve big problems.
In the automotive industry, this might be parallel-processing streams of data and then bringing them all back together into a composite answer that tells what the effect of wind will be on fuel consumption, based on a new vehicle design. In the past, this was done with physical wind tunnels. Today, this is being accomplished via computerized data simulations-and the wind tunnels are being sold.
In other cases, online content providers are analyzing which areas of content are most popular so they stay on top of their customer usage patterns.
The list is endless-and for many business managers, it can inspire fear. Xerox, which has its own consulting practice that is focused on big data analytics and high performance computing (HPC) for enterprises, sponsors what it calls "dreaming sessions." In these interactive sessions, executives and high-ranking managers from an assortment of companies in different industries come together to discuss what needs to be accomplished in their businesses over the next five years. They also talk about the "tough" problems that they have not been able to solve for years. It is these elusive problems that they want to apply big data and high performance computing to.
"We work with business managers, and we focus on these problems and create algorithms designed to mine the data to arrive at the answers," said Nathan Gnansambandam a Xerox Senior Research Analyst. Companies like Xerox—-and research institutes and universities around the world-have statistical engineers and computer scientists with the skillsets required to develop these complex algorithms and data mining techniques that are able to extract ordinarily elusive answers from a sea of big data.
However, now that companies are beginning to see the power of harnessing big data, they also want the ability to do some of this big data "harnessing" themselves.
One way to create "big data, big problem solving" within the enterprise is to hire it. But whether you set out to hire computer systems analysts, industrial engineers, information engineers, or whatever name this skillset runs under, you will find yourself in a highly competitive market involving high dollar people who are such demand that it could be difficult to retain them. If you are in pharmaceuticals, where the correctness of a new drug design can be pivotal to your success; or in financial services, where an accurate risk analysis performed against world currencies for a particular day could mean everything-then you might have to consider hiring some of these people.
What many companies are doing, however, is looking internally at their own engineering and research people. A market research analyst in Marketing can potentially be stretched into a big data analytics expert on customer and product trends. If you are in the automotive industry, a product design engineer can potentially be developed into a skilled programmer who can develop computerized simulations for product testing by harnessing big data. There are also pre-developed big data programs for various industry verticals that both third party vendors and the Open Source community offer. These programs are designed to answer the questions that you and your competitors are most likely to ask of your big data.
Of course, there is a "catch." If the answers your company wants from its big data aren't the "usual" ones-and if you deem it sufficiently strategic to your business to be able to innovate your own questions for competitive advantage-you are going to have to develop your own big data expertise.
For many corporate executives today, hiring or developing these people is the biggest problem they are facing with big data.
Mary E. Shacklett is president of Transworld Data, a technology research and market development firm. Prior to founding the company, Mary was Senior Vice President of Marketing and Technology at TCCU, Inc., a financial services firm; Vice President of Product Research and Software Development for Summit Information Systems, a computer software company; and Vice President of Strategic Planning and Technology at FSI International, a multinational manufacturing company in the semiconductor industry. Mary is a keynote speaker and has more than 1,000 articles, research studies, and technology publications in print.