Just where are we with big data?

Featured Content

This article is courtesy of TechRepublic Premium. For more content like this, as well as a full library of ebooks and whitepapers, sign up for Premium today. Read more about it here.

This article is courtesy of TechRepublic Premium. For more content like this, as well as a full library of ebooks and whitepapers, sign up for Premium today. Read more about it here.

Join Today

Business analysts are closely following the trajectory of big data trends. Here's a look at some numbers, roadblocks, and success stories that reflect the current and future state of big data.

Big data is still in its early stages, presenting companies with both organizational and technological challenges. However, the promise of the business intelligence that big data can deliver is too great for companies to ignore. In August, Tech Pro Research conducted a survey on big data trends and found that 46 percent of respondents have implemented or are planning to implement big data.


Enjoying this article?

Download this article and thousands of whitepapers and ebooks from our Premium library. Enjoy expert IT analyst briefings and access to the top IT professionals, all in an ad-free experience.

Join Premium Today

Other analyst reports suggest even more robust intentions for adopting big data.

In a September 2012 Harvard Business Review survey, 85 percent of respondents -- consisting of Fortune 1000 companies and government agencies -- said they either had big data initiatives underway or that they were planning for them. The same percentage responded that C-level executives in their organizations were sponsoring these initiatives.

One year later, in September 2013, a Gartner research survey revealed that 64 percent of responding organizations have invested or have planned to invest in big data, compared to 58 percent of organizations surveyed in 2012. According to the survey, media and communications (39 percent), banking (34 percent), and services (32 percent) were front-runners in the adoption of big data. The industry verticals projecting the greatest number of planned investments in big data for the future were transportation (50 percent), healthcare (41 percent), and insurance (40 percent).

In a recent press release, Frank Buytendijk, research vice president at Gartner, summed up the survey results. "For big data, 2013 is the year of experimentation and early deployment," he said. "Adoption is still at the early stages with less than eight percent of all respondents indicating their organization has deployed big data solutions. Twenty percent are piloting and experimenting, 18 percent are developing a strategy, 19 percent are knowledge gathering, while the remainder has no plans or don't know."

Organizations are hoping to use big data to understand their customers better and improve business processes. However, knowing how to harness big data is still a challenge for most. This point was clearly illustrated in the Harvard Business Review study, which showed that:

  • Only 15 percent of respondents ranked their access to data today as adequate or world-class
  • Only 21 percent of respondents ranked their analytic capabilities as adequate or world-class
  • Only 17 percent of respondents ranked their ability to use data and analytics to transform their business as more than adequate or world-class

"One of the largest issues facing many organizations that wish to use big data today is knowing which questions to ask, and how to formulate questions," said Srikanth Velamakanni, CEO and founder of Fractal Analytics, a provider of big data analytics for companies in a number of key industry sectors. "Many companies ask themselves if they asked the right questions of their big data in the first place."

Velamakanni and other analysts believe that the key is to learn how to ask the right "trigger" questions, which can push out highly actionable information that was never considered before. This is also an area where there are few established technologies that can get early big data practitioners to the results they desire. Consequently, the task for corporate C-level executives and those charged with developing a successful big data analytics program is to develop the right sets of business practices, analytics, and technologies that can unearth new answers to old business problems so they can achieve breakthroughs for the business.

Developing a business case

Despite the challenges, some organizations are already implementing big data analytics with great success:

The Hamilton County Department of Education

The Hamilton County Department of Education oversees nine K–12 school districts consisting of 76 schools in and around Chattanooga, TN. Its total student population is 40,000, served by a staff of 6,400 employees.

Hamilton County wanted to improve its student high school graduation rates and recognized that to do so, it needed to understand sooner which students were most at risk of not graduating. Early risk detection would give educators an opportunity to intervene with those students before it was too late. In the past, Hamilton County had relied on manual reports, where it was impossible to gain performance insights that could assist educators in identifying at-risk students and how to best help them. Using big data analytics, Hamilton County consolidated all student information into a single data repository that enabled it to more fully construct student risk profiles, which then became actionable for educators. Hamilton County improved its high school graduation rate by 10 percent.

General Electric Co.

In 2011, General Electric Co. (GE) invested one billion dollars in a big data effort that will position it to better manage two-thirds of its business, which was $94 billion in 2011. This business is characterized by digital data emanating from sensors and other digital devices that are embedded in machines such as GE's jet engines, turbines, trains, and hospital MRI equipment. GE refers to it collectively as the "industrial Internet." Harnessing industrial Internet data will enable GE to help customers identify maintenance problems before they occur, improve fuel efficiency, and make other operational improvements that could add up to trillions of dollars in savings.


Ooyala, a video services company, uses big data to furnish media firms like Bloomberg, ESPN, Telegraph Media Group, and Yahoo! Japan with "actionable analytics" that provide detailed visibility of how consumers are using their video content and how that content can be optimized to produce more revenue. Ooyala's analytics engine processes over two billion analytics events each day, derived from nearly 200 million viewers worldwide who watch video on an Ooyala-powered player. With the help of big data analytics, Ooyala is creating business value for its clients and revenues for itself.

Each of these companies either built revenues or improved performance by carefully constructing a business case that everyone knew would add value to the business -- along with delivering immediate returns from the implementation of a big data program.

A second element of success in each of these use cases was that the initial big data project paved the way for future "take off" projects that could use the same data and analytics foundation that the initial project built. For example, Hamilton County's student profiling analytics methodology was capable of being applied to "high success" students and other segments of the student population as easily as it was being applied to at-risk students. GE's industrial Internet, sensor-based data capture and analytics is capable of growing with GE's business as the company brings more sensor-based products to market. And Ooyala's business model can be expanded to many other venues (and revenue opportunities) beyond those of its current clients.

Querying data in new ways

Companies have already demonstrated that carefully constructing business cases with definite, prescribed boundaries for outcomes brings success to big data analytics. However, there is also a counterargument to such tightly construed business cases that emerges during the development of big data queries. It rests on the fact that for big data to deliver innovative and groundbreaking insights, the knowledge workers who are querying the data have to leave enough room for experimentation to see what the data can offer. According to Fractal Analytics' Velamakanni, if companies focus their initial queries and business cases for this big data too narrowly, they may miss other areas of revelation that the data can potentially provide.

"We find this often when we work with clients," he said. "Companies come to the big data analytics already narrowly focused on solving a particular problem. This is different from just looking at the data itself."

Objectively looking at data to see what new information it reveals is difficult in most corporate environments, where the pressure to produce results is continuous and never-ending. On the other hand, in a more open-ended data query approach, it's possible to acquire fresh insights that a narrower focus on business case or query structure might have preempted.

Finding a balanced path that allows for both highly focused research and insightful experimentation with big data is a challenge for many enterprises. To facilitate this balanced path to big data intelligence acquisition, organizations are finding that they require new skill sets and work groups to carry out their big data missions.

Acquiring a big data skill set

For many corporate executives today, hiring or developing personnel is the biggest problem they are facing with big data. In its 2013 Digital IQ Survey, PwC Advisory Services surveyed more than 1,100 business and technology executives and found that only 44 percent of those surveyed felt that they had the internal talent to capitalize on big data.

The Tech Pro Research survey revealed problems with hiring and retaining qualified staff, as well. Although 33 percent of respondents reported no hiring obstacles, 46 percent said they had run into problems filling big data job roles.


The skill set needed for an effective enterprise big data team is actually a combination of old and new skills. The "new skills" needed for big data, which many enterprise data analysts today lack, are data science programming skills, as well as strong background in mathematics and statistical analysis and data modeling techniques. Great data scientists also can experiment in new and innovative ways with data. They are traditionally coming from academic and research environments where they aren't afraid of failing on their way to attaining results.

These data scientists must be complemented with a new set of technical hard skills, which include knowing how to build out databases that span terabytes of data, the ability to administer software frameworks like Hadoop, and expertise in databases such as noSQL, Cassandra, and HBase or in analytics  programming languages and facilities like R and Pig.

These capabilities are in sharp contrast to the skill sets and goals of traditional corporate data and business analysts, who live by project deadlines and are expected to produce tangible business value from every task they do. The very nature of their work precludes them from experimenting -- and the very thought of failure from a data experiment that doesn't work out can threaten a career.

Nevertheless, enterprises must find ways to mesh traditional data and business analytics skills with more "new wave" data discovery techniques that allow for experimentation. To illustrate, 59 percent of companies responding to a 2012 survey conducted by analyst firm Information Difference said that their big data projects were "highly linked" to their master data repositories (customer data, product data, etc), and that master data was driving big data. They were using their master data from systems of record data as "vectors" into big data queries that were probing for customer behavior patterns, preferred product combinations in certain markets, etc.

The bottom line for enterprises is this: To develop an effective big data team, they need an interdisciplinary group of team members who can work with both unstructured and semi-structured "new" data and enterprise systems of record data in new ways.

The model data science team in a company now consists of a business process analyst who is understands the business and what it wants to know; a data scientist who has the statistical, mathematical, and programming skills and data know-how to create algorithms that extract meaning from big data; and a technical IT person who can interact with the corporate database architect and coordinate physical HPC (high-performance computing) resources in the data center for purposes of scheduling and executing big data jobs. Although some organizations choose to place this data science team under end business areas, a growing number is finding that IT is the best place to assign the team.


According to Silicon Valley Data Science, a big data science consulting company, "The best data driven product companies build with small, agile teams of data scientists and engineers who are good at working together. The engineer understands the algorithm that the applied math person ends up using, because they have been discussing it as it evolves. Both are aware of each other's progress and can offer each other ideas. This approach results in an effective algorithm that scales, and teammates who will share a beer in celebration, rather than retreating to separate taverns to drink away their mutual frustration."

This is where many enterprises are now -- in the process of constructing interdisciplinary teams with a mix of business, analytics, and technical skills that will drive their big data efforts.

Final recommendations for companies

Big data presents numerous challenges, but potential benefits are too compelling to ignore. Accordingly, most enterprises are hard at work defining business cases, organizational structures, and technology architectures that will be capable of supporting their big data efforts now and in the foreseeable future. Some organizations are already doing this with great success. The best practices that characterize their success include the following:

Start with a big data plan that has long-term duration

Big data analytics should establish a foundation for future research as well as produce answers to current questions. Companies that are building momentum with big data projects understand this. They carefully construct big data query structures and repositories for reusability as well as for present use to derive long-term value from their investments.

Construct an interdisciplinary data science team that can work with both IT and the business

Blending technical and business savvy has never been easy, but it is a critical element in corporate data science teams. Most successful enterprises are assigning the data science team reporting relationship to IT, but the business process analysts on this team have strong ties and accountabilities to the business as well. The core team of business analysts, data scientists, and technical support personnel works shoulder to shoulder in a tightly integrated and collaborative environment.

Blend data experimentation with "fast" data results

The global business environment is fiercely competitive, and C-level executives and line managers expect answers to problems right away. In some cases, big data can deliver this. However, in other cases, the data science team needs the opportunity to experiment with the data if they are to come up with new and groundbreaking insights. Companies successful in their big data efforts accept this and allow for it.

Keep options open

Companies doing well with their big data efforts continuously seek out opportunities to acquire critical big data expertise that they lack internally.  In some cases, they have employed cloud-based big data solutions to assist them with market visibility if they lacked the technology for it or to optimize profit margins by improving the performance of back-office operations.

Join Premium Today