Big data is still in its early stages, presenting companies
with both organizational and technological challenges. However, the promise of
the business intelligence that big data can deliver is too great for companies
to ignore. In August, Tech Pro Research conducted a survey
on big data trends
and found that 46 percent of respondents have
implemented or are planning to implement big data.

Other analyst reports suggest even more robust intentions
for adopting big data.

In a September 2012 Harvard Business Review
, 85 percent of respondents — consisting of Fortune 1000
companies and government agencies — said they either had big data initiatives
underway or that they were planning for them. The same percentage responded
that C-level executives in their organizations were sponsoring these

One year later, in September 2013, a Gartner research
revealed that 64 percent of responding organizations have
invested or have planned to invest in big data, compared to 58 percent of
organizations surveyed in 2012. According to the survey, media and
communications (39 percent), banking (34 percent), and services (32 percent)
were front-runners in the adoption of big data. The industry verticals
projecting the greatest number of planned investments in big data for the
future were transportation (50 percent), healthcare (41 percent), and insurance
(40 percent).

In a
recent press release
, Frank Buytendijk, research vice president at
Gartner, summed up the survey results. “For big data, 2013 is the year of
experimentation and early deployment,” he said. “Adoption is still at
the early stages with less than eight percent of all respondents indicating
their organization has deployed big data solutions. Twenty percent are piloting
and experimenting, 18 percent are developing a strategy, 19 percent are
knowledge gathering, while the remainder has no plans or don’t know.”

Organizations are hoping to use big data to understand their
customers better and improve business processes. However, knowing how to harness
big data is still a challenge for most. This point was clearly illustrated in
the Harvard Business Review study,
which showed that:

  • Only 15 percent of respondents ranked
    their access to data today as adequate or world-class
  • Only 21 percent of respondents ranked
    their analytic capabilities as adequate or world-class
  • Only 17 percent of respondents ranked their
    ability to use data and analytics to transform their business as more than
    adequate or world-class

“One of the largest issues facing many organizations
that wish to use big data today is knowing which questions to ask, and how to
formulate questions,” said Srikanth Velamakanni, CEO and founder of Fractal Analytics, a provider
of big data analytics for companies in a number of key industry sectors. “Many
companies ask themselves if they asked the right questions of their big data in
the first place.”

Velamakanni and other analysts believe that the key is to
learn how to ask the right “trigger” questions, which can push out
highly actionable information that was never considered before. This is also an
area where there are few established technologies that can get early big data
practitioners to the results they desire. Consequently, the task for corporate
C-level executives and those charged with developing a successful big data
analytics program is to develop the right sets of business practices,
analytics, and technologies that can unearth new answers to old business problems
so they can achieve breakthroughs for the business.

Developing a business case

Despite the challenges, some organizations are already
implementing big data analytics with great success:

The Hamilton County Department of Education

The Hamilton County Department of Education oversees nine
K–12 school districts consisting of 76 schools in and around Chattanooga, TN. Its
total student population is 40,000, served by a staff of 6,400 employees.

Hamilton County wanted to improve its student high school graduation
rates and recognized that to do so, it needed to understand sooner which
students were most at risk of not graduating. Early risk detection would give
educators an opportunity to intervene with those students before it was too
late. In the past, Hamilton County had relied on manual reports, where it was
impossible to gain performance insights that could assist educators in identifying
at-risk students and how to best help them. Using big data analytics, Hamilton County
consolidated all student information into a single data repository that enabled
it to more fully construct student risk profiles, which then became actionable
for educators. Hamilton
County improved its high school graduation rate by 10 percent

General Electric Co.

In 2011, General
Electric Co. (GE) invested one billion dollars in a big data effort

that will position it to better manage two-thirds of its business, which was
$94 billion in 2011. This business is characterized by digital data emanating
from sensors and other digital devices that are embedded in machines such as GE’s
jet engines, turbines, trains, and hospital MRI equipment. GE refers to it
collectively as the “industrial Internet.” Harnessing industrial
Internet data will enable GE to help customers identify maintenance problems
before they occur, improve fuel efficiency, and make other operational
improvements that could add up to trillions of dollars in savings.


Ooyala, a video services company, uses big data to furnish
media firms like Bloomberg, ESPN, Telegraph Media Group, and Yahoo! Japan with “actionable
analytics” that provide detailed visibility of how consumers are using their
video content and how that content can be optimized to produce more revenue. Ooyala’s
analytics engine processes over two billion analytics events each day
derived from nearly 200 million viewers worldwide who watch video on an
Ooyala-powered player. With the help of big data analytics, Ooyala is creating
business value for its clients and revenues for itself.

Each of these companies either built revenues or improved
performance by carefully constructing a business case that everyone knew would
add value to the business — along with delivering immediate returns from the
implementation of a big data program.

A second element of success in each of these use cases was
that the initial big data project paved the way for future “take off”
projects that could use the same data and analytics foundation that the initial
project built. For example, Hamilton County’s student profiling analytics
methodology was capable of being applied to “high success” students
and other segments of the student population as easily as it was being applied
to at-risk students. GE’s industrial Internet, sensor-based data capture and
analytics is capable of growing with GE’s business as the company brings more
sensor-based products to market. And Ooyala’s business model can be expanded to
many other venues (and revenue opportunities) beyond those of its current clients.

Querying data in new ways

Companies have already demonstrated that carefully
constructing business cases with definite, prescribed boundaries for outcomes
brings success to big data analytics. However, there is also a counterargument to
such tightly construed business cases that emerges during the development of
big data queries. It rests on the fact that for big data to deliver innovative
and groundbreaking insights, the knowledge workers who are querying the data
have to leave enough room for experimentation to see what the data can offer.
According to Fractal Analytics’ Velamakanni, if companies focus their initial
queries and business cases for this big data too narrowly, they may miss other
areas of revelation that the data can potentially provide.

“We find this often when we work with clients,” he
said. “Companies come to the big data analytics already narrowly focused
on solving a particular problem. This is different from just looking at the
data itself.”

Objectively looking at data to see what new information it
reveals is difficult in most corporate environments, where the pressure to
produce results is continuous and never-ending. On the other hand, in a more open-ended
data query approach, it’s possible to acquire fresh insights that a narrower
focus on business case or query structure might have preempted.

Finding a balanced path that allows for both highly focused
research and insightful experimentation with big data is a challenge for many
enterprises. To facilitate this balanced path to big data intelligence
acquisition, organizations are finding that they require new skill sets and work
groups to carry out their big data missions.

Acquiring a big data skill set

For many corporate executives today, hiring or developing personnel
is the biggest problem they are facing with big data. In its 2013 Digital IQ
Survey, PwC Advisory Services
surveyed more than 1,100 business and technology executives and found that only
44 percent of those surveyed felt that they had the internal talent to capitalize
on big data.

The Tech Pro Research survey revealed problems with hiring
and retaining qualified staff, as well. Although 33 percent of respondents
reported no hiring obstacles, 46 percent said they had run into problems filling
big data job roles.

The skill set needed for an effective enterprise big data team
is actually a
combination of old and new skills
. The “new skills” needed for
big data, which many enterprise data analysts today lack, are data science programming
skills, as well as strong background in mathematics and statistical analysis
and data modeling techniques. Great data scientists also can experiment in new and
innovative ways with data. They are traditionally coming from academic and
research environments where they aren’t afraid of failing on their way to attaining

These data scientists must be complemented with a new set of
technical hard skills, which include knowing how to build out databases that
span terabytes of data, the ability to administer software frameworks like Hadoop, and expertise
in databases such as noSQL, Cassandra,
and HBase
or in analytics  programming languages and facilities like R and Pig.

These capabilities are in sharp contrast to the skill sets
and goals of traditional corporate data and business analysts, who live by project
deadlines and are expected to produce tangible business value from every task
they do. The very nature of their work precludes them from experimenting — and
the very thought of failure from a data experiment that doesn’t work out can threaten
a career.

Nevertheless, enterprises must find ways to mesh traditional
data and business analytics skills with more “new wave” data
discovery techniques that allow for experimentation. To illustrate, 59 percent
of companies responding to a 2012 survey conducted by analyst firm Information Difference said
that their big data projects were “highly linked” to their master data
repositories (customer data, product data, etc), and that master data was driving big data.
They were using their master data from systems of record data as “vectors”
into big data queries that were probing for customer behavior patterns, preferred
product combinations in certain markets, etc.

The bottom line for enterprises is this: To develop an
effective big data team, they need an interdisciplinary group of team members who
can work with both unstructured and semi-structured “new” data and
enterprise systems of record data in new ways.

The model data science team in a company now consists of a
business process analyst who is understands the business and what it wants to
know; a data scientist who has the statistical, mathematical, and programming
skills and data know-how to create algorithms that extract meaning from big data;
and a technical IT person who can interact with the corporate database
architect and coordinate physical HPC (high-performance computing) resources in
the data center for purposes of scheduling and executing big data jobs. Although
some organizations choose to place this data science team under end business areas,
a growing number is finding that IT is the best place to assign the team.

According to Silicon Valley Data Science, a big data science
consulting company, “The best data driven product companies build with small,
agile teams of data scientists and engineers who are good at working together
The engineer understands the algorithm that the applied math person ends up
using, because they have been discussing it as it evolves. Both are aware of
each other’s progress and can offer each other ideas. This approach results in
an effective algorithm that scales, and teammates who will share a beer in
celebration, rather than retreating to separate taverns to drink away their
mutual frustration.”

This is where many enterprises are now — in the process of
constructing interdisciplinary teams with a mix of business, analytics, and
technical skills that will drive their big data efforts.

Final recommendations for companies

Big data presents numerous challenges, but potential
benefits are too compelling to ignore. Accordingly, most enterprises are hard
at work defining business cases, organizational structures, and technology
architectures that will be capable of supporting their big data efforts now and
in the foreseeable future. Some organizations are already doing this with great
success. The best practices that characterize their success include the

Start with a big data plan that has long-term duration

Big data analytics should establish a foundation for future
research as well as produce answers to current questions. Companies that are
building momentum with big data projects understand this. They carefully
construct big data query structures and repositories for reusability as well as
for present use to derive long-term value from their investments.

Construct an interdisciplinary data science team that can work with both IT
and the business

Blending technical and business savvy has never been easy, but
it is a critical element in corporate data science teams. Most successful
enterprises are assigning the data science team reporting relationship to IT,
but the business process analysts on this team have strong ties and accountabilities
to the business as well. The core team of business analysts, data scientists,
and technical support personnel works shoulder to shoulder in a tightly
integrated and collaborative environment.

Blend data experimentation with “fast” data results

The global business environment is fiercely competitive, and
C-level executives and line managers expect answers to problems right away. In
some cases, big data can deliver this. However, in other cases, the data
science team needs the opportunity to experiment with the data if they are to
come up with new and groundbreaking insights. Companies successful in their big
data efforts accept this and allow for it.

Keep options open

Companies doing well with their big data efforts
continuously seek out opportunities to acquire critical big data expertise that
they lack internally.  In some cases,
they have employed cloud-based big data solutions to assist them with market
visibility if they lacked the technology for it or to optimize profit margins
by improving the performance of back-office operations.