Big Data

How to find the real value in operational big data

One of the best ways to get the most from massive amounts of data is to implement an operational intelligence system. Here's how to apply big data analytics to operational data.

Image: iStock/Aleutie

Of the three Vs of big data—volume, velocity, and variety—volume is the biggest, literally. Processing large volumes is a signature characteristic of big data analytics; it's not necessarily the most important or the most challenging, but it fosters a strategic tipping point when incubated properly.

There are many ways to use large volumes of data for a strategic advantage, and operational data is one of my favorites. I've defined big data in competitive terms as: the massive amount of rapidly moving and freely available data that potentially serves a valuable and unique need in the marketplace, but is extremely expensive and difficult to mine. What I like most about operational data is that it's freely available to you, but completely unavailable to your competition. Let me show you where the real value is in your massive amount of free operational data.

SEE: Quick glossary: Business intelligence and analytics (Tech Pro Research)

What you need to understand about Operational Intelligence

I was working with the leaders at Cargill, a large food and agricultural business that owns processing plants all over the world, and we had an interesting discussion about Operational Intelligence (OI) and how their company could benefit from expanding their base of data.

Since I started as a business intelligence and data warehousing consultant, my view of OI started as the natural outgrowth from an operational data store. When I moved into management consulting, I took on a perspective that's more oriented to business operations, and the value information can provide to that function. The two are not mutually exclusive, but the shift in perspective is worth noting.

There's a hierarchy that's important to understand. Data that's collected for an individual unit (plant, facility, site, etc.) can be referred to as Site Intelligence. We can then think of OI as the aggregation of site intelligence collected from all units that comprise business operations. When you combine OI with other important business functions like Sales & Marketing, Finance, and Customer Service, you have real business intelligence. OI represents a significant point of strategic maturity for the company.

The strategic importance of OI

OI represents breadth of knowledge—an important and challenging milestone in a company's analytic maturity. If you have 1,000 sites around the world and each site collects, processes, and analyzes its own operational data, then the most any one site can know is what's within its own walls. This is a common situation, as there is typically one site manager who manages data in a way that best helps him or her accomplish the plant's performance goals.

Consolidating and standardizing site data broadens the company's information base for the benefit of all. With a centrally organized OI system, site managers can garner insights into how other sites are doing; it opens channels for data-based performance ranking, continual learning from best practices, and accurate benchmarking. Executives benefit from having access to individual site data and aggregated analytics to drive overall strategic performance. Everyone benefits once the OI system is deployed and matured.

How to build an OI system

Implementing an OI system is not an easy task—there are strategic and technological challenges to overcome. You should handle the strategic part first and then address the technical aspects.

You must first understand how to conceptually rationalize all your sites. What is the strategic importance of all the sites in your business operations, and how will you measure performance in aggregate? How do business operations contribute to the overall health of the company, and how will you communicate that through key performance indicators? This exercise should result in a scorecard that aligns with your technical efforts.

Metadata analysis is your next step. The strategic team and the technical team must work together to map data from the scorecard all the way down to the sites. Generalization, filtration, and transformation are three key activities this team must perform. Generalization involves mapping up similar data points in to one common concept; filtration is the purposeful exclusion and inclusion of data; and transformation is the rule set that moves included data to common concepts.

Finally, hardware needs to be considered—it wouldn't be big data if it wasn't extremely expensive and difficult to mine. That should be a clue in your approach. If, after developing your scorecard and metadata analysis, you ascertain that you can tackle this with a relational database, you don't have a big data problem.

A real-life example of OI in action

To help you see this in action, I'll give you a use case from my work at Chevron.

Chevron has several operated refineries around the world that process crude oil into other petroleum products. Each refinery has several plants that each use a process technology (crude distillation, cracking, etc.). In total, I worked with about 250 plants to help them transition to standardized, risk-based asset strategies.

To accomplish this, we first needed to understand the risk profile in every plant. We could have stopped there, but we didn't. Recognizing the opportunity for OI, we brainstormed on how we could compare risk profiles between plants that have similar process technologies. This is more than just a one-time effort—implementation of this requires constant monitoring of operational data to assess risk profiles in real-time. And although we could keep this at the site level, the risk profiles are far more accurate when OI from similar plants are incorporated.

SEE: Big Data Policy (Tech Pro Research)


Volume is one of big data's biggest challenges (pun intended), and business operations is an area in many companies that throws off large amounts of data. The application of big data analytics to operational data is a terrific way to get your OI efforts in place, if it makes sense.

Even if you suspect that you'll need fancy technology to aggregate and process all the operational data that's coming from all your sites, that's no reason to drive straight to the big data store to purchase hardware. Start with a strategic scorecard and a thorough analysis of how you'll map data down to the sites.

Once everything's in place, and you have a good business case, go for it! Operational insights are just waiting to be discovered.

Also see

About John Weathington

John Weathington is President and CEO of Excellent Management Systems, Inc., a management consultancy that helps executives turn chaotic information into profitable wisdom.

Editor's Picks

Free Newsletters, In your Inbox