Recently, I had a chance to speak with the leaders of some companies focused on the business and technology sides of Big Data and predictive analytics. We spoke about what companies can do today to meet the Big Data challenges of tomorrow.

Here are some insights I gathered from those discussions:

Hype and then Fear, Uncertainty, and Doubt (FUD)

Scott Knau, CEO of Teradata, says a big issue in the Big Data market right now is the hype and FUD that challenges its overall adoption. Hype and FUD send out mixed messages that decision makers can misinterpret, which complicates the need many enterprises have to get value from Big Data while avoiding it becoming another problem set.

Olly Downs, senior vice president of Globys, a Big Data firm, uniquely sums up getting past this challenge, “It’s all about getting into a situation where Big Data is no longer concerning.” He elaborates. “In some ways Big Data is on the same path as business intelligence was on.” It seems companies learning from their BI past might help lead to their success in the Big Data future.

Scarce talent

One topic that came up frequently during the discussions is the shortage of Big Data talent. The good news is that a growing number of Data Science undergraduate, graduate, and certificate programs are launching around the United States.

Corporate/higher education collaboration is key to the success of such programs. Downs and Globys are working with a higher education institution in their local area to establish a Big Data certificate program.  He also adds, “Data science education is also shifting, with course offerings appearing across a number of different majors, especially business.” He stresses that companies should align with educational institutions. Getting your data scientists and other Big Data team members involved in the local data science education scene can be an investment in your own Big Data future.

A corporate-based model for building Big Data talent can be the more expedient route for countering the issue of talent scarcity. According to Knau, “Teradata is building out their Big Data talent pool through a mix of toolsets and education. Teradata also has universal data architecture as a reference blueprint in place for our professional services teams and customers.”

The Teradata model, or something similar, could be key to overcoming the scarce talent challenge. As a company, they’ve worked hard to find and/or nurture the right people by:

  • Building out toolsets for their professional services teams.
  • Building out existing knowledge worker skill sets about technology and the business sides of Big Data.

There is a “sweet spot” between tasks requiring a data scientist and those where the tools of a skilled knowledge worker come into play. Companies finding their sweet spot will be in the best position to seize Big Data as a business platform. However, such efforts need to begin now to find what works for your staff and Big Data projects. An example of one “sweet spot” is the “discovery platform” that Teradata deploys with their teams.

Teradata has invested heavily in their discovery platform to use it as a bridge between their data scientists and other team members. It allows data scientists to create modules that other team members can reuse on a given project. Knau tells me that their discovery platform enables them to conserve time and put Big Data tools into the hands of their staff that have SQL experience. Data scientists can then focus on higher-end tasks not just the routine tasks that can be accomplished with the right combination of tools and knowledge workers.

Growth of data

Russ Kennedy, VP of strategic marketing for CleverSafe, sums up the growth of data: “There is obviously a deluge in information as  Big Data expands to capture social media, mobile, and other data intensive areas.” He adds, “This leads to questions about how to best capture the data more economically and reliably.” Managing the growth of data in your organization requires a lot of work around designing and managing, replicating, and protecting the data.  Planning a Big Data storage strategy is definitely part of meeting this Big Data challenge of tomorrow.

As a storage provider, Kennedy and CleverSafe see another challenge from the growth of Big Data. “There are the legal aspects of storing Big Data–especially what data can reside where–that can impact where a company stores their Big Data,” according to Kennedy. “If you have international data storage plans and must adhere to compliance programs, take a look at the potential legal issues up front that might come from storing your growing data across geographic locations ( especially international borders).”

FICO, a major Big Data player, also recognizes the impending data storage challenge, which a spokesperson referred to as an “arms race.” FICO is showing that, as the ability to create and store data continues to expand, you have to innovate on the analytics side to turn that data into useful (and secure) information. The FICO spokesperson points to credit and fraud as an example of where technology makes credit easier and better for the borrower/economy, but also provides more opportunity for fraud.

Different uses of data

While many Big Data headlines are about the tracking of customer behavior across multiple industries, Bug Data plays an important role in heavy machinery industries like mining, construction, and industrial manufacturing.  Experts from Hitachi Data Systems (HDS), including Michael Hay, Vice President of Product Planning, Sara Gardner, Senior Director, Software Product Marketing, and Asim Zaheer, Senior Vice President, Worldwide Marketing, talked about how they use Big Data in their company.

They put analytics to work onboard their machinery for streaming data back about the equipment’s operating conditions and progress. Since a data scientist won’t be down in the mine with the equipment, it means more automation to enable analytics on the equipment to power decision making at the surface where the machine operators work.

“In turn, machine data growth is going to require different tools and technologies because one size doesn’t fit all with machine data,” advises Hay of HDS. “While machine data is sparking data growth, it’s going to spark greater innovation in both software and applications.” In fact, HDS does much of their innovative Big Data work inhouse.

Face the Big data challenges of tomorrow starting now

While Big Data challenges are on the rise, putting in the tools, infrastructure, process, and staffing investments now can help position companies to prosper in the Big Data world of the future.