The vast majority of enterprises (84%) finally have a clue about big data, according to a new CA Technologies survey. These same companies report higher revenue and better insight into customer requirements.
These are great, if unsurprising, results.
What is surprising, however, is how few of these companies are actively working to expand access to data among their employees. After all, if data is only as useful as the questions we ask of it, expanding the population of people asking questions seems like a good idea.
The data science cabal
Despite 92% of the enterprises surveyed still struggling with their big data projects, a whopping 90% “are experiencing or anticipate seeing, more effective targeted marketing and selling campaigns.” This, in turn, is generating cash. According to the report, 88% of enterprises see or anticipate increased revenue.
That’s the good news.
The bad news is that enterprises may be getting their priorities all wrong as they move forward:
I understand that enterprises must balance competing priorities. No one has an unlimited budget.
However, it strikes me as odd that expanding access to a broader swath of the enterprise wouldn’t top the list. When a mere 35% of enterprises express interest in “improving ease of use for internal users,” they’re essentially ensuring that big data remains the province of a select few data scientists, as Mare Lucas writes:
“For years, the BI and data analytics conversation was framed around how to aggregate massive volumes of data and then unleash the data scientists to find the value. Today, despite the information deluge, enterprise decision makers are often unable to access the data in a useful way. The tools are designed for those who speak the language of algorithms and statistical analysis. It’s simply too hard for the everyday user to ‘ask’ the data any questions — from the routine to the insightful. The end result? The speed of big data moves at a slower pace… and the power is locked in the hands of the few.”
Sure, data scientists might be the best qualified to comb through complex data sets to find patterns. Mitchell Sanders has persuasively argued that the most effective data scientists are those that combine programming, math, and domain knowledge. That’s a difficult trifecta to master.
But we’re not talking about turning everyone into a PhD-toting data scientist. Rather, the key is to find ways to democratize data.
Etsy, for example, built out its Hadoop cluster in such a way that it delivered a 10X improvement in utilization across the company (and not merely the data science gearheads). In fact, 80% of its employees access the Etsy data warehouse on a weekly basis.
This is what every company needs, given the importance of data.
Nor do you have to be an Etsy to get this kind of result. Tools like Tableau help enterprises to tame data and make it approachable. Given that 98% of the enterprises surveyed acknowledge big investments are needed to get the most from big data, they would do well to spend some of that money on improving access.
Oh, and about that cloud thing
Ironically, one of the areas many companies will waste plenty of cash is in scaling out their infrastructure.
According to the report, the most-cited roadblock to big data success is insufficient existing infrastructure (noted by 32% of respondents), followed by organizational complexity (27%), security/compliance concerns (26%), lack of budget/resources (25%), and a lack of visibility into information and processes (25%).
As I’ve written, big data really needs elastic infrastructure, because you don’t want to hardwire resources when the nature of the questions you need to ask will constantly evolve.
Or, as Amazon Web Services (AWS) data science chief Matt Wood puts it, “Those that go out and buy expensive infrastructure find that the problem scope and domain shift really quickly. By the time they get around to answering the original question, the business has moved on.”
By building data infrastructure as an elastic cloud service, and focusing that service on as broad an audience as possible, enterprises will discover far bigger benefits from their data than a few data scientists laboring away in some data center ever will.