In the last month, ‘big data’ has become big news. Oracle for one has unveiled two appliances – Big Data and Exalytics – as part of its assault on this market, which promises to allow businesses to take the unstructured data they gather – information from social media, sensors or CCTV, for example – and mine it for useful insights.

The launch of the pair this month at Oracle’s OpenWorld conference, coupled with its recent purchase of Endeca, shows how the market is hotting up. But it’s not only Oracle looking keenly at this market – EMC is also targeting the same market with its Greenplum big data software and appliances.

COO of information infrastructure products Pat Gelsinger

EMC’s Pat Gelsinger, who joined the company in 2009, says the big data market is still relatively youngPhoto: EMC

EMC’s COO of information infrastructure products Pat Gelsinger also made an appearance at OpenWorld, with EMC as one of Oracle’s partners. So how does Gelsinger see the new competition, now both Oracle and EMC are interested in big data?

“We’re both pursuing this big, new, emerging market that [big data] is enabling. It hasn’t been done before. I don’t know if we’re going to be co-operative or competitive yet on that level,” Gelsinger added.

A market undecided

The market is a relatively young one, and consequently much about it is in flux, according to Gelsinger.

“This is a huge new market – neither one of [EMC and Oracle] could necessarily look at it and say, ‘This is how it will work out’,” he said.

“It’s way too presumptive to say we’ve got that all figured out… There is so much activity going on in the space,” Gelsinger added.

But with a market so much up in the air, how does any company go about working up products, let alone those that businesses may actually need?

“By creating a bandwagon and jumping on it as fast as you can,” Clive Longbottom, founding analyst at Quocirca, told silicon.com.

And that bandwagon is in full swing. The big data market, according to Longbottom, was born out of the oil and gas market, where every explosion a company set off would generate 20 terabytes of information.

Today, it’s being pitched by vendors at a far wider spectrum of industries and evidently with some success – according to…

…the Gartner hype cycle, big data is showing the first signs of early adopter investigation this year, while the analyst has also chosen big data as one of its top 10 strategic enterprise technologies for next year.

But how much substance does big data have< beyond the vendor hype, and does it have the scope to break out of niches such as oil and gas, pharmaceuticals and healthcare?

EMC is headquartered in Hopkinton, Massachusetts

EMC is headquartered in Hopkinton, MassachusettsPhoto: EMC

“It’s not going to be too long before we start to look at organisations with 200 people in it who go, ‘Why have we got five terabytes of information and when was the last time we actually touched it? Let’s use a big data approach to have a look at what we’ve got and see if we can uncover any gold that makes us better than the next company’,” Longbottom said.

Longbottom said big data products try to tackle the problem of bringing the intellectual property of the organisation together in a manner that can be analysed and reported upon, so that execs can make a sensible decision. “That needs a completely different approach than anyone’s done before from the database, the business analytics, the storage side and even to a certain extent down at the compute level,” he added.

A global datacentre

Another piece of the big data puzzle, at least in EMC’s view, is its VPlex technology which allows data to be accessed and shared between distant datacentres, creating a shared pool of storage resource.

When VPlex was launched in 2010, EMC’s Gelsinger told silicon.com: “This year it’s a category creation so we have very modest expectations in the near term. But in 2011/12 we expect this to grow to be a meaningful product line for us. We’re looking forward to a good revenue stream out of this product line next year.”

It’s a similar message from Gelsinger now, who says the company is working on a “market creation effort”. “You have to do a fairly extensive sales cycle to get customers to understand the products,” he explained.

“We do expect the true impact of the technology will be felt next year,” Gelsinger added.

When VPlex debuted, analysts thought EMC was running ahead of the market – providing products that companies aren’t yet…

…ready for – and it appears buyers still haven’t caught up a year after launch. The long-awaited year of the cloud has yet to truly arrive, and having become comfortable with server virtualisation, organisations are only now looking to do the same with storage.

“I think it’s all following on the coat-tails of cloud,” Jane Clabby, research analyst at Clabby Analytics, told silicon.com. “Maybe a year ago, businesses were predicted to start adopting cloud technologies, and maybe they did – on a small scale with private cloud, that type of thing.

“Now you’re seeing businesses looking for hybrid cloud architectures. What can they purchase as a cloud service rather than having it internally on a private cloud, and how can you blend those two environments together? Certainly, products like VPlex can enable that,” she told silicon.com.

Quocirca’s Longbottom agrees. “VPlex will be a slow burner – cloud is not going to have happened overnight. 2012 is not the year of cloud. It will be the year where cloud starts to be mainstream but in 2020 we will still be looking at companies and saying, ‘Look, this company hasn’t fully taken to cloud yet’. VPlex and other similar techs will all be part of following this long tail.”

VPlex offers what EMC’s Gelsinger describes as “volume over distance”. At present, that distance is in the range of 1000km, or where latency is less than five milliseconds, with the likes of AOL using it to connect up datacentres along the East Coast of the US.

However, the overarching vision for such products is to have them able to link datacentres over a far larger distance – across the globe.

For most businesses, a distance of hundreds or a thousand kilometres is likely to be enough, allowing them to build in a sufficient level of redundancy – if a datacentre in one location goes down due to a natural disaster, say, another at a distance of a 100km is likely to be unaffected. But for some global players, there’s still that need to tackle greater distances.

How long until the vision of global connectedness becomes a reality? According to Gelsinger, we’re still some way away.

“The full vision is on the order of two years,” he said.

“As you stretch the distance [over which you’re] running datacentre A with B with C, that’s where more complex topology needs to be addressed and it will be a couple of years till those issues are addressed.”