Big Data and working with what you have

Until organizations get a handle on the full needs and processes for big data, they could be well served by making existing assets work for them.

Connecting with customers over a multitude of different commerce channels and understanding what they want and how they feel isn't easy. It's certainly an area where big data harvesting and a little bit of outside industry expertise can help.

This was one reason eBay Enterprises was organized by eBay as an independent company and a cloud provider of both expertise and big data management for customer experience technologies.

The question, of course, quickly became how to commandeer all of this big data management and database expertise to meet the standards of reporting timeliness that clients expected.

eBay's story

"We had clients that were looking for trends information," said Kevin Struckhoff, senior IT manager at eBay Enterprises' Los Angeles Direct Response Operation. "Some wanted to look at what their customer direct response trends had been over the past ten years, but our database reporting limitations were making it a challenge to deliver two years of historical results in a timely manner."

To meet the big data analytics needs of its clients, eBay Enterprises needed an alternate data base approach that enabled it to get to the depth of the data it was processing quicker so it could deliver more timely reports to clients. These clients wanted information on consumer trends, as well as data for back year taxes.

"One obstacle we were up against was that we didn't always have sufficient disk space on the system to run large jobs quickly," said Struckhoff. "This forced us to work on weekends to get the work done."

Extended cycle times didn't fit well with eBay Enterprise's service culture. "We have a very short time to market cycle in application development, and when it comes to maintenance, we are normally fixing a bug within one half hour and certainly within a day," noted Struckhoff. "We wanted to match this level of performance on the reporting side of our operation."

The temptation is always there to replace servers and find an entirely new suite of software, but eBay Enterprises elected instead to stick with the x86 servers in its data center that were already processing reports for clients with the help of an Informix database.

"We had experienced success with this combination, and we already understood the technology," said Struckhoff, "But what we wanted to do now was to improve the performance of our big data processing and analytics reporting."

Retaining its servers and its database technology, eBay found a relational database software application designed for data analytics processing that was able to solve some of the processing limitations in the x86 chipset by more effectively using in-memory cache with the CPU.

"For our proof of concept, we benchmarked twelve different queries that we commonly ran for our clients," said Struckhoff. "In some cases, queries that had taken 4.5 to five minutes to process in the past were now taking under ten seconds to complete. Needless to say, we also solved the problem of processing large analytics reports that could span ten or twelve years of data."

What is the takeaway from the eBay Enterprises experience?

That whether or not big data is a new frontier for organizations, the preference for many is to stick with what they know.

There is common sense in this - because until organizations get a handle on the full set of needs and processes for big data, they could be well served by making existing assets work for them if there is the ability to do so.