The question, of course, quickly became how to commandeer all of this big data management and database expertise to meet the standards of reporting timeliness that clients expected.
"We had clients that were looking for trends information," said Kevin Struckhoff, senior IT manager at eBay Enterprises' Los Angeles Direct Response Operation. "Some wanted to look at what their customer direct response trends had been over the past ten years, but our database reporting limitations were making it a challenge to deliver two years of historical results in a timely manner."
"One obstacle we were up against was that we didn't always have sufficient disk space on the system to run large jobs quickly," said Struckhoff. "This forced us to work on weekends to get the work done."
Extended cycle times didn't fit well with eBay Enterprise's service culture. "We have a very short time to market cycle in application development, and when it comes to maintenance, we are normally fixing a bug within one half hour and certainly within a day," noted Struckhoff. "We wanted to match this level of performance on the reporting side of our operation."
The temptation is always there to replace servers and find an entirely new suite of software, but eBay Enterprises elected instead to stick with the x86 servers in its data center that were already processing reports for clients with the help of an Informix database.
"We had experienced success with this combination, and we already understood the technology," said Struckhoff, "But what we wanted to do now was to improve the performance of our big data processing and analytics reporting."
Retaining its servers and its database technology, eBay found a relational database software application designed for data analytics processing that was able to solve some of the processing limitations in the x86 chipset by more effectively using in-memory cache with the CPU.
"For our proof of concept, we benchmarked twelve different queries that we commonly ran for our clients," said Struckhoff. "In some cases, queries that had taken 4.5 to five minutes to process in the past were now taking under ten seconds to complete. Needless to say, we also solved the problem of processing large analytics reports that could span ten or twelve years of data."
What is the takeaway from the eBay Enterprises experience?
That whether or not big data is a new frontier for organizations, the preference for many is to stick with what they know.
There is common sense in this - because until organizations get a handle on the full set of needs and processes for big data, they could be well served by making existing assets work for them if there is the ability to do so.
Mary E. Shacklett is president of Transworld Data, a technology research and market development firm. Prior to founding the company, Mary was Senior Vice President of Marketing and Technology at TCCU, Inc., a financial services firm; Vice President of Product Research and Software Development for Summit Information Systems, a computer software company; and Vice President of Strategic Planning and Technology at FSI International, a multinational manufacturing company in the semiconductor industry. Mary is a keynote speaker and has more than 1,000 articles, research studies, and technology publications in print.