In mid-2013, O'Reilly Media observed that analysts working with big datasets five-six years ago made queries and got the results back overnight, with the wait time since then moving to minutes as Hadoop and other tools entered the market."But the revolution continues," said O'Reilly. "Analysts now demand sub-second, near real-time query results."
Enterprises might want results like this, but the reality for most companies is that big data queries are run in batch mode against data marts and warehouses that contain reasonably fresh but still static data. The good news is that this data frequently gets refreshed, and it certainly can meet a plethora of enterprise "needs to know." However, if the need is for actionable information in real-time, storing data in marts and warehouses and using hard disk-based systems for data retrieval and storage isn't going to meet the need.
Enter the in-memory data grid (IMDG), which operates on real-time data in ways that most high-performance computing (HPC) and enterprise big data processing can't. IMDG also meets the needs of systems that range from stock trading, to airline and hotel reservations, e-commerce, smart grids, and credit card fraud detection.
"With IMDG, high availability of your application is baked right into the computing grid," said David Brinker, COO of ScaleOut Software, which provides enterprise in-memory data grids. "If you lose a server in your computing cluster, you can just keep going. The solution is scalable across multiple servers, and you can also use asynchronous replication for purposes of disaster recovery."
What makes IMDG ideally suited to real-time big data processing and analytics is its reliance on in-memory computing instead of external storage resources such as a hard disk.
IMDG holds data that are undergoing changes in-memory instead of going out to disk. Within the in-memory data grid is a real-time analytics engine that makes real-time changes to data and that also sends out alerts (e.g., if there is a sudden outage at a specific location in the smart grid). Large amounts of data are stored in-memory, and there is also a store of the original, unchanged persistent data that is stored on an external disk.
ScaleOut Software CEO Bill Bain calls this process "analytics for the moment." "IMDG is like business intelligence, except that the analytics, compute, and storage are all in-memory," he said. "It allows businesses to analyze data while it is changing."
For businesses in dynamic operating environments, real-time insights into operational changes can mean millions in revenue that they would otherwise miss out on.
Bain gives the example of a hedge fund that was using a SQL server that it queried for stock price changes. "Unfortunately, they couldn't get the information on price changes any faster than every 15 minutes," he said.
The company needed real-time results, and it identified several objectives. It wanted to see the long and the short position for each stock, and it wanted to see the signals on when to buy or sell. Using IMDG technology, the company was able to move from 15-minute intervals on stock price monitoring to intervals of 200 milliseconds. When it was time to make a trade, alerts were automatically triggered and routed directly to the trader.
Similar needs for real-time analytics that return information in milliseconds can be made for retail ordering and inventory systems, and for systems in many other industries.
Does this mean that every big data application requires this kind of velocity? No. But for those who need true real-time big data analytics in their businesses, it's reassuring that IMDG technology can deliver it.
- Hadoop success requires avoidance of past data mistakes
- Does open source matter to Hadoop?
- How Big Data is changing the security analytics landscape
Mary E. Shacklett is president of Transworld Data, a technology research and market development firm. Prior to founding the company, Mary was Senior Vice President of Marketing and Technology at TCCU, Inc., a financial services firm; Vice President of Product Research and Software Development for Summit Information Systems, a computer software company; and Vice President of Strategic Planning and Technology at FSI International, a multinational manufacturing company in the semiconductor industry. Mary is a keynote speaker and has more than 1,000 articles, research studies, and technology publications in print.