Data streaming on the rise according to developers

A survey of 800 IT professionals found that developers are trying to keep with up with increasing industry demands for real-time data services.

Is a data science degree worth it? Data scientists are in demand, but a master's degree in the field may not open as many doors as you think.

A report from analytics company The New Stack and microservices creator Lightbend says more than ever before, companies are turning to streaming as an avenue to relay data.

"We see a renaissance right now where developers are being asked to be a lot more 'data smart,'" said Mark Brewer, CEO at Lightbend. "Streaming data is table stakes for the most interesting future use cases—Artificial Intelligence and Machine Learning most notably—and that's giving rise to the number of programming languages, frameworks and tools for building and running streaming data-centric applications."

Although many organizations have struggled to find developers capable of handling complex systems, they still have an understanding that streaming is the way of the future whether they are ready for it or not.

"Competitive pressure is driving organizations to embrace streaming data to extract useful information more quickly from incoming data, as well as to serve impactful results from Artificial Intelligence/Machine Learning to customers," the report says.

"The 'always on' characteristics of streaming pipelines require the same scalability, resiliency, and efficiency that microservices deliver, which is why mature tools, like Akka Streams, that bridge the gap are popular."

SEE: Tips for building a successful career as a software engineer (free PDF) (TechRepublic)

For their report, the New Stack spoke to more than 800 IT professionals, mostly from Europe and North America, and found that the number of companies processing data in real time for Artificial Intelligence/Machine Learning jumped from 6 percent in 2017 to 33 percent this year.

There was also a marked increase in real-time data processing used for Internet of Things (IoT) devices, which the report said would only increase as these devices become more popular and companies seek to make them friendlier to more kinds of content.

Thankfully, the skill-levels of developers are improving as demand for more complicated microservices and streaming systems grows. The New Stack said "scaled out, distributed architectures are built by teams of developers whose experience dictates what data streaming technologies to adapt into the services they are building and managing. A data streaming architecture built for microservices becomes a salient decision."

But they cautioned that the barriers to developing and managing the infrastructure needed to run a streaming platform were still high. They called it a "sophisticated technology that requires an understanding of how to keep a long-running application resilient and able to scale up and down."

"Developer teams are adapting and trying new workflows but it can be very risky when the impact on performance is unknown. The right knowledge for the problem is still the biggest challenge in adoption of nascent data streaming technologies," they wrote.

"Further data streaming adoption will soon follow as their services become more advanced and machine learning and artificial intelligence become more important to achieve higher business value."

The report found that microservices and streaming were used most often with application monitoring and log aggregation, mostly because these both required the ability to spot problems quickly as opposed to waiting for analysis to happen offline.

Companies are choosing to aggregate and store data in "timeseries" databases that really only store easily-analyzed metrics instead of keeping all of the raw data that is churned out by modern applications and systems.

"The application's end user is a particular concern when stream processing is utilized in applications that require non-technical teams to actively use an application," they wrote explaining that many organizations have been slow to adopt modern technology because not enough of their staff members are capable of dealing with complex systems.

"Thus, organizations that utilize stream processing are more concerned about one type of user (DBAs) when they have an active data warehouse use case, and another (business analyst) when streaming data is integrated into dashboards for operational insights."

For more, check out "Which are the best new programming languages for software developers to learn?"

Also see

7.jpg
Image: iStockphoto/comzeal