How Microsoft Azure Databricks can help companies speed big data and AI adoption

The analytics platform based on Apache Spark and optimized for Microsoft Azure will help businesses better integrate and scale machine learning projects.

Microsoft Azure ... in less than two minutes
Building a slide deck, pitch, or presentation? Here are the big takeaways:
  • Azure Databricks, the Apache Spark-based analytics platform optimized for Azure, is now generally available from Microsoft.
  • Azure Databricks aims to help businesses speed up and simplify the process of building big data and AI solutions.

On Thursday, Microsoft announced the general availability of Azure Databricks, the Apache Spark-based analytics platform optimized for Azure, aiming to help businesses speed up and simplify the process of building big data and artificial intelligence (AI) solutions.

Apache Spark is an open source standard for advanced analytics, machine learning, AI, and big data, that has seen rapid enterprise adoption in recent years, according to a Thursday Microsoft blog post. It was created by the founders of Databricks. With Azure Databricks, Microsoft aims to help businesses work with data in a faster, easier, and more collaborative way.

The new offering could help continue Microsoft Azure's rapid growth in the enterprise, especially as companies look to integrate AI into products and processes. While Amazon Web Services (AWS) remains in the cloud lead, adopted by 64% of firms in 2018, Azure is quickly catching up, and is now used by 45% of firms, according to RightScale. The lead of AWS among large enterprises is also shrinking, RightScale found: While 68% of enterprises use AWS, that number represents only 15% year-over-year (YoY) growth in adoption. Comparatively, Azure experienced 35% YoY growth, to reach 58% of enterprises.

SEE: The Power of IoT and Big Data (Tech Pro Research)

Azure Databricks was developed in part to improve user productivity in developing big data applications and analytics platforms, the post noted. The interactive notebooks used with Azure Databricks allow data science teams to work in popular languages such as R, Python, Scala, and SQL, and bring in all of their data sets to create machine learning models, rather than only a sample. The solution's native integration with Azure makes it easier to build end-to-end products, the post noted.

Using Azure Databricks, the company renewables.AI was able to increase the productivity of its data science team by more than 50%, according to the post. "Instead of one data scientist writing AI code and being the only person who understands it, everybody uses Azure Databricks to share code and develop together," Andy Cross, director of renewables.AI, said in the post.

With Azure Databricks, Microsoft also wants to help customers scale globally. The solution provides a fully managed, cloud-native service that automatically scales based on an organization's needs, according to the post, and simplifies the process of building data pipelines and deploying machine learning models at scale.

SEE: Quick glossary: Hybrid cloud (Tech Pro Research)

"Every day, we analyze nearly a terabyte of wind turbine data to optimize our data models," Sam Julian, data services product owner for E.ON, said in the post. "Before, that took several hours. With Microsoft Azure Databricks, it takes a few minutes. This opens a whole range of possible new applications."

Finally, Microsoft has built in strong security and compliance policies, including enterprise-grade SLAs, simplified security and identity, and role-based access controls with Azure Active Directory integration. "As a result, organizations can safeguard their data without compromising productivity of their users," the post said.

Azure Databricks is integrated with other Azure services, including the Azure SQL Data Warehouse; Azure IoT Hub, Azure Event Hubs, and Azure HDInsight Kafka clusters; and Azure Blob Storage, Azure Data Lake Store, Azure SQL Data Warehouse, and Azure Cosmos DB.

Also see

Image: iStockphoto/monsitj