Google Cloud Composer makes it simple to build cloud workflows with Python

The service is based on Apache Airflow and supports BigQuery, Cloud Dataflow, and many more tools.

The 3 technologies that are critical to the evolution of cloud computing
Building a slide deck, pitch, or presentation? Here are the big takeaways:
  • Google has launched its Apache Airflow-based workflow management tool, Cloud Composer, in open beta.
  • Google Cloud Composer could make it easier to connect seemingly disparate Google Cloud services, offering a platform more competitive to Microsoft and AWS.

On Tuesday, Google announced the open beta of Cloud Composer, a managed service that makes it easier to create workflows and automate processes with certain services on Google Cloud Platform (GCP).

According to a Tuesday blog post, Cloud Composer is a managed Apache Airflow service, and can help orchestrate workflows whether they live on-premises or in multiple clouds. The goal is for GCP admins to be able to use one service across environments to support their workflows.

Cloud Composer can be used to author, schedule, and monitor enterprise workflows. The workflows made through it are portable, so they can be easily moved if a user were to migrate their environment.

SEE: Cloud migration decision tool (Tech Pro Research)

Since Cloud Composer is a managed Airflow service, users won't need to install or manage workflows themselves. Additionally, Apache Airflow is an open source project, and Google has already been contributing to it, the post said.

Within Cloud Composer, users have access to features including client tooling with the Google Developer Console and Cloud SDK. Additionally, a Cloud Identity-Aware Proxy can help offer easy access to the Airflow web UI, the post said. The Composer also features simple Airflow runtime and environment configuration with support for plugins.

Stackdriver logging and monitoring is also a part of the initial beta release, the post said, along with identity access management (IAM), directed acyclic graph (DAG) management, and Python (PyPi) package management.

"Cloud Composer gives us the flexibility we need in workflow orchestration. The ability to define our workflows in Python and use a range of graphs to monitor them makes the challenges of data processing easier for us," Akshar Dave, engineering manager for Demandbase, said in the post.

Pricing for Cloud Composer is consumption based. Interested users can find more information in the official documentation.

Also see