Big Data

How the "pure magic" of Mesos can make your enterprise run like Google

Building and scaling a datacenter is hard. Mesosphere is making it easier. Matt Asay explains.

Big data

Apple may be the envy of the moneyed class, plump with high-margin and high-volume sales. But most of enterprises aspire to be Google when it comes to putting data to work.

To get there requires more than a willingness to make datacenters a core competency. Enterprises actually have to think differently about software, with open-source projects like Apache Mesos helping uber engineers create datacenter operating systems, treating hardware like "cattle" instead of "pets."

But it's still hard, and you still have to be pretty "uber" to make it work.

Which is what makes Mesosphere'slatest news particularly interesting. On its face, Mesosphere has kicked off a developer program and launched an SDK to make would-be Mesos developers' lives a tad easier. But really it's part of a grand plan to bring Google-like power, with Apple-like simplicity, to the enterprise.

I previously spoke with Mesosphere CEO Florian Leibert about its concept of the "datacenter operating system," and the company has been on a tear. It hired the creator of Mesos from Twitter, plucked a top distributed systems research professor from StanfordUniversity, swallowed a New York design firm, closed a Series B round led by Khosla Ventures, and announced the release of its Datacenter Operating System (DCOS).

Mesosphere CEO Leibert talked to me about how the company thinks about a new deployment model for distributed application software stacks in the datacenter and cloud.

TechRepublic: First, tell me about the new developer news.

Leibert: We're offering developers and partners tools to make it easier to build distributed applications. Let's be honest, building distributed systems is hard. They need to be scalable, fault tolerant, highly available, consistent, secure, elastic, and efficient.

To achieve these properties, distributed systems require many complex components to work together in a very sophisticated manner. For example, Apache Hadoop depends on a highly fault tolerant file system (HDFS) to provide high throughput for processing multi-terabyte data sets in parallel on large clusters.

In the past, each new distributed system, such as Hadoop or Cassandra, had to build its own understructure for messaging, storage, networking, fault tolerance, and elasticity.

Fortunately, systems like Mesos and DCOS simplify the task of building and managing distributed systems by providing operating-system-like primitives for the key building blocks of distributed systems. Mesos abstracts away CPU, memory, storage, and other resources so that developers can write distributed applications as if their datacenter clusters were one giant machine.

Moreover, now that there are dozens of these distributed systems written native to Mesos and DCOS, we're seeing a new model building backend stacks composed from these building blocks to deliver real-world use case solutions.

TechRepublic: What do you mean by "stack" under this datacenter operating system model?

Leibert: Modern developers building back-end applications typically need to work with multiple services to create what we would traditionally think of as "stack" functionality. For example, the best practice today for building a real-time data pipeline means a developer would likely use Apache Kafka and Akka for messaging and tap into a distributed datastore like MemSQL or Cassandra and run analytics via Apache Spark.

To set that up in the old world as a developer, the ops team had to provision separate siloed clusters (one for each framework) and stand them up, statically, with configuration management tools like Chef and Puppet.

Claiming the resources and standing up these silo'd services can take weeks or months, and if there were development and staging environments, the result would be a multidimensional matrix of clusters, each with its own operational complexity, overhead, and correspondingly low utilization.

In this new model of a datacenter operating system, organizations can compose these complex stacks quickly by installing distributed services with simplicity that reminds you of installing an app from the iPhone app store.

TechRepublic: Does that mean, literally, browsing a catalog of services and picking the ones you want to install?

Leibert: Yes, very much so. One of the things that we've done at Mesosphere, that feels like pure magic, is build a sophisticated packing system that makes it possible to download and deploy a datacenter-scale distributed system via a single command on the command line.

For example, to install Kafka on your cluster, you simply type dcos package install kafka. It's that easy. In seconds, the Kafka package is downloaded and instantiated on the cluster in a highly-available configuration, ready to scale to hundreds of nodes if need be and ready to start accepting data streams.

Our DCOS approach lets developers focus on the service they're creating and the business logic of the application. We've abstracted away the back-end complexity. Services just work, getting the resources they need from the cluster. And we've taken ops out of the equation. Those ops teams can now focus on more interesting and important things in running a datacenter than managing and responding to developer help tickets.

Also see

About Matt Asay

Matt Asay is a veteran technology columnist who has written for CNET, ReadWrite, and other tech media. Asay has also held a variety of executive roles with leading mobile and big data software companies.

Editor's Picks

Free Newsletters, In your Inbox