Pagoda Box, a cloud service provider in Idaho, is launching V2 of its Platform as a Service (PaaS). The new V2 service is a complete overhaul of the technology stack, using new hardware, networking, operating system, and provisioning tools. What makes an organization go to such lengths to build a new PaaS?

Pagoda Box and the V1 PaaS

Pagoda Box, like other PaaS offerings, is built for customers who find infrastructure a distraction, not a competitive advantage. Tyler Flint, Pagoda Box CTO, summarized the company’s mission. “The majority of companies out there have a specialism that keeps them relevant in their market. They don’t also specialise in infrastructure and provisioning and deployments. We’re trying to bottle up the deployment and infrastructure management layer.”

Six years ago Pagoda Box was a software house, before the cloud revolution had made much impact on data centers and delivery times. Flint said the Pagoda Box PaaS started life as an internal project. “All our modern applications didn’t fit traditional hosting, so we created a virtualized layer where we didn’t have to engage the data center operations.” Flint described how the new Pagoda Box abstraction layer scratched its own itch. “We didn’t have to engage data center operations, and we didn’t have to engage a hosting company to deploy code on these servers.”

The development team saw its wider potential. Flint said the team “realized this would be useful to others as well and so we engaged in what was the start of V1 Pagoda Box.”

When Pagoda Box built its PaaS, it rented cloud infrastructure. While the new service was a success, overcoming long-standing obstacles blocking its customers, it wasn’t all good news. The Pagoda Box team found plenty of new obstacles in the modern cloud world. Flint said, “We built on some third parties and some technologies and that did not prove to be as stable as we initially thought. It wasn’t a good fit for the workloads that our clients would provide.”

The Pagoda Box team started work on a new and improved replacement. From their experience of V1, the developers decided Pagoda Box V2 needed bare metal hardware, a customized OS, and new virtual networking layer.

Pagoda Box V2 hardware

Building on top of the big Infrastructure as a Service (IaaS) providers did not provide the quality that Pagoda Box was looking for. The V2 PaaS is built on traditional physical machines, provided by SoftLayer’s bare metal servers. Pagoda Box now provides a bare metal option to V2 customers, along with its virtual service options.

Flint said, “We assumed we would be able to build a platform on top of an infrastructure — like Amazon, like Rackspace, like SoftLayer — and that portion of launching virtual servers is taken care of.” But relying on IaaS providers was not delivering enough. “We had to build all of it. We had to start from the ground up. And that’s essentially what V2 is. We built IaaS — our own private cloud.”

Pagoda Box V2 networking

Pagoda Box V2 contains a new virtual networking layer, giving every Pagoda Box application a private network. Flint said this completely virtualized network namespacing, which means “when a client launches an app, that app can live in its own isolated network.”

The V2 virtual network has given Pagoda Box greater resilience, Flint explained. “On V1 we didn’t have the virtual network. It was difficult for us to balance public IPs back and forth. On V2 we are able to leverage all the internal services during a partial outage.” The network also helps application deployment. “We couldn’t get MondoDB on V1,” continued Flint. “On V2, it just drops into place.”

Pagoda Box V2 OS

The Pagoda Box V2 host OS is an illumos distribution — not Linux. Flint explained why. “One of the biggest issues we’ve run into is running a MySQL instance that gets hot and starts throttling the disk. The Linux operating system doesn’t have any great way of throttling disk IO for one customer, to let the next customer use the disk.”

The ZFS file system provides disk IO throttling based off of priorities. One customer with a low priority can hammer the disk without blocking another customer with a higher priority. The first customer may create a queue of a thousand disk operations, but the second customer’s operations go straight to the front of the line.

Pagoda Box V2 user services

Flint believes “the services your app depends on, such as databases, caching components, queuing systems — those need to live inside of the infrastructure that hosts your application code.” Flint said “one of the reasons so many people use Amazon Web Services as their infrastructure when they build a platform is because there are so many hosted services that run on AWS.”

Pagoda Box did not want to build a service that deploys a customer’s code and connects it to a local network of shared services. There isn’t even a remote network of shared services for Pagoda Box customers — the company does not support an ecosystem of third-party providers like Heroku does with its add-on provider program.

Pagoda Box deploys a complete set of components into an application’s virtual network. The customer pushes code to Pagoda Box, the provisioning system inspects the application, and deploys dependencies like Redis, Riak, or PostgreSQL.

Building a PaaS from the ground up

Flint’s team wanted to solve the technical problems of the Pagoda Box V1 PaaS. “We were not able to accomplish that on top of AWS, or on top of Rackspace, or even on top of SoftLayer’s Virtual Compute Layer. We ended up building from the ground up. We ended up building on bare metal, provisioning our custom operating system, and having a custom provisioning layer with virtual networks.”

V2 has cost the company years of work, but Flint said that’s OK. “We’ve always been about longevity over short-term payouts, and that’s why it’s taken us so long.”

V2 is currently being rolled out to Pagoda Box customers. The full public release of V2 will happen in 2015.