Ten years of IT transformation, from the launch of AWS to the growth of the container ecosystem today, has put new tools, new APIs, new techniques, and new algorithms in the hands of developers. The latest big shift to a developer-centric world is the rise of the Docker ecosystem.
Developers heart Docker
If there's one thing developers like, it's coding. All those infrastructure headaches — and that includes everything from defining networks to dealing with infrastructure operators — are annoying overhead. Docker, a Linux container system, is the pill that makes infrastructure pain go away (if you work in ops, you may now have steam coming out your ears).
Developers should love Docker. It simplifies and speeds up IT. For instance, Docker lets developers treat infrastructure as code. Their application, neatly packaged up in a Docker container, is easily copied around the infrastructure, from dev to test and test to production. With the addition of third-party management tools like Kubernetes, Docker infrastructure can be described as a YAML file — home turf for devs. Nice and tidy.
Docker is not all that for distributed applications
Docker is sold as the platform to help developers create distributed applications, but Docker has not been so hot in some areas.
Container Solutions is an Amsterdam-based consultancy based around Docker. Adrian Mouat is chief scientist at Container Solutions, and author of "Using Docker." I took the opportunity to talk to Mouat at QCon London 2015, a developer conference. Mouat gave an example of a problem. "You can have multiple containers on a host, but moving a container from one host to another is an issue — especially getting its data to travel with it. Getting Docker, out of the box, to communicate with different hosts is a lot more difficult."
The Docker ecosystem has grown up to connect the distributed application dots in areas like these.
- Motion: Stateless containers can be destroyed, but stateful containers need to be moved around hosts.
- Networking: The ability to talk to other containers elsewhere in the network is essential for a distributed application.
- Orchestration: Getting many containers to work together requires a management system.
- Extension management: How to add many optional extras to Docker.
- Service discovery: A new containerized application needs an authoritative source to tell it where to find services.
The Docker ecosystem fills in the gaps
Docker was a good fit with the cloud dev, but it was a single box solution that didn't come with the large-scale components that an enterprise needed. Organizations have been rolling their own solutions to fill in the gaps, creating a whole Docker ecosystem in the process.
The Docker people decided to fill in these gaps themselves to create a more appealing platform. Jessica Frazelle, software engineer at Docker, described the Docker approach to solving these problems as "batteries included, but replaceable." That means Docker comes with a full set of default components, but they can be easily replaced with third-party products.
"Docker out of the box will communicate with different containers on the same host. Containers on separate hosts, it becomes a lot more difficult," Mouat explained. Docker bought SocketPlane to provide its "batteries included" solution to this problem.
Mouat explained how Weave is one of the "replaceable batteries" solutions. "Weave is an SDN (Software Defined Network). That lets you set up a network - it's all done in software — which makes it much easier to talk across hosts, without doing any of the hard wiring work."
Mouat said, "You can have multiple containers on a host, but moving a container from one host to another is an issue — especially if you want the data to travel with it. Flocker is a solution to that." Flocker, from ClusterHQ, aims to solve the problem of managing stateful containers. For instance, an organization may run a database in a container, and they may not be happy to just destroy that container and start again.
Mouat talked about the challenges of managing a large fleet of containers. "If you've got a large network of containers, how do you manage them all? How do you decide which job goes in which container, [and] how many containers you need? What do you do about monitoring, job control, and scaling?"
Simply swapping one component for another is limited. Docker does not come with an extension model for adding many extra components. A developer could not use both Weave and Flocker because each one wants to wrap itself around the Docker binary.
Mouat described how ClusterHQ and others have come up with a solution called Powerstrip. "Powerstrip is a way to plug extensions into Docker. It's essentially a hack that relies on the fact that the Docker daemon talks over HTTP. You can intercept that call, do your own bit, and pass the connection onto Docker." Mouat continued, "With Powerstrip between the Docker client and daemon, you call Powerstrip first, which will then call Weave and then Flocker, and then it's passed onto the Docker daemon."
A new application, fired up in a new container, needs to know where to find services. Containers require some kind of central authority with configuration answers. "The etcd solution is you go to this distributed key-value store, and you ask it — where can I find my database," said Mouat. "ZooKeeper is similar — that's the one that is used by Hadoop."
These gaps have been filled before
These problems have been solved before, and some of these solutions can be extended to include Docker. JFrog Artifactory is a binary repository manager that was recently extended to manage Docker binaries. Cloudsoft added Clocker to Apache Brooklyn to make a Docker cloud system. Ansible and Puppet can manage Docker.
The Docker ecosystem is young, and it is incomplete. But it's surprising how much whitespace has already been filled in.
- 10 things you should know about Docker
- Docker and Mesos: Like peanut butter and jelly
- Weave connects containers and cures a major Docker headache
- SocketPlane strives to alleviate Docker networking challenges
- Containers: The pros and the cons of these VM alternatives
- What Docker means for enterprises (Tech Pro Research)
Disclaimer: TechRepublic, ZDNet, and Tech Pro Research are CBS Interactive properties.
Nick Hardiman builds and maintains the infrastructure required to run Internet services. Nick deals with the lower layers of the Internet - the machines, networks, operating systems, and applications. Nick's job stops there, and he hands over to the designers and developers who build the top layer that customers use.