Continuous integration promises lofty benefits, automating much of the dev process and speeding up time-to-market. But as Nick Hardiman explains, it's not without its pain points.
Modern cloud developers pack a CI (continuous integration) server in their software toolbox. CI fans say CI turns software development into a colourful agile world of unicorns and rainbows. Where did CI servers come from and what are their pros and cons?
A development team in an enterprise spends its time planning, coding, and testing software. Back in the old millennium, when the grey world of software development required manual effort from many specialists, the development lifecycle was long, expensive, and prone to failure.
The target for a modern development team is to spend only a week or two planning, coding, and testing a deliverable piece of software. A CI server automates much of the process, developers repeat the delivery process in endless iterations, and that's how big projects get delivered. It's a way of working that has been around longer than you may think.
Talking the Agile talk
CI started with a theory. Twenty years ago, new software methodologies like XP (eXtreme Programming) promised to solve the problem of big IT project failures by splitting the work into many small deliverable pieces. Projects would regularly deliver lots of small, complete working software components instead of producing only one big system at the end.
Software delivery would be rapid and regular, quality would be higher, the cost would plummet, everyone would be happy, and software development would be just really great. In 2001 the Agile Manifesto for software development was published. Marketing people stuck this catchy name on all their products, and software developers scratched their heads.
Of course, talking the talk is not walking the walk. Making vast improvements in software development required huge shifts from manual to automatic processes. Many attempts to make this Agile system work led to many new approaches. And the CI server was born.
Walking the CI Server walk
The XP people produced CruiseControl to help automate Java builds. It was the first of the CI servers.
A CI Server automates several steps of the software development process. It can't help with business requirements or design, but it can take care of environment builds, software tests, and deployment. Since CI servers can take on this software heavy lifting, developers love them. Every Linux distribution includes a packaged-up version of the Jenkins CI server, started by Kohsuke Kawaguchi.
Popular CI systems can interface with a wide range of developer tools. One CI project can't possibly manage to connect its server to every language, SCM, and IaaS API - there are just too many. A CI project gets around this problem by providing a plug-in system and encouraging its community to code new plug-ins and maintain old ones.
Who is CI aimed at?
Because it's automated, a CI server helps speed up time-to-market in an IT-driven organization. Some IT-driven producers, like Cisco, Microsoft, and Red Hat, work in the high-tech sector - they make their money by selling faster hardware, writing cheaper software, or providing better technical services. But an IT-driven organization is one that gains a competitive advantage from internal use of technology, not necessarily a company that sells technology. You don't have to be a high-tech producer to be a high-tech user.
IT-driven organizations are businesses where IT is critical to operation. Banks, package delivery companies, and retailers are all IT-driven. Making their IT pipelines better means making their internal communications, market strategy, and cost of production better. And they are the kind of customers every cloud service provider would love to get onboard.
How CentOS uses CI
The CentOS Project team uses CI to run thousands of tests a day. The CentOS Linux distribution is popular with the enterprise and in cloud computing. CentOS is the platform many private cloud systems run on, and it is the OS running in virtual machines in many public clouds. Not only is it IT-driven, it couldn't be more central in the high-tech sector.
The CentOS Project team knows that CentOS works because a CI server tells them it works. Tests are automatically run every night, after every code release, and after every test update. You can see the CentOS CI server doing its work at http://ci.dev.centos.org/. That's right; it's another Jenkins server (Figure A).
Dealing with a complex application like MySQL takes up a lot of time for many developers. For the CentOS distro -- with eight and a half thousand pieces of software -- MySQL is just another unit. The CentOS crew use tests provided by the MySQL developers to test MySQL works, and they create their own tests to make sure MySQL works in the CentOS distro. Karanbir Singh, CentOS project lead, said it is the CentOS project's responsibility to check "MySQL links to the right version of SSL, which links to the right version of zlib, which links to the right version of glibc."
If you work on a Linux distro project, watch Singh describe their CI in detail. And steal his good ideas! Hey, as Pablo Picasso said, "Good artists copy; great artists steal." (Or was that Steve Jobs?)
Introducing CI is painful
AirBnB decided to use the Solano CI server last year. Lou Cosak, software engineer at AirBnB, said, "We've gutted our continuous integration stack and reduced our build times from 1 hour+ to about 6 minutes, while simultaneously handling many times the number of builds. "That's a great result, but check out how much work went into getting those numbers.
Thanks to XP, Agile, and other software methodologies, CI servers are now common in the cloud computing world. But they are not universal, despite their benefits. Laying a new CI foundation for software development is a painful process for an enterprise to go through.
- Software development tools, from Stone Age to cloud age
- Linux, the overweight king of cloud: Will this change anytime soon?
- Take in the big cloud picture with this comprehensive text