The term DevOps was first used in a 2008 Agile conference in Toronto. DevOps methodology has been on the rise in organizations ever since. Many companies have seen big gains in time to market for new applications with DevOps, and it has enabled them to keep pace with business competition. Among the much-heralded use cases are:

Facebook, which has accelerated its biweekly apps update cycles to constant rapid refreshes of mobile apps that happen daily.

Sony Pictures Entertainment’s Digital Media Group, which went from a manual process of software development and implementation that took months to a continuous delivery model for new software updates that takes minutes.

Adobe, which has moved away from large, semiannual software releases to continuous software updates and has seen a 60% improvement in its ability to meet new app demands.

The inherent risk, of course, is that DevOps and Agile software principles focus on one main metric for measuring software efficiency: the frequent delivery of working software.

But manual and automated testing methods do not always work as well as they should when it comes to QA-ing new changes in the DevOps environment. I have met end users who conceded that it’s “alright” to live with broken links and other artifacts in websites that don’t work, as long as they can get a website right away.

Unfortunately, these same internal tolerance levels don’t work for outside customers who visit your website, expect to make an online purchase, and have difficulty doing so–or for visitors who visit your site but come away with a less-than-favorable impression of your brand because the site doesn’t work well.

One way companies using DevOps for website development can plug in first-level QA to improve quality is by using a tool to check some of the more superficial elements of websites, such as whether links still work after a new software release is installed.

“Etailers and even corporate websites where many departments are constantly changing content, can benefit from a QA check like this,” said John Trembley, vice president of marketing at Atmosera platform. “When there are multiple functions producing new content for a website, there is always a risk that a link gets ‘stepped on’ or fails to function properly.”

Scott Harvey, Atmosera’s vice president of engineering, explained how the tool works:

“Whenever Web content is changed, a ticket is issued,” he said. “The content is placed into a staging bucket and checked by the release management software. The software does a sanity check on how the links are working and on the timings on the website. It then issues either a pass or a fail to the developer. If the new software release passes, an email is issued that tells the developer, or whoever is managing the new release, that it is okay to place the new release in production.”

Tools like this don’t yet address unit testing, systems testing, or integration testing, but they are a step forward in DevOps quality control.

Quick glossary: DevOps

The ability to rapidly develop, deploy, and integrate new software and features is essential to the overall success of many organizations. DevOps is the solution. This glossary of 20 DevOps-related terms will provide you with a working vocabulary. Free for Tech Pro Research subscribers.

This brings us to where CIOs and application managers are today with DevOps.

DevOps is working and returning value–but if bugs and fix cycles are happening as fast as new software is being developed and released, these upfront gains in time to market can be lost.

What best-practice steps can CIOs and application managers take now?

If the DevOps is for internal development only, there isn’t much risk from error introduction, and your IT staff and users are happy with imperfect but rapid outcomes, you may not need to take a lot of special steps. But you should still continue to investigate testing automation tools and techniques that can make new DevOps software releases less error prone.

If DevOps is being used for outside websites and apps that customers see, the software released should be zero-defect. If this means using more standard QA methods that unit test, system test, integration test, etc., until more complete toolsets come along, so be it.

Finally, you can expand the metrics you use to measure DevOps effectiveness. Time to market is one gauge, but frequency of repair is another.