SaaS: Keep complexity and short timelines from affecting quality

This blog discusses some strategies for building high quality SaaS products.

This is the last part of a series on SaaS. Here are the first parts:

It's widely known that most software companies struggle to consistently maintain quality. You may have even heard the jokes about why Microsoft could never manufacture cars. There are many reasons why quality is so difficult to maintain in the software industry-the industry is still maturing, software is often particularly complex and the quintessential quality of trade off due to tight deadlines are among them. And in the SaaS world, that complexity is even higher due to scale and performance requirements, support for third-party developers, and so on.

SaaS does, however, give you some additional levers that can help with quality. There's an interesting trend emerging where some SaaS companies are moving away from the notion of a QA engineer altogether. This blog will discuss some strategies for building high quality SaaS products, and also a point of view on this trend.

Commitment and culture

Building high quality software needs organizational commitment from everyone on the team starting with the engineering and product leaders. When the decision comes down to whether you need a little more time to ensure product quality or meeting a deadline; which way do most decisions go? Judgment is of course required in all cases, but quality needs a voice and must be part of the discussion alongside other business considerations. Team members must be empowered to raise their hand and make a case where they believe a product isn't shippable. This requires a culture that encourages and rewards those hand-raisers.

Testing strategy

We've learned that small, incremental releases have the highest quality because they contain the least amount of risky change.  So why don't we release every day?  Every minute?  The usual response to this question is that because there is some fixed cost or duration to release, and that fixed cost is dominated by any manual regression testing or other validation of the final build.

This is why automation is king. The pyramid below shows the types of testing and coverage it affords. Unit tests, integration and UI testing give more coverage in that order, and should all be automated.  Automated testing delivers the most reliable results and the broadest coverage in the fastest time. UX testing is done with the end-user hat on to test product usability. Ideally, it's the only type of manual testing. This approach requires a significant investment in testing infrastructure, which goes back to why organizational commitment comes first.

SaaS levers

There's a high price to pay for a leaked defect for both shrink-wrapped software and SaaS. Patches are expensive and that the defect ever got out damages the brand. For SaaS products however, there are some additional levers available to mitigate this risk with a controlled release of a feature or app.

Make sure that you can control how many customers get a new version of the software based on that risk. As an example, assume feature X is rolled out only to .5% of your customer base and monitored closely before gradually increasing the dial and rolling it out to more customers. This lets you uncover issues before it hits the masses.

Also make sure that you have instantaneous rollback capability if you uncover a serious issue. This does not replace the criticality of getting it right the first time and building quality in-it should be viewed as an insurance policy and not the way to test the product.

Quality trends

There are some SaaS companies that are moving away quality engineering roles. The expectation is that engineers will do both development and testing. Ask a QA engineer for their opinion on that structure and you'll hear that this will never work. I believe this trend will gain steam in the future but the broader industry isn't ready for it.

If you are evaluating moving to this model for your company, proceed with caution. Think carefully about where your organization is and the transformation required for this model to work. Even if you are not ready to move to this model entirely, having developers go further and further up the testing triangle is a good thing and will help you get more coverage.

Ideally there is an internal recognition within the development team that everybody owns quality.  At Lithium we call it being "Addicted to Done". It's not Done until it's tested, and usually the fastest way to getting to Done is to test it yourself (or write the automated test, see above).

As the industry continues to mature, we can only expect to face increasing complexity over shorter timelines-two major threats to product quality.  But with the right commitment, culture, and testing strategy, your SaaS engineering team can meet those pressures head on without those old familiar sacrifices to product quality keeping you up at night.

As SVP of Engineering and Operations, Sunil Rajasekar oversees core development and delivery of Lithium's enterprise social customer platform.