In late 2012 Capgemini, Sogeti, and HP released its 2012-2013 World Quality Report (PDF). The report is based on more than 1,500 telephone interviews with senior executives conducted in April and May of 2012. The survey sample included those in IT as well as business-related functions and spanned 25 countries. The report concluded that enterprises are “behind the curve” when it comes to testing mobile. Only 31% of the respondents admitted even bothering to test mobile.

There are a litany of reasons to explain the shortfall in mobile testing at the enterprise level, starting with its sheer complexity. Multiple devices, running on multiple platforms using bandwidth provided by multiple networks is a dramatic change in how enterprises manage computing.

“At the end of the day, mobile is not an isolated effort,” says Gal Tunik, senior project manager of Functional Testing and Mobile Delivery, Software at HP. “It must be integrated into the normal delivery process.”

Legacy processes play into the mix

There is a full lifecycle approach that Tunik has helped many companies set up over the years, so he says the question becomes: How do you deal with the specific problems of mobile, but still continue to support the full lifecycle approach? Most developers prefer to develop native apps and leverage the full capabilities of the underlying platform. That leads to a huge duplication of work because the assets developed for the underlying functionality are not being reused.

Skills and experience running behind

There can also be experience and skills gaps. Tunik says that many enterprises he speaks with are struggling to define the process they will use to plan their mobile strategies. He also hears that mobile is not being tested as much as it should be because of lack of expertise and the challenges of learning to deal with something new. Ultimately, the process of defining a mobile strategy can take a couple of years because it involves learning what needs to be done, what tools are required, and what new skills people need to acquire.

Indeed, the skills issue was illustrated in the 2012-2013 World Quality Report, with only 20% of North American organizations saying their internal testing resources were above average, and only 26% thought external testers were of above average quality.

Jason Weiss, president of Serious Apps, further refined the experience and skills assessment by drawing a parallel between mobile today and the early days of the web. He says he doubts the average CIO or CTO fully understands mobile yet. He allows there are visionaries who are doing amazing things, and that mobile is moving into its own, but he says it will still be another two years at least before it’s mature. In his previous role at PayFlex, he worked on a customer mobile application that was wildly successful at implementation. In that case he said the first hurdle was in the team recognizing they had no experience in mobile. That led them to hire Lextech to help them define their mobile strategy and that, he says, made all the difference in the success of the implementation. Recognizing limitations is critical.

“Mobile is specific enough and hard enough that it’s worth getting some experts involved. Just because you have Java developers in house, doesn’t mean you’re ready to develop Java on Android,” Weiss says.

Putting abilities and needs assessments in their place

For testing mobile applications, Weiss says that after coming to terms with your abilities the next step is to gain a deep understanding of what your requirements are. Know what the data is, how it is integrated with the enterprise, and what security level is needed. It’s important to know what is and what is not critical and to understand nonfunctional requirements almost as well as functional requirements.

Weiss also encourages making effective use of the 80/20 rule. For example, by testing the top 20% of device/OS combinations, you could typically cover 80% of the market. Likewise, by testing the top 20% of app features, you could cover 80% of actual usage, but he stresses there are exceptions.

In addition, Weiss counsels that by understanding the demographic in the mobile environment you can limit the amount of testing. He thinks it’s highly important to protect existing functionalities before worrying about new functionalities. His experience shows that people are far more forgiving about a new feature that doesn’t work the way they’d like, then they are about old features they’re used to using that get broken.

Tunik says it’s important to know the impact an application has on the core business of the organization, both from customer and employee perspectives. Then, he emphasizes the importance of testing real devices; otherwise, you can’t cover all the variables. He also says you have to use everything: emulators during the development process, real devices on computers, and then real devices on the cloud. Tunik encourages companies to consider outsourcing the management of the devices; if they don’t, they could find themselves in the device management business rather than the testing business.