Android activist Mark Murphy shares app testing best practices

William Francis interviews Android community activist and author Mark Murphy, who offers developers app testing tips that won't empty their wallets.

My interview with Mark Murphy

Francis: When you wrote the first edition of The Busy Coder's Guide to Android Development, the Android operating system was at version 1.5, and the only Android phone actually available to consumers was the G1. A lot has changed since then. In your opinion has testing and supporting Android become significantly more complex in the wake of its success? Murphy: It is certainly more complex than it was when we had just one phone.

Most of that complexity comes from the UI and ensuring it looks good on various screen sizes and densities. For applications that can stick with the standard widget framework, this is usually manageable. For games and other applications that do all their own UI using 2D and 3D graphics, I have to imagine that testing is a serious pain in various body parts.

Some of that complexity comes from device bugs, the truest form of "fragmentation" facing Android. Most applications will not encounter any of these. Some applications will. Whether you find out about them ahead of time or by user experience, and how easily you can work around them, depends a bit on how you are doing your testing.

A bit of this complexity comes from the multiple OS versions in use, but this is usually something that testing with the emulator can uncover, for most apps.

Francis: Along those same lines, do you see the testing burden on developers increasing or decreasing any time soon? Murphy: For those who need to write a lot of code in Java (or C/C++), the testing burden will increase, but a bit more gently, barring major growth in some new form factor (e.g., wearables, such as those from WIMM Labs).

However, I anticipate a lot of growth in development options that minimize how much of that code is needed. Some of this will be aimed mostly at cross-platform stuff (e.g., PhoneGap/Apache Callback/whatever it winds up being called). Some of this will be aimed at just simplifying normal development (e.g., whole-app frameworks, akin to Eclipse RCP for the desktop or Cairngorm for Flex/AIR). Some will be outgrowths of today's generation of app generators. For developers using these solutions, testing will decrease, because much of the complexity will be handled by the platform they have chosen atop of Android. Developers will be focused on their own business logic and will rely upon the platform to ensure compatibility across device profiles and OS versions.

Francis: As an Android developer myself, I find I rely heavily on the emulator. However, as we begin to see greater variations in device capabilities, tablets, Google TV, etc., I have started to question how much testing I should be doing on the emulator, and, what if any baseline of emulator images I should be bouncing my code off. Could you share with me and TechRepublic readers what you think is the ideal amount of testing to do on the emulator vs. actual hardware? Murphy: Well, I have the luxury of having purchased a fair bit of Android development hardware, so I will tend to test more on devices and less on the emulator, personally.

The emulator's historical strength was that it was free, and that is certainly an important feature today for many developers. However, rooted devices and things like Romain Guy's ViewServer have made on-device testing more compelling for things that used to only be the province of the emulator, like using Hierarchy View to diagnose layout issues.

IMHO, use hardware where possible, for speed and fidelity. Use the emulator for filling in any gaps in your available device profiles. Note that there are ways of getting access to hardware beyond simply shelling out money to buy devices outright, as we will get into later.

Francis: Do you have any insights for configuring some representative emulator images considering the current distribution of the various OS versions and display resolutions? Murphy: I would focus less on "the current distribution of the various OS versions and display resolutions" and much more on "the device profiles that you are targeting". Now, the decision as to what device profiles you might target may be based in part on the OS versions and display resolutions that are out there, but there will undoubtedly be other criteria as well, such as required APIs, practicality of your UI in various form factors, particular business requirements (e.g., consulting clients, OEM bundling deals), etc.

While Google would love it if all apps were available for all form factors and extant OS versions, developers are the ones that have to make the decision as to what to test on. The more form factors and OS versions you support, the bigger the development and maintenance cost, but the wider your potential audience is.

In terms of configuring the emulator images themselves, I recommend that the emulators reflect hardware (i.e., use the "Google APIs" emulator images, since most hardware has Google Maps). Beyond that, it's largely mechanical, choosing a density and resolution, which in turn give you a screen size bucket.

However, for devices that have not passed the Compatibility Test Suite (i.e., those that do not legitimately have the Android Market, such as the Kindle Fire), unless the device manufacturer ships their own emulator image, testing on the emulator is likely to be less fruitful.

Francis: I've noticed lately a lot of "mobile phone petting zoos" popping up. For my readers who aren't familiar with these organizations and what they do, I believe the best way to think of them are as data centers for phones instead of traditional blade servers. Companies like BSQUARE, DeviceAnywhere, and Perfecto Mobile offer access to actual racks of mobile devices for testing via a web-based interface. I'm curious what your thoughts are on this type of testing scenario, and how much value you believe it can add to an app roll out? Murphy: Personally, I think services like Keynote/DeviceAnywhere and Perfecto Mobile are much better suited for fixing bugs. If you find out about crashes in the field -- whether from the Android Market or ACRA or whatever -- but they are occurring on devices you do not have easy access to, online access services are crucial for finding and fixing the problem.

For testing, I am more intrigued by services like LessPainful and BitBar's TestDroid Cloud, where you can upload your app and test scripts once and have them exercised automatically on a device farm.

Using something like this as part of your final test process before release can catch device-specific issues without a lot of engineering time manually uploading and running tests to a bunch of remote devices. A peer-to-peer edition of something like this might be compelling as well -- I've toyed with trying to set up something along those lines.

Francis: One of the things that I love about Android development and the mobile app space in general is that you don't have to be a multimillion dollar organization to produce and successfully market an app. I find Android ideal for budding entrepreneurs because the free tools and open nature make the cost of entry very low. However, when it comes to testing apps on real hardware, being a one-man show can severely restrict the number and type of devices a developer can afford to purchase and have on hand. Even when you do mobile development for a large company like I do, there are limits to how many different "toys" my boss is willing to spring for. Can you offer Android developers any advice on choosing and purchasing test hardware on a budget? Murphy: While the aforementioned online access services can help, you certainly need some hardware in hand. I recommend developers have one device per targeted form factor per distinct market. So, for example, if you are only trying to create apps for phones on the Android Market, have one such phone. If you also want to do tablet apps for the Market, get a tablet. If you want to develop for the Kindle Fire, Nook Color, or other non-Market devices, you will want hardware for those as well.

Ideally, you would own such hardware, simply so that you use it routinely, for your own app and for other apps. Writing an app for a device you never touch is a bit risky, in that your guess as to how the device would be used may differ from reality. eBay and similar outlets for used equipment would be a fine place to start hunting for hardware.

Beyond that, turn to your local Android developer group or GTUG. It would be awesome if these groups would host periodic Android app testing equivalents of "speed dating", where people pair up, swap devices, test their apps, then switch off in 10-15 minutes.

Also, if you live near a major metropolitan area, watch for events like Google Developer Days that might host some form of testing labs.

Final thoughts

If you keep up with the news on the mobile landscape, you know the experts don't predict the Android train slowing down any time soon. As the success of Google's mobile platform continues, both the opportunity for and responsibility of app developers grows too.

Testing is a crucial piece of the success of your Android projects. Thanks to Mr. Murphy for taking time out of his schedule to share his views on this subject. I think his insights are the ideal balance of practical and philosophical. I hope TechRepublic readers find this interview as interesting and valuable as I did.


William J Francis began programming computers at age eleven. Specializing in embedded and mobile platforms, he has more than 20 years of professional software engineering under his belt, including a four year stint in the US Army's Military Intellige...