A former US Secretary of Defense once said:
“There are known unknowns; that is to say there are things that, we now know we don’t know. But there are also unknown unknowns - there are things we do not know we don’t know.”
Nowhere is this statement truer than your app in the hands of a customer. Unit testing will not take into account all the conditions under which the customer will run your app or the different things a customer will try. There will be network, multi-threading, and memory management issues that come to light only when the app is used “in the wild.” You must assume a customer will, sooner or later, and probably sooner, find your app performing poorly or worse, make it crash.
If you are a developer on a limited budget, who wants to submit an app that will be highly rated to the Apple store, here are a few strategies you can try.
Test Flight provides a free testing platform for iOS devices. After you sign up you can create a test team, upload builds, and invite testers. They also have an SDK. Once added to your app you get remote logging, crash reporting, and checkpoints, that record when the tester has achieved a specific goal, along with some basic analytics.
So how do you get testers? Test Flight suggests you advertise on Twitter. This may work if you are patient, but you want as many testers as possible, hopefully at the same time, so they will stress the network connection to your backend server. I suggest you advertise more widely on Facebook, Google+, and other sites and offer the chance to win a prize like a new iPod or iPhone. With this offer you will be able to do the following:
- Have each tester sign up at your website. Describe your app and make them comfortable with giving you their iOS device’s UDID. Point out the favorable odds.
- Assign an open and close date for testing and send e-mails to all testers to let them know when the testing period has started and ended.
- Ask them to fill out a survey after the testing period has ended.
- Disable the app when the testing period has ended.
- Randomly pick the winner and post on your website.
- Let them know when the app has been released.
Make sure the steps are spelled out up front and include a legal disclaimer. And when you are done, congratulations, you tested your app and completed a social media campaign.
Test Flight is the only test platform I’ve used but there are others you can try. A notable alternative to Test Flight is Apphance. It also has an SDK (most if not all third-party tools ask you to include their library) that adds remote logging and crash recording to your app, for production use as well as for testing. Unlike Test Flight it supports Android and other mobile OS’s.
Apphance’s parent company UTest has its own community of professional testers. The testers are from around the world and for a fee will test your app. This video explains how it works. This will not be an option if you are on a limited budget, but Apphance has waived all pricing plans until the end of 2012.
There are other vendors providing similar services:
Mobile analytics should be part of your app beginning with testing. You need know right away, among other things, how often your app is used, how and where it is used, and how long the user stays in your app. Because it’s such an important topic, analytics will have its own tutorial at a later date.
Like testing platforms there are many third-party vendors to choose from. Two notable vendors are Google Analytics and Flurry. These two services are both free, except that Google has an account limit of ten million hits per month.
After your app is available on the Apple store you can get crash reports submitted by users through iTunes connect. But most customers won’t bother; it’s easier for them to give a low rating to your app and a low rating is difficult to repair.