It's hard to dispute the importance of QA, yet some organizations still leave this step out of the software testing process due to lack of resources. If you're looking for some QA help, Justin James outlines why you should evaluate uTest as a potential provider.
Most developers want a QA process, but not every organization is able to provide a good one. A new company named uTest, which opened its doors to the public on August 19, 2008, has put together a novel approach to this problem. Last week, I had the chance to talk to Doron Reuveni, the CEO and founder of uTest, and learn more about the company's services.
Doron describes uTest as a cloud testing marketplace. In practice, this means that the company farms out the QA department and then makes it available on-demand to organizations. To accomplish this goal, uTest recruits testers across a wide variety of demographic profiles. The testers range from highly experienced industry pros to "man on the street" typical users. Additionally, the testers represent more than 130 countries, giving you the ability to test applications across many languages and in many cultures. Don't think that this is all code for "inexpensive Asian workers" who do not know English very well; as of July 2008, 27% of uTest's testers are in the United States and another 9% are in the United Kingdom. All testers sign NDAs to protect the clients. (One client even changes logos and other information in the test software to not allow testers to see the entire application in order to preserve its pre-release secrecy.)
The second half of the equation is the clients. Clients create a budget for testing (the minimum balance for a new account is $2,000). That budget lasts for as many testing cycles as needed and can be refilled. When a customer needs testing to be performed, they define: a project, what tests they need to be done, and a profile of the testers they want. These tester profiles can be as broad as "anyone" to some very narrow specifications, including computer platform, Web browser of choice, years of testing experience, country, and so on.
Test cycleTesters have the opportunity to participate in the test cycle, and the client gets the chance to accept or reject each tester. When the test cycle begins, the testers submit bug reports through uTest's system. The client receives the bug reports, and then they perform an independent verification of the bug. Once a bug is verified (or not), the client marks it as Approved or Rejected, and the tester that found the bug gets credit for it (testers get paid every two weeks). The cost of a bug depends on many factors, including the product's maturity (bugs in mature products are harder to find and, therefore, cost more), the tester's "grade" (each tester builds up a "grade" over time based on a variety of inputs), and more. (An important note: While most of uTest's clients are testing Web applications, they have the ability to test desktop and mobile applications as well.)
High-quality core test teams
One of the things that I was concerned about was the ability to build a high-quality long-term team. Over the years, I've found that while having some fresh faces on the QA for a product is always good (especially for initial releases of new versions and usability testing), it is also good to have some experienced hands on the software too.
"Experienced" does not always mean being in the industry for 10 years; it can mean that someone has been involved with a specific project for a period of time. Experienced testers already are familiar with the application (of particular usefulness for sophisticated applications or applications that are not quite user friendly). They are also familiar with the types of tests you are going to run. For instance, if you are doing regression testing, they may be well aware of undocumented behaviors that should not change.
Doron let me know that you can request particular testers as part of the tester profile that you specify for any given test cycle. Conversely, if you feel that a tester did a bad job, you can request that a tester not be allowed to participate. Using these features, a company can put together a core test team that they are happy with and then do a big "broadcast test cycle" for releases that require wider testing.
Benefits and potential drawbacks
From what I have seen and heard, uTest offers the following real advantages:
- There is access to a wide variety of platforms to test on.
- Because the testing is mostly done by part-time workers and some of it is done on other continents, it is possible for a domestic development team to release code to testing in the evening and have results in the morning. This can significantly reduce the time needed per cycle.
- There is on-demand access to a wide variety of testers.
- The payment model is based on results not effort or time of the QA team, which can be a significant savings.
- The tester filtering process seems pretty granular, giving you a better chance of getting the testers you need and want in comparison to in-house testers.
- The on-demand nature of uTest is well suited to: an agile project, companies without a dedicated QA team, and companies with a dedicated QA team but who have "spikes" or the occasional need for a larger team when they want to supplement their existing QA teams.
On the other hand, there are a number of factors to consider that may be potential drawbacks.
- Some organizations will be uncomfortable with the idea of having people who they have not been vetted by them seeing their software (particularly pre-release software).Yes, there is the NDA in place, but not every organization feels that they are adequate protection.
- While uTest currently integrates with a number of bug tracking systems, the list is not very inclusive at this point.
- I am not sure what guarantees uTest can provide that the testing will be done by the client's deadline.
- uTest will not be able to deliver the "show me what you mean" experience that you can have with in-house testers. It is also not going to replace traditional listening labs and other usability methods, although you may get some useful feedback regarding usability.
- Running a test cycle through uTest with a distributed team working at all hours may not provide the level of coordination that some development teams like.
Selling a service
When uTest's PR team initially contacted me, I was extremely skeptical. I thought, "Ho-hum, another vendor selling QA miracle pills." I've been pitched a lot of snake oil when it comes to QA, and I bet most of you have too. The only development pipedream that might get more play is "drag 'n drop your way to an application... no code required!" systems.
I asked Doron some tough questions, and I got reasonable answers. While I think that uTest is not for everyone or every organization, after giving Doron a fair hearing, I think it is safe to say that uTest seems to be providing a needed service at a potentially high level of quality at what could be an extremely attractive price.
What makes a huge difference is that uTest is selling a human service rather than a software solution. Many aspects of QA do not have a particularly high barrier to entry, depending on what kind of testing is being done. This means that many types of tests often require patience and attention to detail even more than formal training or experience. And, of course, when you need "typical user" testing, most organizations resort to hijacking half of the Marketing department at the last minute to click through the application really quick, which is hardly a proper QA test. uTest seems extremely well suited to provide these kinds of tests.
Check out uTest
So, if you're looking for additional QA help, you'll want to evaluate uTest as a potential provider. And if you're interested in scooping up some extra cash for part-time work, you may want to look into becoming one of uTest's testers.
J.JaDisclosure of Justin's industry affiliations: Justin James has a working arrangement with Microsoft to write an article for MSDN Magazine. He also has a contract with Spiceworks to write product buying guides.
----------------------------------------------------------------------------------------------------------------Get weekly development tips in your inbox Keep your developer skills sharp by signing up for TechRepublic's free Web Developer newsletter, delivered each Tuesday. Automatically subscribe today!