Tech & Work

User testing: Setting up the test

Pick up some tips on how to properly set up a user test.

By Seth Gordon

It takes only five users to uncover 80 percent of high-level usability problems, according to Jakob Nielsen. But what if the five you've invited don't all show up for the evaluation? Because a typical turnout for evaluations is about one no-show per six respondents, it's a good idea to have a sixth person who can be ready if an extra respondent is needed.

Test respondents should approximate the intended users of the site (representative respondents). Unless you're evaluating a niche-oriented site (online brokerages, sports sites, and so on), bring in people of various levels of Internet/computer fluency, different age groups, and different genders. These people all think differently and provide unique insights.

Temp agencies and marketing research firms can often handle the respondent recruiting. Just keep in mind that people coming from recruiting agencies all have similar skews: they are looking for supplemental work. Just give the recruiting company a bit of notice and a screener (questions to ask potential respondents to ensure they fit the sample criteria).

Keep It Casual
Keep the setting of the evaluation informal and comfortable. When performing discount usability, consider using a small conference room with a "typical" PC or Mac, a little decor (so the room doesn't look sterile), and a work area. A bit of background distraction is actually a plus, because it creates a testing scene that's closer to reality. Nobody works in complete silence. If conference rooms in your office are in short supply, reserve one in advance. Nothing is worse than scrambling for space the day of the evaluation because you forgot to reserve the room. If your client wants to observe the test, make sure to rent a facility that has an observation room behind one-way glass or a closed-circuit TV, which won't distract the participants.

Regardless of where the evaluation is held, prep and inspect the room the day prior to the test. You should make sure that all the required supplies, such as computers, software, and chairs are there, and check that it all works properly. You should also have some snacks and beverages on hand for the testers. Once everything is in order, post a highly visible sign stating that the room will be in use for a usability evaluation.

What Are You Going to Ask?
The moderator's guide is an informal list of questions, scenarios, and key points that you may explore during the evaluation. Here are a few high-level issues to explore:

  • Describe the first items you notice on the page.
  • Identify which elements on a page are actionable/clickable.
  • What do you expect to find behind this link?
  • Please describe your experience when trying to complete x task.
The guide will be influenced by the broad goals defined prior to the test, but the specifics of the evaluation are really determined by the respondent's actions. It's important to remain flexible and be willing to deviate from the guide when necessary. If the respondent is supplying useful information that wasn't anticipated in the guide, don't be shy about exploring that new route.

Writing some of these ideas down is a useful activity for the team because it gets them thinking about the specifics of the site. As moderators gain experience and confidence, they will be able to run the evaluation without the guide. But until they have the process down pat, keep a guide handy so they can refer to it.

Note: The term client is used in this article to describe the entity that owns the site being tested. It can describe an internal team or a paying customer.

Seth Gordon is a frequent contributor to CNET In his spare time, Seth enjoys touring national parks and restoring mechanical antiques.

Editor's Picks

Free Newsletters, In your Inbox