CXO

User testing: Conducting the evaluation

Learn what you should tell respondents before the evalutation, and get some tips on serving as the test's moderator.

By Seth Gordon
2/15/00

If you have selected the respondents properly, you'll have a mix of individuals with varied skills and experience for your evaluation. Give them an overview of what a usability evaluation is, how it works, and what they can do to help you get the best information. If your site is not public yet, or if you're making major upgrades, have them sign nondisclosure forms. Here are a few things that I often tell respondents before starting an evaluation:

  • Be honest in your responses. I didn't build the site you are evaluating, so you won't hurt my feelings with any criticism.
  • It's not you who is being evaluated, but the site. We'll encounter some problems, but if the site was perfect, we wouldn't need to test it.
  • As you move through the site, think aloud: verbalize what you are doing and why you are doing it.
  • Don't do things any differently than you would if you were at home or at work. (Respondents often try too hard to complete tasks to appease the moderator. For example, make sure they don't spend more time reading instructions than they normally would.)

The Moderator's Job
Responses will be more insightful when the evaluation is run like a conversation rather than an interview. Don't bombard the respondent with rapid-fire questions. Weave multiple questions together to create real-life scenarios. Here are a few tips to keep in mind:

  • When a respondent is frustrated or confused, don't belabor the point. The moderator only needs to see if the respondent could complete the task under "normal" conditions in a reasonable amount of time.
  • When there is a problem so prominent that it will hang up every user, the moderator doesn't need to burn time and have every respondent address the same concern.
  • When respondents identify a problem, make sure you clearly document it. Also, ask them how they would fix it. (And if they give you a really good answer, hire them on the spot!)

Be Observant
Don't lead respondents. It's about their opinions, not yours. Don't say things such as "This is a great way to find products, isn't it?" If you think they're trying to make a point but can't quite articulate it, paraphrase what you think they are saying and ask if that's correct. For instance, you could phrase your question like this: "Just to make sure I understand you correctly, I think I hear you saying it was difficult to register for the site because it asked for too much information? Is that right?"

Look for body language and facial expressions. It's possible to observe emotions such as confusion, frustration, satisfaction, or surprise. Nonverbal communication is often more revealing than spoken responses.

It's hard to moderate and collect feedback at the same time. Have an assistant help take notes and observe. Make sure he or she captures quotes from the respondent, because that makes the final report more convincing. This is also a great way to expose other members of your team to the inner workings of a usability evaluation. Audio and video are good for archival purposes, but they increase cost and are rarely watched. If video was shot, it's useful to create a highlights reel.

Seth Gordon is a frequent contributor to CNET Builder.com. In his spare time, Seth enjoys touring national parks and restoring mechanical antiques.

Editor's Picks