At the recent Web Directions South UX conference in Melbourne, Lisa Herrod, the Principal Usability consultant at Scenario Seven offered advice on usability testing with her presentation — “User testing for the rest of Us”.

Testing is an important part of the Web design and development process, but can often be overlooked, because of the costs associated with it.

During the presentation, Herrod proposed some usability testing methods that are both efficient and cost-effective.

Usability Research

Usability research can be conducted using three methods: Expert review, Comparative review and User testing.

Expert Review

Expert Review, also known as a Heuristic Review, is an investigation at a high-level aimed to discover some broad usability issues. This technique does not involve users. Websites are assessed against a certain criteria and a checklist or matrix is used to document the findings.

Authors of some of the better-known usability heuristics are Jakob Nielsen and Denise Pierotti — 1990 and Xerox 1998 respectively.

Heuristics need to be assessed in categories. Herrod presented us with the following 11: homepage, visual design, coding, JavaScript, content, accessibility, feedback form, error handling, navigation, search interface, and search results.

When assessing the homepage of a website for instance, these are some examples of questions we should be asking:

|> Is the purpose of the website clear?
|> Is search available?
|> Does the Homepage offer different means of reaching content?

Comparative Review

Comparative review is an extension of the Expert review, as it evaluates a website against other websites. This technique compares websites with a similar purpose or certain qualities or websites that are different. For instance, we can asses booking systems on different types of websites, like travel, cinema etc.

When conducting these types of reviews, you need to:

|> Develop heuristics and put them into categories
|> Identify websites to be tested
|> Populate the matrix and compare the sites against the different categories
|> Write an overview of the findings to determine which site performed best in a category

User Testing

Some of the user testing methods available are listed below:

|> One-to-one: a test session with each individual user. The tester watches the user as they carry out tasks and takes notes. Any difficulties encountered are recorded and the user might also provide additional feedback. This is the preferred method, as it’s more intimate and can produce more information.
|> Focus group: discussions with users to see what they like and don’t like. While this can be effective for preliminary research, it should not be the only technique used. It is a lot harder to pay attention to everyone in a group and capture the information accurately.
|> Lab-based: the tester observes the user while they are doing the test. Again, a weakness of this method is that one cannot give their full attention to every participant. Typically there are two consultants and around 10 participants to a session.
|> Contextual: similar to one-on-one, but is conducted in the user’s own environment.
|> Comparative: several websites are compared to determine what the best interface is. Users are asked to carry out the same task on a number of different websites.


There are three main testing stages: prepare, research and report.


During the first stage, objectives of testing should be set and user profiles created. You will need to select appropriate people to interview for your evaluation. Herrod advises against self-recruitment, as we can end up with a more diluted audience.
A professional recruitment agency will use screeners and select the right candidates for you.

The alternative is to use a consultant, but this is a more expensive option.

At the preparation stage, you also need to decide what research method to use and develop the relevant tasks and questionnaires for testing later on.

Don’t design tasks so users feel they are expected to find something. It’s better to come up with scenarios that is typical tasks they would be expected to perform and observe their behaviour.


During this stage, the research is collected using the method chosen during the preparation stage.

Observe the users as they carry off the testing tasks. Focus on their actions and what they are saying, as this can be valuable feedback. Keep in mind that the minimum number of participants should be six.

Decide whether you will be taking notes or recording during the tests. Recordings might be tedious to review later on.

Once the results have been gathered, you need to analyse and draw conclusions from them.


When reporting the results, decide on the format, level of detail and quality required.
The format of the report does not necessarily need to be paper. It can be anything from a PowerPoint to a webpage.

It is up to you what level of detail you want to go into. Place different issues into categories, then clearly explain what the problems are and how to correct them.

The most important part of reporting is making recommendations, so ensure that this section is adequately covered.

Reducing Cost

Herrod provided a number of tips for cutting down on testing costs.

|> Interview location: It can be time-consuming and costly to visit the people you are interviewing. Just consider the time spent travelling and parking costs, for instance. Try to get people to come to you instead.
|> Documentation: Documentation doesn’t need to be fancy. All you need is a short task list and to capture the user’s responses.

|> Recording sessions: Recording sessions should be avoided, as they can get lengthy and time-consuming to review.
|> Reporting: Identify what areas you want to report and the level of reporting.
|> Participant numbers: Cut down on the number of participants. Keep in mind that six is considered the minimum and around 12 the maximum.

Conversely, there are some things you shouldn’t try to cut back on:

|> Recruitment: Although it is cheaper to do the recruiting yourself, you should employ the services of a professional agency, as they are more likely to find better candidates for you. Having the right people as users is paramount to the success of testing.
|> Incentives: You should not try to cut down on incentives, otherwise you may have trouble recruiting. It’s important to give people something in return for their time.
|> Recommendations: Recommendations are the most important part of the testing process and should not be cut back on.

Extreme Usability

Extreme Usability or Guerilla Usability might be an attractive option, because it’s faster and cheaper to perform.

The technique, while useful, is not ideal. Problems can occur, especially in regards to recruitment, when surrogate users are used. For instance, employing people with call centre experience when reviewing a services website. Problems could arise if they are not accurate portrayals of users.

It is vital to have the right user representatives testing the website, people who will actually be using the site later on.

This method might be appropriate for Web agencies, small in-house reviews and certain agile stages. A Web agency would require less documentation as there would be more contact between the developers and everything is done in-house.

Traditional usability

This is a more thorough and in turn a more costly method. More time is spent on the analysis, so more detailed results are produced.

Traditional usability is suitable for larger organisations and government departments.

Additional Advice

Following the presentation, we asked Lisa Herrod for some additional advice on user testing.

Herrod said when designing a website from scratch, there is a lot of research we can do at the start. Usability testing can be performed on low-fidelity diagrams, mock-ups or high-fidelity visual designs.
“Just before that we might be looking at things like wireframes, the labelling and the site structure”, she said.

Herrod said that we should be aware that bias can exist. On a large project we may have more than one testing stage. If we recruit a round of users for the first stage and then use those same users again, we might get bias, especially if we didn’t take all their feedback from the first stage on board.

When using heuristics in addition to visual design it’s important to consider things like accessibility and the structure of the site. Some good heuristics are available online to help you, she said.

Herrod says some different ways of measuring user responses are the simple yes/no, using different scales, for instance one to 10 or Humphrey field testing.