System testing to validate documentation, training, and usability

System tests don't stop with the technology. Your programs should be usable and well documented--and don't forget the user training. Learn about how system testing plays into these aspects of your applications.

System testing is a collection of tests designed to verify that a program or system of programs is ready for production. In the previous two articles on system testing, we looked at performance and stress testing and security testing and requirements testing. But many other aspects of system testing may be applicable for your projects. Not all of these aspects are directly related to an IT system, including two of the tests we’ll look at here: documentation and training testing. The third test we’ll discuss, usability testing, determines how easy and intuitive the system is for people to interact with.

Documentation testing
Do you know how frustrating it is to buy something that needs assembly, only to find that the instructions are vague or wrong? Imagine a user of your system trying to follow your procedures, only to find that crucial information is missing. Good documentation can keep frustration levels manageable and learning curves short.

The purpose of documentation testing is to thoroughly go through all written material that will be presented to the user as a part of implementation. In many cases, the customers themselves prepare this documentation, but that doesn’t mean it is correct. The testing needs to be done a few different ways.
  • Ask people who are unfamiliar with the application to follow all documented procedures. These should be written to provide step-by-step guidance on how to accomplish a given task. If the testers can’t successfully complete the procedures, the documentation needs to be improved.
  • Have people with strong English, punctuation, and grammar skills review all the material for professionalism and readability. It would be great to use professional proofreaders. Otherwise, use employees with strong written communication skills.
  • Try out all alternative ways that are documented to accomplish a task. In many cases, the primary way works, but the alternatives do not.
  • If you are describing policies or standards, make sure that the appropriate authorities in your organization review and approve your wording. You don’t want to misquote or misapply an important company policy.
  • Evaluate any manual forms, checklists, and templates to ensure that they are accurate and that the appropriate information is being collected.

If you have the right people identified for this test, and you go over all the manual documentation thoroughly, you will reduce or eliminate a source of frustration and resistance after the application is implemented.

Training testing
When do you first perform training on your application? Do you experiment with the first group of participants? A better approach is to test your training as a part of system testing. This implies that the training must be ready at this point in the life cycle and not created at the very end of the project. And, like all testing, you want to perform testing in a controlled test environment to ensure that it is effective, accurate, and bug-free.

First, you want to have your instructor practice all of the instructor-led training at least once. This will help prepare the instructor for the live classes and ensure that all the material is complete and prepared correctly. You don’t want the instructor to go into the first class and realize he or she doesn’t have color slides or necessary flipcharts, or that too many people have been invited.

In addition, you will want to run through and evaluate all the self-paced training, including computer-based training. Again, as in previous examples, have people who do not know of your application review the training to determine whether it is understandable.

In the case of distance learning, be sure of the technology you will be using and test it. Do you have all of the equipment you will need at your location? Will the remote locations have the appropriate equipment, as well as a proper environment (i.e., one free of distractions and interruptions) for learning? Is the training staff sufficiently comfortable with this training approach? Make sure that the technology is capable of supporting any remotely delivered multimedia training, self-paced or otherwise.

Training material is a deliverable of your project. It all needs to be tested for accuracy and defects before you roll it out for the first time.

Usability testing
Have you ever been on a project where you delivered all the functional and technical requirements the customer asked for and yet the users of the system complained that it was a piece of junk? If you haven’t, you’re lucky, because it happens fairly often. These are applications that are hard to use and difficult to learn.

Usability doesn’t come directly from the business requirements. While you may have a requirement for an application to be easy to learn, usability decisions are made in the design phase. Most project teams don’t include human factors experts, but many commonsense techniques will ensure that a system is simple to navigate and easy to understand. You can design a separate usability test as part of the system test to validate that the end users understand and use the system to its fullest.

In general, you don’t want IT people to perform the usability tests. The users, or people who have not seen the application, are the best candidates. Since this is a subjective test, a survey can work well. Have a group of users access the system and carry out scenarios that mirror their normal jobs. Then, using a 1-through-5 scale, ask them how satisfied they were in the following areas:
  • Ease of navigation from screen to screen
  • Ability to spot necessary navigation boxes and buttons
  • Overall complexity of the screens
  • Color schemes used on the screens
  • Error messages and the ability to understand what was required to correct errors
  • Availability and clarity of online help
  • Sharpness of graphics, charts, and graphs
  • Consistency among different screen sizes, browsers, and other hardware types

As you can see, usability is mostly an online concern, but you can perform usability surveys for batch reports too. These and other questions can help determine whether the application is usable. If survey questions are rated low, ask for specific examples. Performing usability testing and correcting poorly designed screens and reports can go a long way toward ensuring that the application will be successfully adopted by the customers when it is implemented.

Make it effective for the users
If your customers won’t use your application, there isn’t much point in creating it. Keep the following points in mind as you determine whether you’re providing the best materials you can to help them learn and use the system and you verify how usable your application is.
  • Documentation testing ensures that all written material designed to supplement or assist the user is understandable and accurate.
  • Training is also a deliverable of your project. It should be tested in a controlled environment to make sure that the material educates the attendees as expected.
  • Many factors make the difference between full and partial success on a project. One key consideration is how the system interacts with the users.
  • Usability testing ensures that the system is understandable and can be fully utilized by the people who need to interact with it.
  • Usability testing is not just for online operations. Batch reports, for example, should also be verified with users.

Next steps
We’ve covered most of the aspects of system testing, but three tests are left. The next article will cover disaster recovery testing, multisite testing, and interface testing. After that, we will be on to the last of the major testing stages—user acceptance testing.

Project management veteran Tom Mochal is director of internal development at a software company in Atlanta. Most recently, he worked for the Coca-Cola Company, where he was responsible for deploying, training, and coaching the IS division on project management and life-cycle skills. He's also worked for Eastman Kodak and Cap Gemini America and has developed a project management methodology called TenStep.


Editor's Picks