No one likes to waste money—especially in this turbid economy—and yet that is exactly what you will risk doing if you don’t take the necessary precautions when choosing training courseware. Using an evaluation process to determine the courseware that best fits your instructors, your users, and your budget is vital to ensuring that this often-expensive purchase is done correctly.
Here are the details from my company’s recent instructor-led training courseware evaluation process, which could also be used to fit other types of training materials, such as CBT or Web-based learning.
Assemble your team
Deciding who will be involved in the evaluation process is relatively straightforward. At this point, the people who will actually be using the material in their training classes should make up the majority of the evaluation team. However, keep the size of your team limited to no more than four or five people. The more people involved, the harder it will be to reach any sort of decision.
In my organization, besides trainers, we also included someone with basic knowledge of the applications and who was familiar with end-user needs from working with the help desk. While additional end users were involved later in the process, selecting someone at this stage provided a unique perspective, sometimes differing from the trainers involved.
Evaluate the courseware
Each of the evaluators on our team rotated through sample materials from each vendor. We requested, when possible, the same lessons for each application. For instance, evaluating each vendor’s lesson on mail merge allowed us to compare the approach and coverage of the same material.
Each vendor was evaluated on a numeric scale from 1 to 5 on a variety of criteria, including:
- Level of difficulty
- Level of detail
- Balance of concepts and skills presented
- Visual appeal
- Fit with style of instruction
- Ease of customization (we specifically looked at customizable courseware materials)
It’s a good idea to have each trainer involved in the process prepare a short lesson from the materials. This allows you to test the ease of preparation and to present the materials to a small group of people to better determine if it fits with your instructional style.
After we arrived at this point, the numeric rankings were tallied to give overall average scores for each vendor. The three favorites of each person on the team were also compiled.
Narrowing down the choices
Following the review and ranking of materials, team members met to decide on three finalists. Using the overall ranking information, we were able to quickly eliminate several vendors. Using the overall rankings and the top favorites of each team member, we then discussed the pros and cons of each vendor and, within an hour, we selected three top vendors. At no point in this process was pricing information disclosed to the evaluators. This was done to avoid skewing the results through any preconceptions that may have revolved around price.
One of the most important parts of our evaluation process was to enlist the help of three or four end users. One thing we learned during this process is to meet with the end users to be sure they understand the purpose of the evaluation and the materials. One of our evaluators received information secondhand via their supervisor and did not fully understand the goals of the evaluation, which skewed his results considerably.
End users were given two sample lessons from each vendor and a simplified evaluation form. Their evaluation form consisted of only eight points (along with enough room to write out comments):
- Well-organized and easy to follow
- Helpful graphics and screen shots
- Visually pleasing layout/format
- Appropriate level of difficulty
- Covers both concepts and practical skills
- Clearly stated objectives
- Useful as a reference after class
Negotiations with vendors
At the same time end users were undergoing their evaluation process, we contacted each of the top three vendors and explained they were finalists in our selection process. We determined the list of courses we were interested in purchasing and asked for pricing information. We also worked to clarify licensing issues. (For more on licensing issues related to training courseware, see TechRepublic’s article "Reduce your prep time with customizable courseware.") When necessary, we negotiated by quoting prices from other vendors. This process can be quite lengthy, so I recommend starting before selecting just one vendor. You should also be prepared to choose from any of your top choices.
After reviewing the evaluations from the end users and comparing pricing information, a final selection was made based on overall scores and pricing.
The training debate
Some view end-user training as a waste of the support staff’s time and energy, with no tangible benefits. Others believe this proactive process directly affects the number and severity of a help desk’s service calls. What do you think? Post a comment or write to Janice Ward and share your opinions and experiences.