Professional Documents
Culture Documents
CHAPTER 6
TRAINING EVALUATION
This chapter provides an overview of how to evaluate training programs, including the types of
outcomes that need to be measured and the types of evaluation designs available. The chapter
focuses on the evaluation of training programs and learner outcomes. It explains the criticality of
evaluating whether the training has accomplished its objectives and, particularly, whether job
performance and organizational results have improved as a result. Formative and summative
evaluation are discussed and compared and reasons for evaluating are identified. The process of
evaluating training is outlined and outcomes used to evaluate training are described in some
detail. Kirkpatrick’s five-level framework incorporating five major levels of evaluation is
highlighted, and the six major categories of outcomes are presented more extensively. Another
important issue, regarding how good the designated outcomes are, is addressed. Perhaps most
importantly, evaluation designs, important elements of evaluation design, and the preservation of
internal validity are discussed as well as the calculation of return on investment for the training
dollar. In an environment of accountability, knowledge of how to show return on investment is
invaluable. Further, this chapter gives the student knowledge of the various evaluation strategies
and how to choose an approach. A list of key terms, discussion questions, and application
assignments follow the end of the chapter.
Objectives
I. Introduction
A. Training effectiveness refers to the benefits that the company and the trainees experience
as a result of training. Benefits for the trainees include learning new knowledge, skills,
and behaviors. Potential benefits for the company include increased sales, improved
quality and more satisfied customers.
B. Training outcomes or criteria refer to measures that the trainer and the company use to
evaluate training programs.
C. Training evaluation refers to the process of collecting data regarding outcomes needed to
determine whether training is effective.
6-1
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
A. Companies have made large dollar investments in training and education and view
training as a strategy to be successful, they expect the outcomes or benefits related to
training to be measurable.
B. Training evaluation provides a way to understand the investments that training produces
and provides information needed to improve training.
Formative Evaluation
Formative evaluation refers to the evaluation of training that takes place during program
design and development. It is conducted to improve the training process; ensuring that the
training program is well-organized and runs smoothly and trainees are learning and are
satisfied with the training.
A. As a result of the formative evaluation, training content may be changed to be more
accurate, easier to understand, or more appealing; the training method can be adjusted to
improve learning.
B. Introducing the training program as early as possible to managers and customers helps in
getting them to buy into the program, which is critical for their role in helping employees
learn and transfer skills; it also allows their concerns to be addressed before the program
is implemented.
C. Pilot testing is the process of previewing a training program with potential trainees and
their managers, or other customers. The pilot testing group is then asked to provide
feedback about the content of the training as well as the methods of delivery. This
feedback enables the trainer to make needed improvements to the training.
Summative Evaluation
Summative evaluation is evaluation conducted to determine the extent to which trainees have
improved or acquired knowledge, skills attitudes, behaviors, or other outcomes specified in
the learning objectives, as a result of the training. Reasons training programs should be
evaluated:
A. To identify the program’s strengths and weaknesses, including whether the program is
meeting the learning objectives, the quality of the learning environment, and if transfer of
training back to the job is occurring.
B. To assess whether the various features of the training context and content contribute to
learning and the transfer of learning back to the job.
C. To identify which trainees benefited most or least from the program and why.
D. To gather information, such as trainees’ testimonials, to use for marketing training
programs.
E. To determine financial benefits and costs of the program.
F. To compare the costs and benefits of training versus other human resource investments.
G. To compare the costs and benefits of various training programs in order to choose the
most effective programs.
6-2
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
A. The evaluation process should begin with determining training needs. Needs assessment
helps identify what knowledge, skills, behavior, or other learned capabilities are needed.
B. The next step in the process is to identify specific, measurable training objectives to
guide the program.
C. Based on the learning objectives and analysis of transfer of training, outcome measures
are designed to assess the extent to which learning and transfer have occurred.
D. Once the outcomes have been identified, the next step is to determine an evaluation
strategy. Factors such as expertise, how quickly the information is needed, change
potential, and the organizational culture should be considered in choosing a design.
E. Planning and executing the evaluation involves previewing the program (formative
evaluation), as well as collecting training outcomes according to the evaluation design.
The results of the evaluation are used to modify, market, or gain additional support for
the program.
One of the original frameworks (five-level) for identifying and categorizing training
outcomes was developed by Kirkpatrick. Levels 1 and 2 measures are collected before
trainees return to their jobs. Levels 3, 4, and 5criteria measure the extent to which the
training transfers back to the job.
Reaction Outcomes
Reaction outcomes refer to the trainees’ perceptions of the training experience, including the
content, the facilities, the trainer and the methods of delivery. These perceptions are typically
obtained at the end of the training session via a questionnaire completed by trainees, but
usually are only weakly related to learning or transfer.
6-3
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
Affective Outcomes
Affective outcomes include attitudes and motivation. Affective outcomes that might be
collected in an evaluation include tolerance for diversity, motivation to learn, safety attitudes,
and customer service orientation. The attitude of interest depends on training objectives.
Results
Results are those outcomes used to determine the benefits of the training program to the
company. Examples include reduced costs related to employee turnover or accidents,
increased production, and improved quality or customer service.
Return on Investment
Return on Investment involves comparing the training program’s benefits in monetary terms
to the program’s costs, both direct and indirect.
A. Direct costs include salaries and benefits of trainees, trainers, consultants, and any others
involved in the training; program materials and supplies; equipment and facilities; and
travel costs.
B. Indirect costs include office supplies, facilities, equipment and related expenses not
directly related to the training program; travel and expenses not billed to one particular
program; and training department management and staff support salaries.
C. Benefits are the value the company receives from the training.
D. Training Quality Index (TQI) is a computer application that collects data about training
department performance, productivity, budget, and courses and allows for detailed
analysis of the data. TQI tracks all department training data into five categories:
effectiveness, quantity, perceptions, financial impact, and operational impact.
Relevance
A. Criteria relevance refers to the extent to which training outcomes appropriately reflect the
content of the training program. The learned capabilities needed to successfully complete
the training program should be the same as those required to successfully perform one’s
job.
B. Criterion contamination refers to the extent that training outcomes measure inappropriate
capabilities or is affected by extraneous conditions.
C. Criterion deficiency refers to the failure of the training evaluation measures to reflect all
that was covered in the training program.
Reliability
Reliability is the degree to which training outcomes can be measured consistently over time.
Predominantly, we are concerned with consistency over time, such that a reliable test
contains items that do not change in meaning or interpretation over time.
6-4
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
Discrimination
Discrimination refers to the degree to which trainees’ performance on the outcome actually
reflects true differences in performance; that is, we want the test to discriminate on the basis
of performance and not other things.
Practicality
Practicality is the ease with which the outcome measures can be collected. One reason
companies give for not including learning, performance, and behavior outcomes in their
evaluation of training programs is that collecting them is too burdensome.
Surveys of companies’ evaluation practices indicate that reactions (an affective outcome) and
cognitive outcomes are the most frequently used outcomes in training evaluation. Despite the
less frequent use of cognitive, behavioral, and results outcomes, research suggests that
training can have a positive effect on these outcomes.
The design of the training evaluation determines the confidence that can be placed in the
results. No training evaluation can be absolutely certain that the results of the evaluation are
completely true.
6-5
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
6-6
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
group and comparison group for which outcomes are measured both pre and post training
and a training group and comparison group for which outcomes are measured only after
training. This design provides the most controls for internal and external validity.
A. Cost-benefit analysis of training is the process of determining the net economic benefits
of training using accounting methods. Training cost information is important for several
reasons:
1. To understand total expenditures for training, including direct and indirect costs
2. To compare the costs of alternative training programs
3. To evaluate the proportion of the training budget spent on the development of
training, administrative costs, and evaluation as well as how much is spent on various
types of employees e.g., exempt versus nonexempt
4. To control costs
B. There is an increased interest in measuring the ROI of training and development
programs because of the need to show the results of these programs to justify funding and
to increase the status of the training and development function.
C. The process of determining ROI:
1. Understand the objectives of the training program.
2. Isolate the effects of training from other factors that might influence the data.
3. The data are converted to a monetary value and ROI is calculated.
D. Because ROI analysis can be costly, it should be limited only to certain training
programs.
6-7
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
Determining costs
A. The resource requirements model compares equipment, facilities, personnel, and
materials costs across different stages of the training process (needs assessment,
development, training design, implementation, and evaluation).
B. There are seven categories of cost sources: costs related to program development or
purchase; instructional materials; equipment and hardware; facilities; travel and lodging;
and salary of the trainer and support staff along with the cost of either lost productivity or
replacement workers while trainees are away from their jobs for the training.
Determining Benefits
A. Determining benefits can be done via a number of methods, including:
1. Technical, practitioner and academic literature summarizes benefits of training
programs.
2. Pilot training programs assess the benefits from a small group of trainees before a
company commits more resources.
3. Observing successful job performers can help to determine what successful job
performers do differently than unsuccessful performers.
4. Asking trainees and their managers to provide estimates of training benefits.
B. To calculate return on investment, follow these steps:
1. Identify outcomes.
2. Place a value on the outcomes.
3. Determine the change in performance after eliminating other potential influences on
training results.
4. Obtain an annual amount of benefits from training by comparing results after training
to results before training.
5. Determine the training costs.
6. Calculate the total savings by subtracting the training costs from benefits.
7. Calculate the ROI by dividing benefits by costs. The ROI gives an estimate of the
dollar return expected from each dollar invested in training.
6-8
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
A. Metrics are valuable for benchmarking purposes, for understanding the current amount of
training activity in a company, and for tracking historical trends in training activity.
B. The value of learning activities is best determined through the use of workforce analytics.
Workforce analytics refers to the practice of using quantitative methods and scientific
methods to analyze data from human resource databases, corporate financial statements,
employee surveys, and other datasources to make evidence-based decisions and show that
human resource practices (including training, development, and learning) influence
important company metrics.
C. Dashboards refer to a computer interface designed to receive and analyze the data from
departments within the company to provide information to managers and other
decisionmakers.
Chapter Summary
This chapter provides sound base of knowledge regarding training evaluation, the issues
surrounding it, and how to approach it. Reasons for evaluating training were described, and the
process of evaluating training was outlined. Kirkpatrick’s model of evaluation was explained, as
well as the five major categories of outcomes that can be measured to evaluate training
effectiveness. The six outcomes (reaction, cognitive, skill-based, affective, results, and ROI)
used in evaluating training programs were explained. Good training outcomes need to be
relevant, reliable, discriminate, and practical. Next, threats to both internal and external validity
were discussed. Various evaluation designs were explained with an emphasis on related costs,
time, and strength. Return on Investment (ROI) and cost-benefit analysis were explained, and
examples given. The chapter concluded with a listing of key terms, discussion questions, and
application assignments.
6-9
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
Discussion Questions
Answer: The company’s management would need to be made aware of the investment made
by the company for the training and the need for its summative evaluation to determine if the
training program is effective, and formatively evaluate it to identify its strengths and
weaknesses to better accomplish training. Evaluation is vital to improving a training
program, or deciding whether to replace it completely with a better program or a non-training
option. (p. 234-235)
2. What do threats to validity have to do with training evaluation? Identify internal and external
threats to validity. Are internal and external threats similar? Explain.
Answer: If threats to validity exist, the evaluator may question whether a training program
was really effective or if possible benefits were the result of other factors. Internal threats to
validity affect the believability of the study, while threats to the external validity affect the
believability of the program’s benefits for future use. Characteristics of the company, the
outcome measures, and the persons in the evaluation study are the internal threats to validity.
Reaction to pretest, reaction to evaluation, interaction of selection and training, and
interaction of methods are the external threats to validity. Internal and external threats are not
similar. Threats to external validity relate to how study participants react to being included in
the study and the effects of multiple types of training, whereas internal validity provides
confidence that the results of the evaluation are due to the training program and not to
another factor. (p. 249-250)
3. What are the strengths and weaknesses of each of the following designs: posttest-only,
pretest/posttest with comparison group, and pretest/posttest only?
Answer: A post-test-only design requires less time and effort than the other, but fails to
recognize factors such as initial differences between the control group and the group that
received the training. Pretest/post-test only acknowledges the performance of the trained
group before the training, but ignores other business factors that may be occurring between
the time of the two evaluations. Pretest/post-test with comparison group is the most thorough,
acknowledging the highest number of factors, but involves the most time and effort to collect
data. (p. 251-253)
4. What are results outcomes? Why do you think that most organizations don’t use results
outcomes for evaluating their training programs?
Answer: Results outcomes are level 4 criteria in Kirkpatrick’s framework. They are the
benefits of the training program for the company. These types of payoffs could be difficult to
determine, as many are long-term benefits and are difficult to track, while for others, such as
increased customer satisfaction it may be difficult to place an exact monetary value. (p. 243)
6-10
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
5. This chapter discussed several factors that influence the choice of evaluation design. Which
of these factors would have the greatest influence on your choice of and evaluation design?
Which would have the least influence? Explain your choices.
Answer: Student answers would vary. The factors that influence the choice of evaluation
design are:
• Change potential-which determines whether the program can be modified
• Importance-which determines whether ineffective training affects customer service,
product development, or relationships among employees
• Scale-which determines the number of trainees involved
• Purpose of training, which determines whether training is conducted for learning, results,
or both
• Organization culture-which determines whether demonstrating results is a part of
company norms and expectations
• Expertise-which determines the possibility of analyzing a complex study
• Cost-which determines whether the evaluation is expensive
• Time frame-which determines when the information is needed (p. 256; Table 6.11)
6. How might you estimate the benefits of a training program designed to teach employees how
to use the Internet to monitor stock prices?
Answer: Test could be performed to see if the employees are capable of monitoring the stock
prices; surveys could be used to see how frequently they monitor stock in a given day to
determine how well they have integrated that knowledge into their behavior. Then employees
could estimate how much behavior benefits the company. To determine the economic
benefits of the training program, ROI can be calculated through a cost benefit analysis. Also,
the following methods may help to identify the benefits of the training program:
• Technical, academic, and practitioner literature summarizes the benefits that have been
shown to relate to the specific training program.
• Observance of successful job performers helps a company determine what successful job
performers do differently than unsuccessful job performers.
• Trainees and their managers provide estimates of training benefits. (p. 259)
Answer: If the evaluation is done after the program is completed, a posttest only design could
be used, comparing the group of managers to a control group. Areas of evaluation may
include looking at various decisions made by the managers over several weeks and
determining their effectiveness in solving various problems, and the amount of time taken to
6-11
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.
Chapter 06 - Training Evaluation
organize solutions. The outcomes that would be collected will be skill-based and affective
outcomes.
If the managers had not yet attended the program, they can be used as a comparison group or
pre-test skills and productivity can be recorded for later comparison. Hence, pretest/post-test
with comparison group method would be applied. (p. 241-242; 253)
8. What practical considerations need to be taken into account when calculating a training
program’s ROI?
Answer: The costs and benefits of a training program must all be considered. The costs are
usually calculated using accounting, while the benefits are not always as clearly defined.
Showing the link between training and higher-level strategic business outcomes can be very
problematic. The effects of training from other factors that might influence the data must be
isolated. Lastly, the data is converted to a monetary value and ROI is calculated. (p. 263)
9. What metrics might be useful for evaluating the effectiveness of a company’s training
function? Discuss and rate their importance.
Answer: Student’s answers will vary. The metrics are valuable for benchmarking purposes,
for understanding the current amount of training activity in a company, and for tracking
historical trends in training activity. However, collecting these metrics does not address such
issues as whether training is effective or whether the company is using the data to make
strategic training decisions. Each company needs to choose metrics that are related to its
business strategy or goals. (p. 265; Table 6.15)
10. What is return on expectations (ROE)? How can it be used to show the costs and benefits of
training without collecting statistics and conducting analyses? Explain its strengths and
weaknesses compared to a cost-benefit analysis.
Answer: One way to establish the value of training and overcome the difficulty of evaluating
training using a design that can rule out and isolate its effects on results is to rely on return on
expectations. Return on expectations (ROE) refers to the process through which evaluation
demonstrates to key business stakeholders, such as top-level managers, that their
expectations about training have been satisfied. ROE depends on establishing a business
partnership with business stakeholders from the start of a training program through its
evaluation. The ROE is used as an estimate in an ROI analysis. (p. 264)
6-12
© 2013 by McGraw-Hill Education. This is proprietary material solely for authorized instructor use. Not authorized for sale or distribution in any
manner. This document may not be copied, scanned, duplicated, forwarded, distributed, or posted on a website, in whole or part.