Chapter Six

Training Evaluation
 Objectives
O Explain why evaluation is important

O Identify and choose outcomes to evaluate a

training program
O Discuss the process used to plan and
implement a good training evaluation
O Discuss the strengths and weaknesses of
different evaluation designs

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Objectives
O Choose the appropriate evaluation design
based on the characteristics of the company
and the importance and purpose of the
training
O Conduct a cost-benefit analysis for a
training program
O Explain the role of workforce analytics and
dashboards in determining the value of
training practices
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Key Terms
O Training effectiveness: benefits derived
training
O Training evaluation: process of determining
training effectiveness
O Training outcomes: measures to evaluate
training effectiveness
O Evaluation design: data will be collected
and which data will be collected for
training evaluation
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Formative Evaluation
O Refers to evaluation of training that takes
place during program design
O Helps to ensure training is well organized,
runs smoothly, and that trainees learn and
are satisfied
O Provides information on how to make a
program better

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Summative Evaluation
O Refers to evaluation conducted to
determine if training has lead to desirable
outcomes
O Examines whether trainees have improved
or acquired knowledge, skills, attitudes,
behaviors, or other outcomes
O Includes examining the business impact of
training

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Importance of Evaluation
O There are multiple reasons to evaluate:
o identify a program’s strengths and
weaknesses
o assess what features of training content
and context matter
o identify which trainees benefited
o gather information for marketing
training
o determine financial benefits and costs

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 The Evaluation Process

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Training Outcomes

Learning & Behavior &

Reactions Cognitive Skill-Based
Outcomes Outcomes

Affective Return on
Results
Outcomes Investment

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Reactions
O Trainees’ perceptions of the training
experience relating to the content,
facilities, trainer, and methods
O Key questions to consider include:
o Did the trainees like the program?
o Did the environment help learning?
o Was the material meaningful?
O Typically measured at the end of training

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Learning & Cognitive
Outcomes
O Relate to familiarity with information,
including principles, facts, techniques,
procedures, and processes
O Typically measured via paper-and-pencil
tests and self-assessments
O Tests often preferred over self-assessments

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Behavior & Skill-Based
Outcomes
O Relate to proficiency with technical skills,
motor skills, and behavior
O Include learning and transfer
O Learning often assessed via work samples
O Transfer often assessed via observation or
managerial/peer ratings

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Affective Outcomes
O Include attitudes and motivation
o e.g., self-efficacy, employee engagement,
motivation to learn, tolerance for
diversity, attitudes toward safety
O The attitude of interest depends on the
training objectives
O Affective outcomes often measured via
surveys
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Results
O Did training have an impact on meaningful
business outcomes?
O Outcomes used to determine the benefits
of training to the company
o e.g., reduced costs, increased employee
retention, increased sales, improved
quality or customer service

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Return on Investment
O Involves comparing the training program's
benefits to its costs in monetary terms
o Benefits: value the company gains

o Direct costs: salaries for employees

involved in training, program materials,
facilities, and travel
o Indirect costs: costs not related directly
to design and delivery
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Which Outcomes?

Relevance Reliability

Discrimination Practicality

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Relevance
O The extent to which training outcomes are
related to the learned capabilities
emphasized in the program
O Contamination refers to the inclusion of
inappropriate or irrelevant outcomes
O Deficiency refers to the omission of
important information

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Deficiency, Relevance, &
Contamination

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Reliability
O The extent to which outcomes can be
measured consistently over time
O Evaluators are concerned with consistency
over time, such that items do not change
in meaning or interpretation over time

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Discrimination
O The extent to which measured
performance reflects a true difference
O We want tests that discriminate between
high and low performance
o A test that is too easy may not
discriminate
o In this instance, both high and low
performers would do well and appear
“good” even though they are not
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Practicality
O The extent to which outcomes can be
easily measured and collected
O Companies often claim that measurement
is too burdensome

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Training Evaluation Practices

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Which Outcomes?
O While collecting data on all outcomes is
ideal, it may not always be necessary
O Consider the scope of the training and
practical considerations
O Do not assume that positive reactions lead
to transfer

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Threats to Validity
O Threats to validity refer to factors that will
lead an evaluator to question the results
O Internal validity is the believability of the
study
O External validity is the generalizability of
the evaluation results to other groups and
situations

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Controlling for Threats
O There are three ways to control for threats:
o Use pre-tests and post-tests
o Use a control group
o Random assignment of employees to
control and training groups

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Post-test Only
O Involves collecting only post-training
outcomes
O Appropriate when trainees can be
expected to have similar levels of
proficiency prior to training
O Strengthened with a control group

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Pre-test / Post-test
O Involves collecting both pre-training and
post-training outcomes to determine if a
change has occurred
O Is there a difference in outcomes before
and after training?

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Pre-test / Post-test with
Comparison Group
O Includes pre-training and post-training
outcomes and use of a control group
O If the post-training improvement is greater
for the training group, there is evidence
that training was responsible

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Time Series
O Involves collecting measures at periodic
intervals pre- and post-training
O A comparison group may be used

O The strength of this design can be

improved by using reversal, which refers to
a time period when participants no longer
receive training

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Solomon Four-Group
O Combines the pre-test/post-test
comparison group design and the
posttest-only control group design
o Pre-test, treatment, post-test
o Pre-test, no treatment, post-test
o No pre-test, treatment, post-test
o No pre-test, no treatment, post-test

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 When Evaluation May Not Be
Necessary
O Time constraints

O Managers and trainees may lack expertise

O The company may view training as an

investment from which it expects little or
no return

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 When Evaluation is a Must
O The training is ongoing and has the
potential to affect many employees
O The training program involves multiple
classes and a large number of trainees
O The expertise exists to evaluate
O The cost of training is significant
O There is sufficient time and interest

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Cost-Benefit Analysis
O Companies may desire to quantify whether
the benefits of training outweigh the costs
O Cost-benefit analysis
o Process of determining the economic
impact of training using accounting
methods that look at training costs and
benefits

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Cost Categories
1. program development or purchase
2. instructional materials
3. equipment and hardware
4. facilities
5. travel and lodging
6. salary of the trainer and support staff
7. cost of lost productivity or replacement
workers while trainees are away

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Determining Benefits
O A number of methods can be used to
identify benefits
o literature that summarizes benefits
o pilot training programs
o observing successful job performers
asking trainees and managers for
estimates

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Calculating Return on
Investment (ROI)
O Identify annual change in outcomes
O Place a monetary value on the outcomes
O Determine the annual change in value
O Determine training costs
O Calculate net benefit
O Divide net benefits by costs

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Utility Analysis
O Assessing the dollar value of training
based on:
o estimates of the difference in
performance between trained and
untrained employees
o number of individuals trained
o length of time training is expected to
influence performance
o variability in performance in the
untrained group of employees

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Success Cases & Return on
Expectations
O Success cases refer to concrete examples
showing how learning has led to results
the company finds worthwhile and
credible
O Return on expectations (ROE)
demonstrates to key stakeholders that
their expectations about training have
been satisfied

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Big Data
O Big data refer to complex datasets
compiled across different systems,
including marketing, sales, HR, finance,
accounting, customer service, and
operations
O Three dimensions characterize big data
o volume
o variety
o velocity

Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
 Big Data & Training
O Big data help make decisions about
human capital based on data, rather than
intuition and conventional wisdom
O Big data can be used to:
o evaluate the effectiveness of programs
o determine their impact on business
results
o develop predictive models for
forecasting training needs, course
enrollments, and outcomes
Copyright © 2017 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.