Because learning changes everything.

Chapter 6
Training Evaluation

© 2023 McGraw Hill, LLC. All rights reserved. Authorized only for instructor use in the classroom.
No reproduction or further distribution permitted without the prior written consent of McGraw Hill, LLC.
 Objectives

1. Explain why evaluation is important

2. Identify and choose outcomes to evaluate a training program
3. Discuss the process used to plan and implement a good training
evaluation
4. Discuss the strengths and weaknesses of different evaluation
designs
5. Choose the appropriate evaluation design based on the
characteristics of the company and the importance and purpose of
the training
6. Conduct a cost-benefit analysis for a training program
7. Explain the role of big data workforce analytics and dashboards in
determining the value of training practices

© McGraw Hill, LLC

 Evaluation is Important in Many Respects

o Identify a program’s strengths and weaknesses

o Assess what features of training content and context matter
o Identify which trainees benefited
o Gather information for marketing training
o Determine financial benefits and costs

© McGraw Hill, LLC

 Key Terms

o Training effectiveness: benefits derived from training

o Training evaluation: process of determining training effectiveness
o Training outcomes: measures to evaluate training effectiveness
o Evaluation design: how data will be collected for training evaluation

© McGraw Hill, LLC

 Formative Evaluation

o Refers to evaluation of training that takes place during program design

o Helps to ensure training is well organized, runs smoothly, and that
trainees learn and are satisfied
o Provides information on how to make a program better

© McGraw Hill, LLC

 Summative Evaluation

o Refers to evaluation conducted to determine if training has lead to

desirable outcomes
o Examines whether trainees have improved or acquired knowledge,
skills, attitudes, behaviors, or other outcomes
o Includes examining the business impact of training

© McGraw Hill, LLC

 The Evaluation Process

The Evaluation Process Long Description

© McGraw Hill, LLC

 Kirkpatrick’s Model of Training Evaluation

Level 1: Level 2:
Reactions Learning

Level 3:
Level 4: Results
Behavior

Level 5: Return-
on-Investment
(ROI)

Kirkpatrick’s Model of Training Evaluation Long

Description

© McGraw Hill, LLC

 Extended Evaluation Framework

Learning or Behavior and

Reaction
cognitive skill-based
outcomes
outcomes outcomes

Affective
Results ROI
outcomes

Extended Evaluation Framework Long Description

© McGraw Hill, LLC

 Reactions

o Trainees’ perceptions of the training experience relating to the

content, facilities, trainer, and methods
o Key questions to consider include:
o Did the trainees like the program?
o Did the environment help learning?
o Was the material meaningful?

Typically measured at the end of training

© McGraw Hill, LLC

 Net Promoter Score (NPS)

o Refers to a score designed to measure satisfaction by asking them to

rate the likelihood of recommending it to a peer by using a 0 to 10
point scale
o The number on the scale that an employee chooses is then classified
into one of three categories: “Detractors,” (0 to 6 on the scale)
“Passives” (7 to 8 on the scale), or “Promoters” (9 to 10 on the scale)

Number of Promoters − Νumber of Distractors

NPS= ×100
Number of Respondents

© McGraw Hill, LLC

 Learning & Cognitive Outcomes

o Relate to familiarity with information, including principles, facts,

techniques, procedures, and processes
o Typically measured via paper-and-pencil tests and self-assessments
o Tests often preferred over self-assessments

© McGraw Hill, LLC

 Behavior & Skill-Based Outcomes

o Relate to proficiency with technical skills, motor skills, and behavior

o Include learning and transfer
o Learning often assessed via work samples
o Transfer often assessed via observation or managerial/peer ratings

© McGraw Hill, LLC

 Affective Outcomes

o Include attitudes and motivation

o For example, self-efficacy, employee engagement, motivation to
learn, tolerance for diversity, and attitudes toward safety
o The attitude of interest depends on the training objectives
o Affective outcomes often measured via surveys

© McGraw Hill, LLC

 Results

o Did training have an impact on meaningful business outcomes?

o Outcomes used to determine the benefits of training to the company
o For example, reduced costs, increased employee retention,
increased sales, improved quality, or customer service

© McGraw Hill, LLC

 Return-on-Investment

o Involves comparing the training program's benefits to its costs in

monetary terms
o Benefits: value the company gains
o Direct costs: salaries for employees involved in training, program
materials, facilities, and travel
o Indirect costs: costs not related directly to design and delivery

© McGraw Hill, LLC

 Which Outcomes?

Relevance Reliability

Discrimination Practicality

Which Outcomes? Long Description

© McGraw Hill, LLC

 Relevance

o The extent to which training outcomes are related to the learned

capabilities emphasized in the program
o Contamination refers to the inclusion of inappropriate or irrelevant
outcomes
o Deficiency refers to the omission of important information

© McGraw Hill, LLC

 Reliability

o The extent to which outcomes can be measured consistently over

time
o Evaluators are concerned with consistency over time, such that items
do not change in meaning or interpretation over time

© McGraw Hill, LLC

 Discrimination

o The extent to which measured performance reflects a true difference

o We want tests that discriminate between high and low performance
o A test that is too easy may not discriminate
o In this instance, both high and low performers would do well and
appear “good” even though they are not

© McGraw Hill, LLC

 Practicality

o The extent to which outcomes can be easily measured and collected

o Companies often claim that measurement is too burdensome

© McGraw Hill, LLC

 Training Evaluation Practices

Training Evaluation Practices Long Description

© McGraw Hill, LLC

 Which Outcomes?

o While collecting data on all outcomes is ideal, it may not always be

necessary
o Consider the scope of the training and practical considerations
o Do not assume that positive reactions lead to transfer

© McGraw Hill, LLC

 Treats to Validity

o Threats to validity refer to factors that will lead an evaluator to

question the results
o Internal validity is the believability of the study
o External validity is the generalizability of the evaluation results to other
groups and situations

© McGraw Hill, LLC

 Controlling for Threats

o There are three ways to control for threats:

o Use pre-tests and post-tests
o Use a control group
o Random assignment of employees to control and training groups

© McGraw Hill, LLC

 Post-Test Only

o Involves collecting only post-training outcomes

o Appropriate when trainees can be expected to have similar levels of
proficiency prior to training
o Strengthened with a control group

© McGraw Hill, LLC

 Pre-Test / Post-Test

o Involves collecting both pre-training and post-training outcomes to

determine if a change has occurred
o Is there a difference in outcomes before and after training?

© McGraw Hill, LLC

 Pre-Test / Post-Test with Comparison Group

o Includes pre-training and post-training outcomes and use of a control

group
o If the post-training improvement is greater for the training group, there
is evidence that training was responsible

© McGraw Hill, LLC

 Time Series

o Involves collecting measures at periodic intervals pre- and post-

training
o A comparison group may be used
o The strength of this design can be improved by using reversal, which
refers to a time period when participants no longer receive training

© McGraw Hill, LLC

 Solomon Four-Group

o Combines the pre-test/post-test comparison group design and the

posttest-only control group design
o Pre-test, treatment, post-test
o Pre-test, no treatment, post-test
o No pre-test, treatment, post-test
o No pre-test, no treatment, post-test

© McGraw Hill, LLC

 When is evaluation not needed?

o Time constraints
o Managers and trainees may lack expertise
o The company may view training as an investment from which it
expects little or no return

© McGraw Hill, LLC

 When is evaluation a must?

o The training is ongoing and has the potential to affect many

employees
o The training program involves multiple classes and a large number of
trainees
o The expertise exists to evaluate
o The cost of training is significant
o There is sufficient time and interest

© McGraw Hill, LLC

 Cost-Benefit Analysis

o Companies may desire to quantify whether the benefits of training

outweigh the costs
o Cost-benefit analysis
o Process of determining the economic impact of training using
accounting methods that look at training costs and benefits

© McGraw Hill, LLC

 Cost Categories

o program development or purchase

o instructional materials
o equipment and hardware
o facilities
o travel and lodging
o salary of the trainer and support staff
o cost of lost productivity or replacement workers while trainees are
away

© McGraw Hill, LLC

 Determining Benefits

o A number of methods can be used to identify benefits

o literature that summarizes benefits
o pilot training programs
o observing successful job performers
o asking trainees and managers for estimates

© McGraw Hill, LLC

 Calculating Return-on-Investment (ROI)

o Identify annual change in outcomes

o Place a monetary value on the outcomes
o Determine the annual change in value
o Determine training costs
o Calculate net benefit
o Divide net benefits by costs

© McGraw Hill, LLC

 Utility Analysis

o Assessing the dollar value of training based on:

o estimates of the difference in performance between trained and
untrained employees
o number of individuals trained
o length of time training is expected to influence performance
o variability in performance in the untrained group of employees

© McGraw Hill, LLC

 Success Cases & Return on Expectations

o Success cases refer to concrete examples showing how learning has

led to results the company finds worthwhile and credible
o Return on expectations (ROE) demonstrates to key stakeholders that
their expectations about training have been satisfied

© McGraw Hill, LLC

 Benchmarking Metrics

Learning hours Expenditure as a Expenditure as a

Cost per learning
received per percentage of percentage of
hour received
employee payroll revenue

Learning hours
Percentage of Average percentage Average percentage
received per training
expenditures for of learning activities of learning content
and development
external services outsourced by content area
staff member

Average percentage
Hours spent in Expenditure on
of learning hours
training related to reskilling or
provided via
reskilling or upskilling per
different delivery
upskilling employee
methods

Benchmarking Metrics Long Description

© McGraw Hill, LLC

 Big Data

o Big data refer to complex datasets compiled across different systems,

including marketing, sales, HR, finance, accounting, customer service,
and operations
o Three dimensions characterize big data
o volume
o variety
o velocity
o Big data can be used to:
o evaluate the effectiveness of programs
o determine their impact on business results
o develop predictive models for forecasting training needs, course
enrollments, and outcomes

© McGraw Hill, LLC