You are on page 1of 67

Measuring

ROI of
Training

1
Agenda

1. Measuring the Effectiveness of Training


Program

2. Measuring ‘Return on Investment’ of Training

3. Enhancing the Effectiveness and ROI of


Training

2
Measuring the Effectiveness of
Training Program

3
Training Process

Training Training Training


Training
Need Objectives Evaluation
Delivery
Analysis

What are Objective Techniques Measure


the training should be include on- reaction,
needs for measurable the-job- learning,
this person and training, behavior,
and/or job? observable action and results
learning, etc.

4
The Four Levels of Evaluation

Level 1 - Reaction

Level 2 - Learning
Four Levels
of Training
Level 3 – Behavior Effectiveness
Application

Level 4 – Business
Impact
5
The Four Levels of Evaluation

Evaluate trainees’ reactions to Level 1 -


the program. Did they like the Reaction
program? Did they think it
worthwhile?

Test the trainees to determine if Level 2 -


they learned the principles, Learning
skills, and facts they were to
learn.

6
The Four Levels of Evaluation

Ask whether the trainees’ behavior on the job Level 3 –


changed because of the training program. For Behavior
example, are employees in the store’s complaint Application
department more courteous toward disgruntled
customers than previously?

What final results were achieved in terms of the Level 4 –


training objectives previously set? Did the number Business
of customer complaints about employee drop? Did Impact
the reject rate improve? Was turnover reduced,
and so forth.

7
The Four Levels of Evaluation

Value of Frequency of Difficulty of


Information Use Assessment
Level
Least Frequent Easy
valuable
I. Reaction

II. Learning

III. Behavior
Most Infrequent Difficult
IV. Results valuable

8
Level 1 - Reaction

Evaluate trainees’
reactions to the program:
Did they like the program?
Level 1 - Did they like the
Reaction
facilitators?
Did they like the training
accommodation and
facilities?

9
Guidelines for Evaluating Reaction

1. Determine what you want to find out


2. Design a form that will quantify reactions
3. Encourage written comments and suggestions
4. Get 100 percent immediate response
5. Get honest response
6. Develop acceptable standards
7. Measure reactions against standards, and take
appropriate action
8. Communicate reaction as appropriate
10
Please give us your frank reactions and comments. They will help us to evaluate
this program and improve future programs.

Sample of Program :

Reaction Facilitator :

Form 1. How do you rate the subject (interest, benefit, etc)


a. Excellent
b. Very Good
c. Good
d. Fair
e. Poor

2. How do you rate the facilitator? (knowledge, ability to deliver and communicate?)
a. Excellent
b. Very Good
c. Good
d. Fair
e. Poor

3. How do you rate the facilities? (comfort, convinience, etc)


a. Excellent
b. Very Good
c. Good
d. Fair
e. Poor

4. What would have improved the program?


…………………………………………………………………………………
…………………………………………………………………………………
…………………………………………………………………………………

11
Level 2 - Learning

Measuring learning means


determining one or more of
the following :
Level 2 - • What knowledge was
Learning learned?
• What skills were
developed or improved?
• What attitudes were
changed?

12
Guidelines for Evaluating Learning

1. Use a control group if practical


2. Evaluate knowledge, skills and/or attitudes both
before and after the program
3. Use a paper-and-pencil test to measure knowledge
4. Use a performance test to measure skills
5. Get 100 percent immediate response
6. Use the results of evaluation to take appropriate
action

13
Guidelines for Evaluating Learning

Pretest and Posttest Scores on Change Management Training


Example :

Experimental Group Control Group


Pre Test Score 45 46
Post Test Score 55 48
Gain 10 2

14
Level 3 – Behavior Application

• The frequency of
application of new
skills/knowledge/
Level 3 – attitudes (on the job)
Behavior
Application • The effectiveness of the
skills/knowledge/
attitudes (as applied on
the job)

15
Guidelines for Evaluating Learning

1. Use a control group if practical


2. Allow time for behavior change and application to take
place
3. Evaluate both before and after the program if practical
4. Survey and/or interview one or more of the following :
trainees, their immediate supervisor, their subordinates,
and others who often observe their behavior
5. Get 100 percent response or a sampling
6. Repeat the evaluation at appropriate times
7. Consider cost versus benefits

16
Example of Survey to Measure Behavior Application
Instruction:
The objective of this questionnaire is to determine the extent to which those who
attended the recent program on Leadership have applied the principles and techniques
that they learned there to the job.

Circle the answer that you consider appropriate for each question.
5 = Much more 4 = More 3 = Same 2 = Less 1 = Much less

Time and energy spent after the prgram


compared to time and energy spent before
the program
Understanding and Motivating
1. Trying to understand my subordinates 5 4 3 2 1
2. Listening to my subordinates 5 4 3 2 1
3. Praising good work 5 4 3 2 1
4. Talking with subordinates about
5 4 3 2 1
their family and personal interests
5. Asking my subordinates for their ideas 5 4 3 2 1
6. Applying "Management by Walking Around" 5 4 3 2 1

17
Level 4 – Business Results

• Indicate the extent to which you


think this program has
influenced each of these
measures in your work unit,
Level 4 – department, or business unit:
Business • Productivity
Results • Quality
• Customer Response Time
• Cost Control
• Employee Satisfaction
• Customer Satisfaction
• Other

18
Guidelines for Evaluating Learning

1. Use a control group if practical


2. Allow time for results to be achieved
3. Measure both before and after the program if practical
4. Repeat the evaluation at appropriate times
5. Consider cost versus benefits
6. Be satisfied with evidence if proof is not possible

19
Performance Indicators

HARD DATA INDICATORS

• Downtime duration
• Number of defect products
• Sales volume
• Production unit
• Customer satisfaction index
• Response time to orders
• Number of accidents at work
• Others

20
Performance Indicators

SOFT DATA INDICATORS (intangible impacts)

• Job satisfaction
• Conducive working relationship
• Effective communication
• Stress rate
• Quality in decision-making

21
Example : Measuring Training Results

200.00
Program : TQM
Training

Results after 3 150.00


120 units
months of
training, number
100.00 80 units
of defects
dropped to 80
units/day
50.00

0.00

Before training After training

22
Example : Measuring Training Results

Program : 50.00

Sales Training
40.00
Results after 3
months training,
training 30 units

number of sales 30.00

per salesman 20 units


increase to 30 20.00

units/month.
10.00

0.00

Before training After training

23
Measuring
Return on Investment of
Training

24
Level 5 : Return on Investment of Training

Level 1 - Reaction

Level 2 - Learning

Level 3 – Behavior
Application
Level 4 –
Business Impact

Level 5 – Return
on Investment of
Training

25
Criteria for Selecting Programs for Levels
4 and 5 Evaluation

• Importance of the program in meeting the


organization’s goals

• Cost of the program

• Visibility of the program

• Size of the target audience

• Extent of management interest

26
Benefits of ROI of Training

• Measure contribution

• Set priorities

• Focus on results

• Alter management perceptions of training

27
ROI of Training Model

Isolate the Convert Data


Collect
Data Effects of to Monetary
Training Values

Identify
Intangible Calculate ROI
Benefits of Training

Tabulate
Program
Costs

28
Return on Investment Formula
Net Program Benefits
ROI = X 100
Program Costs

Example :

• Costs per program (25 participants) $ 88,500

• Benefits per program (1st year) $230,625

$ 230,625 – 88,500
ROI = X 100
$ 88,500

ROI = 161 %
29
Collecting Data

• Identify appropriate
Collect performance indicators
Data
• Develop a collection plan

30
Example of Performance Indicators

Output Time

• Units produced • Equipment downtime


• Items sold • Overtime
• Work backlog • Time to project
• New accounts opened completion
• Productivity • Processing time
• Inventory turnover • Repair time
• Etc. • Lost time days
• Etc

31
Example of Hard Indicators

Cost Quality

• Unit costs • Scrap


• Variable costs • Waste
• Overhead costs • Rejects
• Operating costs • Error rates
• Number of cost • Rework
reduction • Product defects
• Etc. • Product failure

32
Example of Performance Results

• Some performance results after training


program:
• Scrap was reduced from 11 % to 7.4 %
• Absenteeism was reduce from 7 % to 3.25 %
• The annual turnover rate was reduced from 30
% to 16 %
• Lost time accidents were reduced 95 %

33
Isolating the Effects of Training

Using
Control Group

Trend
Methods to Lines
Isolate the
Effects of Participants
Training Estimate

Supervisors of
Participants
Estimate
34
Isolating the Effects of Training

Using • A control group arrangement can be


Control used to isolate training impact.
Group • With this strategy, one group
receives training, while another,
similar group does not receive
training.
• The difference in the performance of
the two groups is attributed to the
training program.

35
Isolating the Effects of Training

Trend • Trend lines are used to project the


Lines values of specific output variables if
training had not been undertaken.
• The projection is compared to the
actual data after training, and the
difference represents the estimate
impact of training.

36
Trend Lines Analysis

Actual sales performance


At the beginning of
The
May, a Sales training difference
Volume of Sales

Program session was represents


held the estimate
impact of
training.

Trend Projection

Jan Feb Mar Apr May Jun Jul Aug

37
Isolating the Effects of Training

• This method rests on the assumption that


Participants and
participants (and their supervisors) are
Supervisors of
capable of estimating how much a
Participants
Estimate of performance improvement is related to the
Training’s Impact training program.
• Because their actions have produced the
improvement, participants (and their
supervisors) may have very accurate input
on the issue.
• They should know how much of the
change was caused by applying what they
have learned in the program.
38
Isolating the Effects of Training

• Typical Questions to Estimate :


Participants and
Supervisors of • What percent this improvement
Participants can be attributed to the application
Estimate of
of skills/techniques/knowledge
Training’s Impact
gained in the training program?
• What confidence do you have in
this estimate, expresses as a
percent?
• What other factors contributed to
this improvement in performance?

39
Isolating the Effects of Training

Example of a Participant’s Estimation


Percent Confidence
Factors Which Influenced Improvement Expressed as
No. Improvement Caused by a Percent
1 Training Program 50% 70%
2 Change in Procedures 10% 80%
3 Adjustment in Standards 10% 50%
4 Revision to Incentive Plan 20% 90%
5 Increased Management Attention 10% 50%
6 Other - -

The confidence percentage is multiplied by the


estimate (50 % x 70 %) to produce a usable training
factor value of 35 %
40
Isolating the Effects of Training
Example of a Participant’s Estimation
Percent Confidence
Factors Which Influenced Improvement Expressed as
No. Improvement Caused by a Percent
1 Training Program 50% 70%
2 Change in Procedures 10% 80%
3 Adjustment in Standards 10% 50%
4 Revision to Incentive Plan 20% 90%
5 Increased Management Attention 10% 50%
6 Other - -

• The confidence percentage is multiplied by the estimate (50 % x 70 %) to


produce a usable training factor value of 35 %
• This adjusted percentage is then multiplied by the actual amount of
improvement (post-program minus pre-program value) to isolate the portion
attributed to training
• The adjusted improvement is now ready for conversion to monetary values,
and used in the return on investment
41
Converting Data to Monetary Values

• Steps to Convert Data to Monetary


Converting
Values
Data to
Monetary 2. Focus on a unit of improvement
Values 3. Determine a value of each unit

4. Calculate the change in performance


data

5. Determine an annual amount of change

6. Calculate the annual value of


improvement

42
Steps to Convert Data to Monetary Values

An example to illustrate the steps to convert data to monetary values

Steps Illustration

1. Focus on unit • One grievance reaching step two in the


improvement four-step grievance resolution process

2. Determine a • Using internal experts, the cost of an


value of each average grievance was estimated to be $
unit 6,500 when considering time and direct
costs (V = $ 6,500)

43
Steps to Convert Data to Monetary Values

Steps Illustration

3. Calculate the • Six months after the program was completed,


change in total grievances per month reaching step two
performance declined by ten.
data • Seven of the then grievance reductions were
related to the program as determined by
supervisors (isolating the effects of training)

4. Determine an • Using the six month value, seven per month


annual amount (grievance reductions), yields an annual
for the change improvement of 84 (7 x 12 months)

44
Steps to Convert Data to Monetary Values

Steps Illustration

5. Calculate the • Annual value = 84 x $ 6,500 = $ 546,000


annual value of
improvement

45
Tabulating Cost of the Program

Tabulating • Tabulating the costs involves


cost of the monitoring or developing all of the
program related costs of the program
targeted for the ROI calculation.

46
Tabulating Cost of the Program

• Cost components that should be


Tabulating included are :
cost of the
program • The cost to design and develop
the program
• The cost of all program materials
provided to each participant
• The cost for facilitator
• The cost of the facilities of the
training program

47
Tabulating Cost of the Program

• Cost components that should be


Tabulating
included are :
cost of the
program • Travel, lodging, and meal costs
for the participants
• Salaries, plus employee benefits
of the participants who attend
the training
• Administrative and overhead
costs of the training function,
allocated in some convenient
way
48
An Example to Illustrate ROI analysis

Average Weekly Sales


Post Training Data
Weeks after
training Trained Group Control Group
1 US$ 9,723 9,698
2 9,978 9,720
3 10,424 9,812
13 13,690 11,572
14 11,491 9,683
15 11,044 10,092
Average for weeks 12,075 10,449
13, 14, 15

49
An Example to Illustrate ROI analysis

Annualized Program Benefits


Data US$
Average weekly sales
Trained Group 12,075
Control Group 10,449
Increase 1,626
Profit Contribution from Training 2 % 32.50
(Training Impacts)
Total weekly improvement 1,495
(32.5 x 46 participants)
Total annual benefit 71,760
($ 1,495 x 48 weeks)
Note : 46 participants were still in job after 3 months
50
An Example to Illustrate ROI analysis

Cost Summary : An Illustration


Cost US$ ROI (%):
Facilitation fees 11,250
Program materials : $ 35 x 46 1,610 $ 71,760 - $ 29,090
Meals : 3 days x $ 28 x 46 1,288
= x 100
$ 29,090
Participant salaries plus benefit (35 %) 12,442
Coordination and Evaluation 2,500 = 146 %
Total Cost 29,090

51
Enhancing Training
Effectiveness

52
How Effective is Your Training Program?

Broad and Newstrom (1992) report studies have


shown less than 30% of what is actually taught
transfers to the job in a way that enhances
performance.
performance
Source : Broad, M., & Newstrom, J. W. (1992). Transfer of training: Action packed
strategies to ensure high payoff from training investments. Reading, MA: Addison-
Wesley.

53
Source of Barriers to Training Transfer

• Lack of reinforcement on the job

• Interference from immediate (work) environment

• Nonsupportive organizational culture

• Trainees’ perception of impractical training programs

• Separation from the inspiration or support of the trainer

54
The Transfer Partnership

Trainee Trainee recognizes need for


new skills

Trainer designs and/or


Trainee Trainer
delivers learning experiences

Manager supports learning


Trainee Trainer
and application on the job
Manager

55
The Transfer Matrix

Time Periods
Before During After

Manager
Role Players

Trainer

Trainee

56
Before Training

• Build transfer of training into supervisory


Manager performance standards
• Collect baseline performance data
• Involve supervisors and trainees in needs
analysis process
• Involve trainees in program planning
• Brief trainees on the importance of the training
(course objective, content, process, and
application on the job)
• Review instructional content and materials
• Plan to participate in training sessions
• Encourage trainees attendance at all sessions
57
Before Training

• Align the training plan with the organization's


Trainer strategic plan
• Systematically design instruction
• Provide proactive opportunities
• Design a peer coaching component for the
program and its follow-up activities

58
Before Training

• Provide input into program planning


Trainee • Actively explore training options
• Participate in advance activities

59
During Training

• Prevent interruptions
Manager • Transfer work assignment to others
• Monitor attendance and attention to training
• Recognize trainee participation
• Participate in transfer action planning
• Review information on employee in training
• Plan assessment of transfer of new skills to the
job

60
During Training

• Develop application-oriented objectives


Trainer • Answer the “WIIFM” question
• Manage the unlearning process
• Provide realistic work-related tasks
• Give individualized feedback
• Provide job performance aid

61
During Training

• Maintain an ideas and application notebook


Trainee • Participate actively
• Form support groups
• Plan for applications
• Create behavioral contracts

62
After Training

• Plan trainees’ reentry


Manager • Provide opportunities to practice new skills
• Have trainees participate in transfer-related
decisions
• Reduce job pressures initially
• Give positive reinforcement
• Schedule trainee briefings for co-workers
• Set mutual expectations for improvement
• Arrange proactive (refresher) sessions
• Provide and support the use of job aids

63
After Training

• Provide follow-up support


Trainer • Conduct evaluation surveys and provide
feedback
• Develop and administer recognition system
• Provide refresher/problem-solving sessions

64
After Training

• Practice self-management
Trainee • Review training content and learned skills
• Develop a mentoring relationship
• Maintain contact with training ‘buddies’

65
Recommended Further Readings

• Donald Kirkpatrick, Evaluating Training Programs : The Four Levels,


Berrett-Koehler Publishers

• Jack J. Phillips and Patricia Phillips, In Action : Measuring Return On


Investment, American Society for Training & Development

• Mary Broad and John Newstrom, Transfer of training: Action Packed


Strategies to Ensure High Payoff from Training Investments, Addison-
Wesley.

66
End of Material

67