Training on Trial

Mid New Jersey Chapter, ASTD May 21, 2009
Presented by: Jim Kirkpatrick , PhD

The future of training
“Training directors might be well advised to take the initiative and evaluate their programs before the day of reckoning arrives”
- From Techniques for Evaluating Training Programs, by Don Kirkpatrick - ASTD Journal, November, - 1959

What are the charges?
“You, learning professionals, are charged with incurring training and development costs that exceed the value you bring to the business”

Who exactly is the “we” who are being charged?
The training industry Organizational Development Learning functions HRD Training departments Us

Who is charging us?
Internal business partners External clients The business world

How do we know we are being charged?
“We would like you to show us your ROI” “We know you run a lot of programs and seem quite busy. Why then are so many good people leaving us?” “We are thinking we need to put our efforts in revenue-producing activities.” “We are going to put that on hold for awhile.” “We have decided to make some cuts.”

What evidence is being used brought against us?

Brinkerhoff Study
Training Application

Did not try new skills: 15% Tried new skills and failed: 70% Achieved sustained new behaviors: 15%
- Josh Bersin and Associates, 2008

Percentage of learning transfer
1975: Percentage of formal learning that is actually applied to the job: 15% 2005: Percentage of formal learning that is actually applied to the job 15%
From Dana Robinson, ASTD ICE, 2008

Causes of “training failure”

Preparation and Readiness: 20% Learning Intervention: 10% Application Environment: 70% Study 2006 ASTD

Typical Learning Investment

Pr -W r 1 % e ok 0 L a n gE e t8 % e r in v n 5
Dr. Brent Peterson, Columbia

Are we guilty?

What do we have to do to be found not guilty?
We need to extend our role beyond the traditional definitions of training and learning

The increase in “informal learning”
Where Learning Takes Place

Prior to being "trained": 20% During "training": 10% On the job: -70% Bersin and Associates, 2008 Josh

Lets get very practical here
We need to redefine and expand our role through the entire “learning and performance” management spectrum:
– Our expertise – Our involvement – Our influence – Our impact – Our value

How does this relate to you so far?

Who will take our case?

Measurement and Evaluation
Gathering data and information to answer specific questions regarding the value of learning and performance solutions; focusing on the impact of individual programs and creating overall measures of system effectiveness, leveraging findings to provide recommendations for change and to increase organizational effectiveness. - taken from Mapping the Future, ASTD, 2004

Why Evaluate?
1. Improve the program (or know when to retire it) 2. Maximize training effectiveness by reinforcing knowledge, skills, and attitudes 3. Demonstrate the value of training

p. 3

Kirkpatrick Four Levels
Level 1: REACTION To what degree participants react favorably to the training

p. 4

Kirkpatrick Four Levels
Level 2: LEARNING To what degree participants acquire intended knowledge, skills, and attitudes based on participation in learning event

p. 4

Kirkpatrick Four Levels
Level 3: BEHAVIOR To what degree participants apply what they learned during training on the job

p. 4

Kirkpatrick Four Levels
Level 4: RESULTS To what degree targeted outcomes occur, as a result of the learning event(s) and subsequent reinforcement

p. 4

Four Level Correlations
Enjoyment (L1) and learning (L2): no correlations Relevance (L1) and immediate learning (L2): r = .26 Relevance (L1) and transfer to job (L3): r = .18 Immediate knowledge (L2) and retention (L2): r = .34 Immediate knowledge (L2) and skill demo (L2): r = .18 Immediate knowledge (L2) and transfer to job (L3): r = .11 Retained knowledge (L2) and transfer to job (L3): r = .08 Skill demo (L2) and transfer to job (L3): r = .18

Kirkpatrick Four Level Evaluation major principles
• Smile sheets, pre and post-tests, and hope for the best • Strong connection between Levels 1 and 2 • Strong connection between Levels 3 and 4 • “The Great Divide” • “The Missing Link”

Let’s take some initial depositions . . .

“What is your job here at the hotel?” “I am a window washer.”

“What is your job here at the resort?” “I am part of a team that creates great experiences for our guests!”

Group Activity
What is significant about these two window washers in relation to: Group 1: individual employees? Group 2: training and development professionals? Group 3: the training and development industry?

Crossing over is not easy

What does the word “charged” mean?
• Accused

• Challenged

Common Myth
“Evaluation” only occurs at the end of the instruction design process, or at the end of a program.

An effective evaluation plan is considered at every step in the program and training development process, not JUST at the end.

Kirkpatrick Four Level Evaluation major principles
• • • • • The end is the beginning ROE is the ultimate indicator of value Business partnership is key Value must be created first Demonstrate value through a compelling chain of evidence

Kirkpatrick Four Level Evaluation Model
Reaction Learning Behavior Results

The Kirkpatrick Model
Business need identified Negotiate stakeholder success indicators, Business outcomes

Identify critical behaviors and key organizational drivers

Determine required KSAs, Learning Objectives

Consider necessary learning environment and conditions

Design and build learning program and evaluation tools Deliver learning program Measure L2 Learning Measure L1 Reaction

Initiate ongoing reinforcement and monitoring Measure L4 Results Measure L3 Behavior

,t s u da, s gn dnf ezy an A j i i l yr ass ec en s a s pe s t aepe t r

Gather final four level data / information, prepare for presentation Present L4 Results findings Present L3 Behavior findings Present L2 Learning findings Present L1 Reaction findings


p. 22

DOT Case Example
Challenge – to increase highway work zone efficiency, effectiveness, and safety.

The Kirkpatrick Business Partnership Model
1. Identify who is on your corporate jury 2. Negotiate jury (stakeholder) expectations 3. Convert expectations to Level 4 outcomes and determine metrics 4a. Identify the key new participant behaviors that will be required to bring about the desired outcomes. If L3 behaviors are not relevant, choose L2 ongoing readiness factors 4b. Identify the few key organizational drivers that will be necessary to bring about the changes in behavior 5. (Determine and implement prerequisites)

DOT Example
1. Jury – safety managers, district engineers, traffic control, risk managers, maintenance supervisors. 2. Expectations – smooth traffic flow; increased safety for workers and motorists in highway work zones; better planning. 3. Success Outcomes – less than 15 minute delays; reduction in injuries and fatalities; increased compliment to complaint ratio. 4a. Key New Behaviors – written work zone plans; plan communicated to public; proper work zone set-up; flaggers following procedures. 4b. Success Drivers – observation, feedback and coaching by supervisors; ongoing compliance tracking; ongoing execution of formal and informal recognition programs.

Airborne Warning and Control System

Activity #2
1. Break up into small groups as instructed. 2. Discuss and select one high impact training program from your group members to work with. 3. Develop your answers to steps 1-4. Use flip chart pages. 4. Be prepared to share your results.

Measurement Strategy
Training -related Activitie s
Level 1
Did the target audience react favorably to the activity?

Demonstr ated SKA

Critical Behavio rs
Level 3
Did the target audience exhibit a sustained behavior change in the work environment?

Busines s Results
Level 4
Did the critical behaviors contribute to a positive impact on business results?

Level 2
Did the activities build sufficient capabilities to drive the critical behaviors?

Business Partnership



Business Results
HR Processe s

Critical Behavior s
Business Processe s Talent

Activitie s

Preparing for and presenting our case

Developing a Plan to Evaluate: Critical Criteria

Evaluation methods
Evaluation Levels Methods
Survey Questionnaire/Interview Focus Group Knowledge Test/Check Work Review Skills Observation Presentations / Teach Bk Action Planning Action Learning Key Business HR Metrics 1 Reaction ● ● ● 2 Learning ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 3 Behavior ● ● ● 4 Results ● ● ●

Kirkpatrick Four Level Evaluation
Feedback Loop to Eliminate ‘Snags’
Delivery of Learning Event
L1 & L2 Formative Evaluation


Level 1
Improve Session
4 Post Session L1 2 3


Was session delivered effectively? YES Post Session L2

Level 2


Improve Reinforcement and/or Coaching YES NO Was failure due to session?


Did targeted learning occur? YES Post Session Reinfcmt, Coaching & L3 Was there successful transfer of learning to behavior? YES Expected results will occur.
Build your chain of evidence & showcase



Level 3




Level 4

Example: Valdez University strategic goal scorecard – goal #1
Up from last month Down from last month Same as last month

Metric 1. Level 1 satisfaction scores - aggregate 2. % courses learning objectives matched to new directive 3. Level 2 skills scores aggregate 4. % leaders certified as trusted advisor coaches 5. Level 3 scores aggregate 6. Gallup scores - aggregate




93% 85% 92% 66% 48% 77%

90% 70% 90% 65% 75% 90%

Help to move our organization from transactionoriented to ‘trusted advisor’

Preparing for our closing arguments

Chain of Evidence

Level 1 Reaction

Level 2 Learning

Level 3 Behavior

Level 4 Results

Gather data at all four levels and show that your training delivers true value to your organization.

Data is a critical part of our Chain of Evidence

But do not underestimate the power of testimonials

What will you do as a result of what you have learned today . . .

that will lead you to the following tomorrow?

that will hopefully lead to . . .

• Kirkpatrick, D.L. and J.D. Evaluating Training Programs, 3rd Ed., Berrett-Koehler Publ., Inc. San Francisco, CA, 2006 • Kirkpatrick, D.L. and J.D. Transferring Learning to Behavior: Berrett-Koehler Publ., San Francisco, CA, 2005 • Kirkpatrick, D.L. and J.D. Implementing the Four Levels, Berrett-Koehler Publ., San Francisco, CA, 2007 • Kirkpatrick, J.K., and W.K. Training on Trial, AMACOM, New York, 2009 • Contact Jim at •