Banishing Evaluation Fears

15/04/2019 Banishing Evaluation Fears
Fear and lack of discipline can cost a

learning function respect, budget, jobs and,
in the end, its own existence. But this spiral
can be avoided.
BY JAMES D. KIRKPATRICK AND WENDY KAYSER

KIRKPATRICK
E
ighty percent of training professionals believe that evaluating
training results is important to their organization, according
to the Association for Talent Development’s 2016 research
report “Evaluating Learning: Getting to Measurements That
Matter.” However, only 35 percent are confident that their training
evaluation efforts meet organizational business goals.
The dramatic disparity between what learning professionals believe the

business wants and what they deliver has been a relatively invariable
dilemma for decades. Countless articles, white papers and programs 2
address this issue and provide solutions that range from simple to
complicated. So why does the problem still exist?
https://magazine.clomedia.com/issue/march-2019/banishing-evaluation-fears/ 1/12
Two causes are frequently cited: a lack of discipline surrounding

evaluation and a fear of evaluation among some professionals.
Lack of discipline in evaluation is most often seen in corporations. A

learning department receives an annual budget and uses it to deliver
pleasant, professional experiences. However, they often are not asked to
provide meaningful data to show how those experiences support the
business, so they don’t. This carries on for a while, but eventually
learning receives a summons to show how they contribute to the
business. Generally, this results in continual budget slashing and other
cost-cutting measures.
Fear and lack of discipline can cost a learning function

respect, budget, jobs and, in the end, its own existence.
But this spiral can be avoided.
BY JAMES D. KIRKPATRICK AND WENDY KAYSER

KIRKPATRICK
E
ighty percent of training professionals believe that evaluating
training results is important to their organization, according
to the Association for Talent Development’s 2016 research
report “Evaluating Learning: Getting to Measurements That
Matter.” However, only 35 percent are confident that their training
evaluation efforts meet organizational business goals.
The dramatic disparity between what learning professionals believe the

business wants and what they deliver has been a relatively invariable
dilemma for decades. Countless articles, white papers and programs
address this issue and provide solutions that range from simple to
complicated. So why does the problem still exist?
Two causes are frequently cited: a lack of discipline surrounding

2
evaluation and a fear of evaluation among some professionals.
Lack of discipline in evaluation is most often seen in corporations. A

learning department receives an annual budget and uses it to deliver
pleasant, professional experiences. However, they often are not asked to

provide meaningful data to show how those experiences support the
business, so they don’t. This carries on for a while, but eventually
learning receives a summons to show how they contribute to the
business. Generally, this results in continual budget slashing and other
cost-cutting measures.
Fear of evaluation is more often seen among learning professionals in

government and military organizations. On the positive side, these
organizations have strict written regulations or doctrine that require
value to be documented and reported. Therefore, these professionals are
hungry for ways to show the value. However, there is fear of what might
happen if value cannot be shown, so instead of evaluating how training
improves performance and contributes to agency mission
accomplishment, they select metrics that are easier to demonstrate. They
water down goals to things that are easier to control, such as presentation
of skill and recitation of knowledge at the end of training.
There is also reticence to evaluate the degree to which participants apply

what they learned in training when they are back on the job, or what we
refer to as behavior (level 3) in the Kirkpatrick Model (see figure on p.
41), a method for evaluating the effectiveness of learning solutions.
Common excuses to avoid it are the perceived difficulty and expense of
monitoring what happens on the job. While there are often logistical
realities to address, the real issue could be the fear that evaluation efforts
might reveal that the training alone has done little to improve
performance or related company results.
The key to showing the

organizational value of training is the 2
pre-training plan.
Lack of discipline to evaluate and fear of evaluating what is truly

important share a commonality: They both can cost the learning function
of an organization respect, budget, jobs and, in the end, its own existence.
But this death spiral can be avoided. A straightforward set of tactics to
use before, during and after training can make an immediate and
measurable difference in the value that learning delivers to an
organization.
The End Is the Beginning

When you receive a request for training, your goal is to discover and
understand the underlying problem that generated the request so you can
recommend an appropriate intervention that will solve it. The true
purpose of learning is to improve on-the-job performance and
measurably contribute to key organizational results (level 4), so you need
to find out what might be lacking in those areas.
Following are sample questions to ask the requestor or stakeholder to

discover their needs and design a program that will meet those needs and
deliver results:
What outcomes do you wish to see after this program? How do they
differ from what you are seeing today?
Are there key metrics that should be improved as a result of this
program?
What would make this program a success in your eyes?
Brandon Huff, learning and performance manager for Nationwide

Insurance, asks stakeholders, “What is an appropriate sample size from
the business to confirm that the program was a success?” and, “With
whom should we partner to gather the business results necessary to
review the program?”
Additionally, here are some questions to ask the requestor or stakeholder 2

to establish what success will look like in the behavior of participants.
You might also ask trusted line managers and supervisors these questions.
Can you describe what training graduates should actually do on the

job as a result of this program?
What would be considered “good performance,” and how does that

compare with what is occurring today?
What support and accountability resources are available after
training?
What will we need to do to ensure that training graduates do what
they are supposed to do after training?
Huff takes this opportunity to obtain buy-in for leadership participation

after the training by asking, “What commitments are you able to make as
it pertains to reinforcing the behaviors necessary to drive success in the
business outcomes?”
Typically, businesspeople will be most interested in data from levels 3

and 4 — employee and department performance and organizational
results. Focus your evaluation efforts on these areas.
You also probably will want to know a few things about the program
itself and how it is received by participants — levels 2 and 1, respectively.
Instead of using the same old post-program evaluation, list out the key
information that will be most useful to you and use it as a guide for the
evaluation you conduct and specific questions you ask.
Once you are clear on the overall program goal and the performance
expectations, you can start to design the intervention. Training will likely
only be one part of an effective program plan. As you build the content,
build the evaluation plan and related collection tools.
Formative Evaluation Saves Resources

A good evaluation plan takes advantage of formative evaluation, meaning
that which occurs during the program. Data related to learning and
reaction (levels 2 and 1) can be gathered during the program with
minimal time and effort, saving resources to support and evaluate the
more important studies of behavior and results (levels 3 and 4). 2
Collecting data during the program also reserves the post-program
evaluation form for only the most important data that will be utilized,
tracked and reported.
Sara Henderson, an Atlanta-based instructional designer, offers this list

of fun activities that are also effective level 2 formative evaluation
techniques:
Teachback: Assign students a topic and ask them to teach it to the

class.
Caption This: Present an image and ask learners to create a
caption. The images should depict scenarios where prior skills could
be applied.
Think-Pair-Share: Learners respond to a question prompt, then
discuss in pairs. They share results with the group.
Doodle It: Ask learners to draw a concept or skill instead of
explaining it. This is particularly good for virtual sessions, and it
usually gets lots of laughs.
Red Light/Green Light: Use a polling technique and ask a series of
yes/no or agree/disagree questions, and discuss learner responses.
This works for both in-person and virtual instructor-led sessions.
Examples of level 1 formative evaluation include:
An instructor asking during the program how things are going for
participants.
A question in an asynchronous online module asking participants to
document the degree to which they are engaged by the content.
A “ticket out” system in which participants are asked to comment on
how they might use what they are learning in class when they
return to work.
Focus Resources on Post-Training Support

Most of your evaluation resources should be focused on what happens
after the training, during the critical time when your graduates are
attempting to apply what they learned in their actual work. Just over half
of organizations evaluate behavior to any extent, according to ATD’s
“Evaluating Learning: Getting to Measurements That Matter” report, so
2
great improvement can and should be made in this area.
Evaluation is not synonymous with a post-program survey. While 74

percent of those who do any evaluation at all choose it as their main tool,
it can generally only gather one-dimensional data related to what is and is

not occurring. For important initiatives, create a plan that addresses all
four types of required drivers — or processes and systems that monitor,
reinforce, encourage and reward performance. When training builds and
helps to implement the driver package, it increases value in the eyes of
stakeholders and contributes meaningfully to organizational performance
and results.
Monitoring: How will you know that graduates are doing what they
learned? Find out if their supervisors are willing to monitor and report
on their performance. If not, consider building a peer-to-peer or a self-
monitoring and reporting system. Find a fun way to share this
information, such as a dashboard, to create friendly competition.
Reinforcing: How can you, supervisors and stakeholders communicate

that the outcome of this program is important? Find out if you can get a
message into the company newsletter, on a bulletin board or in an
intranet message. See if you can schedule email reminders for graduates.
If the content is complicated, add additional education, such as lunch-
and-learns or refresher modules.
Encouraging: Who might be able to help graduates and keep them

going if they get stuck? If supervisors are willing, provide talking points
for their regular employee touch-bases and team meetings. Additionally,
try to set up another method, such as a buddy system assigned during
training or peer mentor/mentee pairs.
Rewarding: For major initiatives, check if formal reward systems are in

2
line with what graduates are being asked to do on the job. For example, if
they do what they are supposed to do, will they get a good performance
appraisal and perhaps an annual pay increase? Also consider small,
informal methods of reward, such as a team lunch or casual Friday for the
department with the highest performance or results scores after training.

Praise from executives can also be a powerful reward.
Taking the First Step

The key to showing the organizational value of training is the pre-
training plan. If you wait until after training to consider how to show
that the training was successful, you might be too busy, it may be difficult
to gain the support you will need from others, and much of the data you
should have collected will already be gone.
Establish a standardized planning process for creating and updating

training programs. For major initiatives, create the evaluation plan and
build the tools and systems to gather the necessary data on all four levels
as you build the content. Have conversations with stakeholders during
the planning process to determine how much support you can expect,
and solicit support from other areas, if needed.
Overcome the fear of levels 3 and 4 with a pilot program. You will likely
find instances of sub-par application, but you may also find graduates
who are applying what they learned and seeing positive outcomes.
Sharing and acting on these truths will move initiatives forward and earn
you trust as a strategic business partner.
As John F. Kennedy said, “There are risks and costs to action. But they are

far less than the long range risks of comfortable inaction.”
James D. Kirkpatrick and Wendy Kayser Kirkpatrick are co-owners of Kirkpatrick Partners and co-
authors of “Kirkpatrick’s Four Levels of Training Evaluation.” They can be reached at
editor@CLOmedia.com.
BACK TO ISSUE 2
CHIEF LEARNING OFFICER

Banishing Evaluation Fears

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Banishing Evaluation Fears

Uploaded by

Copyright:

Available Formats

15/04/2019 Banishing Evaluation Fears

Fear and lack of discipline can cost a

BY JAMES D. KIRKPATRICK AND WENDY KAYSER

The dramatic disparity between what learning professionals believe the

Two causes are frequently cited: a lack of discipline surrounding

Lack of discipline in evaluation is most often seen in corporations. A

Fear and lack of discipline can cost a learning function

BY JAMES D. KIRKPATRICK AND WENDY KAYSER

The dramatic disparity between what learning professionals believe the

Two causes are frequently cited: a lack of discipline surrounding

Lack of discipline in evaluation is most often seen in corporations. A

pleasant, professional experiences. However, they often are not asked to

Fear of evaluation is more often seen among learning professionals in

There is also reticence to evaluate the degree to which participants apply

The key to showing the

Lack of discipline to evaluate and fear of evaluating what is truly

The End Is the Beginning

Following are sample questions to ask the requestor or stakeholder to

Brandon Huff, learning and performance manager for Nationwide

Additionally, here are some questions to ask the requestor or stakeholder 2

Can you describe what training graduates should actually do on the

What would be considered “good performance,” and how does that

Huff takes this opportunity to obtain buy-in for leadership participation

Typically, businesspeople will be most interested in data from levels 3

Formative Evaluation Saves Resources

Sara Henderson, an Atlanta-based instructional designer, offers this list

Teachback: Assign students a topic and ask them to teach it to the

Examples of level 1 formative evaluation include:

Focus Resources on Post-Training Support

Evaluation is not synonymous with a post-program survey. While 74

it can generally only gather one-dimensional data related to what is and is

Reinforcing: How can you, supervisors and stakeholders communicate

Encouraging: Who might be able to help graduates and keep them

Rewarding: For major initiatives, check if formal reward systems are in

department with the highest performance or results scores after training.

Taking the First Step

Establish a standardized planning process for creating and updating

As John F. Kennedy said, “There are risks and costs to action. But they are

CHIEF LEARNING OFFICER

You might also like