Training Best Practices:: Needs Assessment and Evaluation

Training Best Practices:
Needs Assessment and Evaluation
Presented
January 25, 2008
By
Kristi L. Thompson, M Ed
Sierra Associates
www.KristiThompson.com
Objectives
• Participants will be provided the information and tools

needed to conduct a basic Needs Assessment for
course development.
• Participants will be provided the information and tools

needed to conduct a Course Evaluation.
• Participants will be provided some tips on structuring

budget requests.
Sierra Associates 2008 2

Learning is provided in order
to improve performance
on the present job.
(Nadler, 1984)

What have you done today to enhance
(or at least insure against the decline of)
the relative overall useful-skill level
of your work force
vis-a-vis competitors?
- Tom Peters
Thriving on Chaos

To Train or Not
High
Problem: Low Motivation Problem: Systemic
Method
Assess personal Method
Job consequences/ rewards Consider system issues, problem
Knowledge system is out of control of the employee
Problem: Bad Fit Problem: Lack of Knowledge

or Tools
Method
Consider improper Method
placement of employee in Training
the position
Low

My Favorite Two

Training Approaches
Intervention Approach
Training is an intervention for solving problems
involving employees. Focus is on performance and/or
organizational results as corrections to problems.
Steps taken when performance issue is identified.
System Approach
Training is a part of a continuous improvement
process, fully integrated in the regular process of
organizational improvement. Some steps done only
once.

Instructional Design – Our Focus
Analyze
– Analyze system (department, job, etc.) to gain a complete
understanding of it
– Compile an inventory of all tasks associated with each job (if
needed)
– Select tasks that need to be trained
– Build performance measures for the tasks to be trained
– Choose instructional setting for the tasks to be trained
– Estimate what is going to cost to train the tasks
• Design
• Develop
• Implement
• Evaluate
– Review and evaluate during each phase (analyze, design, develop,
implement)
– Perform evaluations
– Revise training system based on evaluations
Full ISD model in Appendix
What is an Needs Assessment?
“Needs assessment is the systematic effort that we make

to gather opinions and ideas from a variety of sources on
performance problems or new systems and
technologies.”
Allison Rossett (1987)

Why do a Needs Assessment?
• To make sure we are applying the right solution to the

problem
• To identify what learning will be accomplished
• To identify what changes in behavior and performance

are expected
• To determine the expected economic costs and benefits

Steps of a Needs Assessment
• Conduct Task Analysis and compile a Task Inventory
• Perform a Gap Analysis
• Select which tasks will be addressed
• Determine performance measures for the trained task
• Select Training Method
• Estimate training costs

Task Analysis/Inventory
• What
• The breakdown of performance into detailed levels of specificity
• Statements of
what will be done
how it will be done
for what end result
Ex: Produce weekly summary Help Desk reports for all calls using a
provided Excel template to be delivered to IT Director on Fridays.
• Why
• To determine the operational components of a job, skill, goal, or
objective
• To describe what and how they are performed
• To describe the sequence and scope
Task Inventory
• When
• Whenever there are new processes or equipment, when job
performance is below standards, or when requests for changes to
current training or for new training are received

Task Inventory – Example
Department: IT Help Desk Date: July 7, 2007

Analyst: Barty Crouch Department Supervisor: Susan Malfoy
Job Title: Administrative Assistant Job Code: 0742
Task Number and Task:

0742-1 Types orders received by mail, telephone, or in person at a minimum rate of
45 WPM into a computer database under general supervision of the Help
Desk Manager to document calls.
0742-2 Receives and answers customer questions under close supervision of the
Help Desk Manager in order to provide good customer relations.
0742-3 Delivers triage and delivery information received by mail, telephone, or in
person under general supervision of the Help Desk Manager to fulfill
customer requests.
0742-4 Acts as liaison between customer and various departments under close
supervision of the Help Desk Manager in order to provide good customer
satisfaction.
0742-5 Post and maintains records in a computer database without supervision to
provide the department with historical records for statistical needs.
….
Gap Analysis  Performance Behaviors
• What
• Comparison of actual performance against new or existing
standards
• Why
• To identify the performance gap between what is actually done and
what is required or expected
• When
• An intervention is required
• New processes and/or procedures
• New equipment/hardware
• New Applications
• New technologies
• Change in staffing
• Reduction in productivity
• Governmental mandates
• Security breaches
• Routinely, as part of continuous improvement process

Gap Analysis Questions
• For each identified task:
• Determine current performance level on task
• Determine desired performance level
Task Current Performance Desired Performance

Level Level
Produce weekly Emp. often makes Reports are fully
summary Help Desk mistakes, missing info complete and accurate
reports for all calls approx 50% of the time every week
using a provided Excel
template to be Emp. repeatedly asks Emp. is able to
delivered to IT Director for assistance on produce report with no
on Fridays. producing report, assistance
particularly on
recalculating Pivot
tables
Tasks Selection
• What
• A determination of which tasks will be addressed in the training
• Directs course objectives
• Why
• To determine the scope and content of the training
• When
• An intervention is required
• New processes and/or procedures
• New equipment/hardware
• New Applications
• New technologies
• Change in staffing
• Reduction in productivity
• Governmental mandates
• Security breaches
• Routinely, as part of continuous improvement process

Tasks Selection
• Required
– The task/topic is required by law or for safety
• Risk
– There is a high risk if task is not done correctly
– The task is critical
• Complexity
– Task is difficult or complex
– Task is done frequently
– Task is time-consuming
– Task is critical to the performance of the role/project
• Team Considerations
– Task requires coordination with other staff or with other tasks
– Task is part of a collective set of tasks
• Performance
– Task is required for acceptable role performance
– Task distinguishes star performers
Task Selection

Task Selection - example
Task Current Desired Select - Criteria
Performance Performance
Level Level
Produce Emp. often Reports are fully Include – Performance
weekly makes mistakes, complete and
summary Help missing info accurate every
Desk reports approx 50% of week
for all calls the time
using a Emp. is able to
provided Excel Emp. repeatedly produce report
template to be asks for with no
delivered to IT assistance on assistance
Director on producing report,
Fridays. particularly on
recalculating
Pivot tables

Select Training Format
• What
• A determination of training delivery mode
• Why
• Identify the best tool for the job
• Determine training materials and format
• Begin to understand possible costs
• When
• For every training intervention or project
Can be done at the task level or just generally

Training Format Options
• One-on-one/small group coaching
• Boot Camp – small, very specialized audience
• Job Aid
• Classroom
• Vendor course
• OJT Training Format
• Mentoring flowchart
• CBT
• e-Learning
• Text-based
• Video
• Active Learning

Budget
• What do They want?
– The greatest needs addressed with the least cost
(ROI)
• Structure your requests by

– Projects supported (the bigger the better)
– Productivity increase expected (SWAG!)
– Number of employees to affected by a particular
course/product (one product, many people)
– Problems “solved” with metrics
Training Costs
Worksheet

No money, no time, no resources
Step Method
1. Identify performance Questionnaire/Survey

problems Focus group (see Lunchtime Assessment)
2. Select which tasks Use the most reported from the survey
3. Develop performance You decide based on time and resource limits

measures for course
4. Select method of Lecture/classroom

instruction/setting
5. Estimate costs/budget On location with internal SME/trainer

Lunchtime Assessment
1. Gather a possible audience (Pizza works well.) Ask each person to write
down their ten most important training needs in a particular area.
Emphasize “specific needs”. A good format is “I need to know how to…”
2. Capture the training needs on the white board or flip chart. Avoid duplicate
answers by questioning.
3. Use a weighted voting process to prioritize the training needs across the
group (sticky dots) to prioritize the list. Give half as many dots as there are
items. Tell needs assessment participants to place their dots on the chart to
vote for their priorities.
4. List the training needs in order of importance, with the number of points
assigned as votes determining priority, as determined by the sticky dot voting
process.
A Step Beyond
• Have each person mark their top two needs on their lists of ten most
important and then collect the lists.
• Share these Top Twos with the manager for personal development work.

Basic Data Gathering
TYPE OF SOURCES OF INFORMATION

INFO DOCUMENTATION
needed OBSERVATION INTERVIEW
Interview/survey Review employee

Observe in work
AUDIENCE audience or files or personnel
environment
supervisors records
Observe Interview expert Review job
audience or or other descriptions, policy
TASKS
expert performers statements, and
performing trouble reports
Observe expert Interview SMEs, Review product
or creators of policymakers, plans, specifications,
CONTENT
product/process marketers, or and marketing
managers guidelines
IT Training Needs Assessment?
IT Trait Process Modification

Obsolescence •Use vendor training for very
specialized skills/tasks
•Define supported tools/apps/skills and
limit training to these
Enormous amt of possible •Define supported tools/apps/skills and
subject matter: limit training to these
New products daily •For static info – develop documentation
New services daily •For active info – consider the fastest
CHANGES daily! Training method
Entire campus as possible •Identify audience groupings related to
audience with different skills/knowledge/performance needed.
needs/levels/application for •Tweak content/materials for each
same tasks audience.
•Train the Trainer – use department IT
Sierra Associates 2008
stars 25
The linkage of Assessment and Evaluation
From Pamphlet 350-70-6

http://www-tradoc.monroe.army.mil/tpubs/pams/p350-70-6.pdf

Evaluation Along the Way
• Phase 1 - Analysis phase
• Determining if it really is a training problem
• Make sure there isn’t already a solution available
• Phase 2 - Design phase

• Ensure the course content link directly to the objectives
• Phase 3 - Development phase

• Perform rapid-prototyping (implement-evaluate-implement-
evaluate-etc) or
• Have another training specialist check the solutions
• Phase 4 – Implementation phase

• Use Level 1 evaluations (next slides)

Four Levels of Evaluation
• Reaction
• Learning
• Behavior or performance
• Outcomes or results

Level One - Reaction
• Measures and evaluates the learner’s reactions and
opinions about training program itself
• Does not measure what new skills the learners have

acquired or what they have learned that will transfer
back to the working environment.
• Method - Attitude questionnaires after classes
COWC Level One Level One

Questionaire

Level One - Questions to Ask
• Rate the instructor
• Rate the course
• Estimate how much you have learned in the course
• Rate the effectiveness of the instructor in stimulating
your interest in the subject
• Rate the effectiveness of this course in challenging you
intellectually
Minimally
– What major conclusion did you draw from today's
session?
– What major questions remain in your mind?

Level Two - Learning
• Measures and evaluates the changes in the participants’
skills, knowledge, or attitudes as a result of the training:
did they learn anything?
• Methods – Pre- and post-testing

• Questions should be directly linked to the selected
tasks and the course content

Level Three - Performance (behavior)
• Measures and evaluates the transfer of the learning to
the job or organization: can and do people use their
newly acquired learning on the job?
• Methods –
• Formal (testing or Manager interview)
• Informal (observation)
• Must take place after the learners have returned to
their jobs
Manager Level 3 Employee Level 3

Evaluation Evaluation

Level Four - Results
• Measures and evaluates the impact of the training on the
productivity and profitability of the organization: what "return" has
the organization received from the training?
• Methods –
– Develop a "balanced scorecard“ with four perspectives
• Financial
• Customer Satisfaction
• Internal Perspective
• Innovation and Learning
– Linked to Vision/Mission statement on these aspects
– Allow time for the behavior change to take place
– Evaluate both before and after the program

Balanced Score Card
• Financial
• A measurement, such as an ROI, that shows a monetary
return, or the impact itself, such as how the output is affected
• (Can be either soft or hard results.)
• Customer
• Improving an area in which the organization differentiates
itself from competitors to attract, retain, and deepen
relationships with its targeted customers
• Internal
• Achieve excellence by improving such processes as supply-
chain management, production process, or support process
• Innovation and Learning
• Ensuring the learning package supports a climate for
organizational change, innovation, and the growth of
individuals
VinFin Balanced
Scorecard example
Big Picture
•

References
• Rossett, Allison, (1987). Training Needs Assessment (Techniques in
Training and Performance Development Series. Educational Technology
Pubns (August 1987)
• Kirkpatrick, Donald, (1994). Evaluating Training Programs – The Four

Levels. San Francisco, CA: Berrett-Koehler Publishers, Inc.
• Multimedia Courseware Development Guide

http://www.tradoc.army.mil/tpubs/pams/p350-70-2.doc
• Writing a learning objective standard that measures Learning Objective (LO)

performance.
http://www.asat.army.mil/briefings/Learning%20objective%20standard
%20writing.doc
• Big Dog's Performance Coaching Page

http://nwlink.com/~donclark/perform/coach.html
• Design Training Needs Assessment Surveys -

http://adulted.about.com/od/trngneedsasst/a/TNAsurveys.htm

Resources
• First Things Fast: A Handbook for Performance Analysis by Allison Rossett (Nov
6, 1998)
• Job Aids and Performance Support: Moving From Knowledge in the Classroom
to Knowledge Everywhere (Essential Knowledge Resource) by Allison Rossett
and Lisa Schafer (Nov 3, 2006)
• A Handbook of Job Aids by Allison Rossett and Jeannette Gautier-Downes (Jun 15,
1991)
• Beyond the Podium: Delivering Training and Performance to a Digital World by

Allison Rossett and Kendra Sheldon (May 23, 2001)
• Article – What is a Balanced Scorecard
http://www.balancedscorecard.org/BSCResources/AbouttheBalancedScorecard/tabid/
55/Default.aspx
Digital Tools Survey

Training
Requirements

• Kristi Thompson is the Principal of Sierra Associates. She has over 20 years of experience in
the development of stronger work environments. Kristi works with groups on workplace and
relationship processes to redirect the focus and perspective, resulting in higher morale, focus
and productivity. She has applied her expertise in the high-performance manufacturing, service
industry, hi-tech companies, academic institutions and the medical field. She has presented on
numerous topics such as Change, Conflict, Team Development and Quality Improvement.
• Kristi received her Masters of Education from the University of Massachusetts Amherst and her
Bachelors of Science in Chemistry from Worcester Polytechnic Institute.
• Sierra Associates
• 8 Munn Road
• Monson, MA 01057
• 413 267 3585
• Cell: 508 789 4112
• www.KristiThompson.com
• sierra@samnet.net

Data gathering methods 1
Method Pros Cons
Questionnaire/ –Can customize per audience –% return is often less than 20%
Survey population –Can be anonymous –Takes time and skill to develop well
–Can and should ask different –Directed/limited answers
style questions –Should be limited to 10 minute
–Efficient in collecting lots of completion time
data –What people tell you and what they
–The easiest way of collecting really do may differ
needs assessment data. –Can’t probe deeper
Documentation -Provides management/HR –May be difficult or impossible to
review (i.e., job perspective obtain desired documents
descriptions, –May require interpretation/
performance explanation
appraisals)
Interview –Good for complex or Time and resource intensive

managers/ undefined problems/areas
supervisors –Can be modified “on the fly”
–Personal touch

Data gathering methods 2
Method Pros Cons
Observations –The most direct method of –Very resource consuming (time
collecting needs assessment and people resources)
data
Review –Time efficient –Dependent upon the quality of
performance documentation and honesty in
appraisals of documents
lesser –Best used in conjunction with
performers conversations with Mgr
Focus groups –Good for complex or –Scheduling issues
undefined problems/areas –Wide range of participants in
–Can be modified “on the fly” attendance
–Personal touch –what people tell you and what
–Ideas can build on each they really do may differ
other –Sometimes important to verify the
–Relatively effective and results with observations and
efficient document analysis

Steps for a full Analysis
Step Method Toolbox Template
1. Understand the • Interview manager/department System Overview

client’s business head Instrument
(Analyze system) System Overview
example
2. Conduct a Task • See next slides Task Inventory

Inventory*
Task Inventory
3. Select which tasks • Discuss with Task Selection

will be addressed manager/department
Task Selection
head/client
4. Build performance • Review performance appraisals Task Performance
measures for the • Interview manager/department Measure
trained tasks head
• Interview manager/department
Task Performance
head Template example
•Ask HR/EAP
Steps for a Needs Assessment
5. Select Lockstep Training

method of Classroom (In-house or Vendor) Method
instruction/ Boot Camp Decision Flow
instruction
setting Video
Self-Paced
Personalized System of Instruction (PSI)
Computer Based Training (CBT)
Text Instruction
e-learning or Internet Distance Learning
(IDL)
Job
Job Performance Aid (JPA)
On-The-Job (OJT)
6. Estimate Training Cost
costs Worksheet

Training Best Practices:: Needs Assessment and Evaluation

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Training Best Practices:: Needs Assessment and Evaluation

Uploaded by

Copyright:

Available Formats

Training Best Practices:

Needs Assessment and Evaluation

• Participants will be provided the information and tools

• Participants will be provided the information and tools

• Participants will be provided some tips on structuring

Sierra Associates 2008 2

Sierra Associates 2008 3

Sierra Associates 2008 4

Problem: Low Motivation Problem: Systemic

Problem: Bad Fit Problem: Lack of Knowledge

Sierra Associates 2008 5

Sierra Associates 2008 6

Sierra Associates 2008 7

“Needs assessment is the systematic effort that we make

Allison Rossett (1987)

Sierra Associates 2008 9

• To make sure we are applying the right solution to the

• To identify what learning will be accomplished

• To identify what changes in behavior and performance

• To determine the expected economic costs and benefits

Sierra Associates 2008 10

• Conduct Task Analysis and compile a Task Inventory

• Perform a Gap Analysis

• Select which tasks will be addressed

• Determine performance measures for the trained task

• Select Training Method

• Estimate training costs

Sierra Associates 2008 11

Sierra Associates 2008 12

Department: IT Help Desk Date: July 7, 2007

Task Number and Task:

Sierra Associates 2008 14

Task Current Performance Desired Performance

Sierra Associates 2008 16

Sierra Associates 2008 17

Sierra Associates 2008 18

Can be done at the task level or just generally

Sierra Associates 2008 19

Sierra Associates 2008 20

• Structure your requests by

Sierra Associates 2008 21

1. Identify performance Questionnaire/Survey

3. Develop performance You decide based on time and resource limits

4. Select method of Lecture/classroom

5. Estimate costs/budget On location with internal SME/trainer

Sierra Associates 2008 22

Sierra Associates 2008 23

TYPE OF SOURCES OF INFORMATION

Interview/survey Review employee

IT Trait Process Modification

From Pamphlet 350-70-6

Sierra Associates 2008 26

• Phase 2 - Design phase

• Phase 3 - Development phase

• Phase 4 – Implementation phase

Sierra Associates 2008 27

Sierra Associates 2008 28

• Does not measure what new skills the learners have

• Method - Attitude questionnaires after classes

COWC Level One Level One

Sierra Associates 2008 29

Sierra Associates 2008 30

• Methods – Pre- and post-testing