1-16 Training Best Practices | Needs Assessment | Educational Technology

Training Best Practices

:
Needs Assessment and Evaluation

Presented January 25, 2008 By Kristi L. Thompson, M Ed

Sierra Associates www.KristiThompson.com

Objectives

‡ Participants will be provided the information and tools needed to conduct a basic Needs Assessment for course development. ‡ Participants will be provided the information and tools needed to conduct a Course Evaluation. ‡ Participants will be provided some tips on structuring budget requests.

Sierra Associates 2008

2

Learning is provided in order to improve performance on the present job. (Nadler, 1984)

Sierra Associates 2008

3

What have you done today to enhance (or at least insure against the decline of) the relative overall useful-skill level of your work force vis-a-vis competitors? - Tom Peters Thriving on Chaos

Sierra Associates 2008

4

To Train or Not High Problem: Low Motivation Method Assess personal consequences/ rewards system Problem: Systemic Method Consider system issues. problem is out of control of the employee Job Knowledge Problem: Bad Fit Method Consider improper placement of employee in the position Low Problem: Lack of Knowledge or Tools Method Training High Low Employee attitude/desire to perform the job Sierra Associates 2008 5 .

My Favorite Two Sierra Associates 2008 6 .

fully integrated in the regular process of organizational improvement. System Approach Training is a part of a continuous improvement process. Sierra Associates 2008 7 .Training Approaches Intervention Approach Training is an intervention for solving problems involving employees. Steps taken when performance issue is identified. Focus is on performance and/or organizational results as corrections to problems. Some steps done only once.

implement) ± Perform evaluations Full ISD model in Appendix ± Revise training system based on evaluations Sierra Associates 2008 8 . job. design.) to gain a complete understanding of it ± Compile an inventory of all tasks associated with each job (if needed) ± Select tasks that need to be trained ± Build performance measures for the tasks to be trained ± Choose instructional setting for the tasks to be trained ± Estimate what is going to cost to train the tasks ‡ Design ‡ Develop ‡ Implement ‡ Evaluate ± Review and evaluate during each phase (analyze.Instructional Design ± Our Focus Analyze ± Analyze system (department. develop. etc.

´ Allison Rossett (1987) Sierra Associates 2008 9 .What is an Needs Assessment? ³Needs assessment is the systematic effort that we make to gather opinions and ideas from a variety of sources on performance problems or new systems and technologies.

Why do a Needs Assessment? ‡ To make sure we are applying the right solution to the problem ‡ To identify what learning will be accomplished ‡ To identify what changes in behavior and performance are expected ‡ To determine the expected economic costs and benefits Sierra Associates 2008 10 .

Steps of a Needs Assessment ‡ Conduct Task Analysis and compile a Task Inventory ‡ Perform a Gap Analysis ‡ Select which tasks will be addressed ‡ Determine performance measures for the trained task ‡ Select Training Method ‡ Estimate training costs Sierra Associates 2008 11 .

Task Analysis/Inventory ‡ What ‡ The breakdown of performance into detailed levels of specificity ‡ Statements of what will be done how it will be done for what end result Ex: Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays. when job performance is below standards. skill. goal. ‡ Why ‡ To determine the operational components of a job. or objective ‡ To describe what and how they are performed ‡ To describe the sequence and scope Task Inventory ‡ When ‡ Whenever there are new processes or equipment. or when requests for changes to current training or for new training are received 12 Sierra Associates 2008 .

Sierra Associates 2008 13 . or in person under general supervision of the Help Desk Manager to fulfill customer requests. telephone. or in person at a minimum rate of 45 WPM into a computer database under general supervision of the Help Desk Manager to document calls. 0742-2 Receives and answers customer questions under close supervision of the Help Desk Manager in order to provide good customer relations. 0742-5 Post and maintains records in a computer database without supervision to provide the department with historical records for statistical needs. «. 0742-3 Delivers triage and delivery information received by mail.Task Inventory ± Example Department: IT Help Desk Analyst: Barty Crouch Job Title: Administrative Assistant Date: July 7. 2007 Department Supervisor: Susan Malfoy Job Code: 0742 Task Number and Task: 0742-1 Types orders received by mail. 0742-4 Acts as liaison between customer and various departments under close supervision of the Help Desk Manager in order to provide good customer satisfaction. telephone.

as part of continuous improvement process 14 Sierra Associates 2008 .Gap Analysis ‡ What Performance Behaviors ‡ Comparison of actual performance against new or existing standards ‡ Why ‡ To identify the performance gap between what is actually done and what is required or expected ‡ When ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ An intervention is required New processes and/or procedures New equipment/hardware New Applications New technologies Change in staffing Reduction in productivity Governmental mandates Security breaches Routinely.

is able to produce report with no assistance Sierra Associates 2008 15 . Current Performance Level Desired Performance Level Emp. often makes Reports are fully mistakes. repeatedly asks for assistance on producing report. particularly on recalculating Pivot tables Emp. missing info complete and accurate approx 50% of the time every week Emp.Gap Analysis Questions ‡ For each identified task: ‡ Determine current performance level on task ‡ Determine desired performance level Task Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays.

as part of continuous improvement process Sierra Associates 2008 16 .Tasks Selection ‡ What ‡ A determination of which tasks will be addressed in the training ‡ Directs course objectives ‡ Why ‡ To determine the scope and content of the training ‡ When ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ An intervention is required New processes and/or procedures New equipment/hardware New Applications New technologies Change in staffing Reduction in productivity Governmental mandates Security breaches Routinely.

Tasks Selection ‡ Required ± The task/topic is required by law or for safety ‡ Risk ± There is a high risk if task is not done correctly ± The task is critical ‡ Complexity ± Task is difficult or complex ± Task is done frequently ± Task is time-consuming ± Task is critical to the performance of the role/project ‡ Team Considerations ± Task requires coordination with other staff or with other tasks ± Task is part of a collective set of tasks ‡ Performance ± Task is required for acceptable role performance ± Task distinguishes star performers Task Selection Sierra Associates 2008 17 .

missing info approx 50% of the time Emp.example Task Current Performance Level Emp. Include ± Performance Sierra Associates 2008 18 . often makes mistakes. is able to produce report with no assistance Select .Task Selection . repeatedly asks for assistance on producing report.Criteria Produce weekly summary Help Desk reports for all calls using a provided Excel template to be delivered to IT Director on Fridays. particularly on recalculating Pivot tables Desired Performance Level Reports are fully complete and accurate every week Emp.

Select Training Format ‡ What ‡ A determination of training delivery mode ‡ Why ‡ Identify the best tool for the job ‡ Determine training materials and format ‡ Begin to understand possible costs ‡ When ‡ For every training intervention or project Can be done at the task level or just generally Sierra Associates 2008 19 .

very specialized audience Job Aid Classroom Vendor course OJT Training Format flowchart Mentoring CBT e-Learning Text-based Video Active Learning 20 Sierra Associates 2008 .Training Format Options ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ One-on-one/small group coaching Boot Camp ± small.

Budget ‡ What do They want? ± The greatest needs addressed with the least cost (ROI) ‡ Structure your requests by ± Projects supported (the bigger the better) ± Productivity increase expected (SWAG!) ± Number of employees to affected by a particular course/product (one product. many people) ± Problems ³solved´ with metrics Training Costs Worksheet Sierra Associates 2008 21 .

Select method of instruction/setting 5. Develop performance measures 4.No money. Identify performance problems Method Questionnaire/Survey Focus group (see Lunchtime Assessment) 2. Select which tasks 3. no time. no resources Step 1. Estimate costs/budget Use the most reported from the survey You decide based on time and resource limits for course Lecture/classroom On location with internal SME/trainer Sierra Associates 2008 22 .

4. Capture the training needs on the white board or flip chart. 3. Share these Top Twos with the manager for personal development work. Tell needs assessment participants to place their dots on the chart to vote for their priorities. List the training needs in order of importance. A Step Beyond ‡ ‡ Have each person mark their top two needs on their lists of ten most important and then collect the lists. A good format is ³I need to know how to«´ 2. 23 Sierra Associates 2008 . Gather a possible audience (Pizza works well.Lunchtime Assessment 1. Emphasize ³specific needs´.) Ask each person to write down their ten most important training needs in a particular area. Use a weighted voting process to prioritize the training needs across the group (sticky dots) to prioritize the list. as determined by the sticky dot voting process. Avoid duplicate answers by questioning. Give half as many dots as there are items. with the number of points assigned as votes determining priority.

specifications. and marketing guidelines 24 TASKS CONTENT Observe expert Interview SMEs.Basic Data Gathering TYPE OF INFO needed SOURCES OF INFORMATION OBSERVATION INTERVIEW DOCUMENTATION Interview/survey Review employee Observe in work files or personnel AUDIENCE audience or environment records supervisors Observe audience or expert performing Interview expert or other performers Review job descriptions. product/process marketers. or creators of policymakers. or managers Sierra Associates 2008 . and trouble reports Review product plans. policy statements.

IT Training Needs Assessment? IT Trait Obsolescence Process Modification ‡Use vendor training for very specialized skills/tasks ‡Define supported tools/apps/skills and limit training to these ‡Define supported tools/apps/skills and limit training to these ‡For static info ± develop documentation ‡For active info ± consider the fastest Training method ‡Identify audience groupings related to skills/knowledge/performance needed. ‡Tweak content/materials for each audience. ‡Train the Trainer ± use department IT stars 25 Enormous amt of possible subject matter: New products daily New services daily CHANGES daily! Entire campus as possible audience with different needs/levels/application for same tasks Sierra Associates 2008 .

pdf Sierra Associates 2008 26 .monroe.army.The linkage of Assessment and Evaluation From Pamphlet 350-70-6 http://www-tradoc.mil/tpubs/pams/p350-70-6.

Design phase ‡ Ensure the course content link directly to the objectives ‡ Phase 3 .Development phase ‡ Perform rapid-prototyping (implement-evaluate-implementevaluate-etc) or ‡ Have another training specialist check the solutions ‡ Phase 4 ± Implementation phase ‡ Use Level 1 evaluations (next slides) Sierra Associates 2008 27 .Evaluation Along the Way ‡ Phase 1 .Analysis phase ‡ Determining if it really is a training problem ‡ Make sure there isn¶t already a solution available ‡ Phase 2 .

Four Levels of Evaluation ‡ ‡ ‡ ‡ Reaction Learning Behavior or performance Outcomes or results Sierra Associates 2008 28 .

Reaction ‡ Measures and evaluates the learner¶s reactions and opinions about training program itself ‡ Does not measure what new skills the learners have acquired or what they have learned that will transfer back to the working environment.Level One . ‡ Method .Attitude questionnaires after classes COWC Level One Level One Questionaire Sierra Associates 2008 29 .

Level One .Questions to Ask ‡ ‡ ‡ ‡ Rate the instructor Rate the course Estimate how much you have learned in the course Rate the effectiveness of the instructor in stimulating your interest in the subject ‡ Rate the effectiveness of this course in challenging you intellectually Minimally ± What major conclusion did you draw from today's session? ± What major questions remain in your mind? Sierra Associates 2008 30 .

knowledge.and post-testing ‡ Questions should be directly linked to the selected tasks and the course content Sierra Associates 2008 31 .Learning ‡ Measures and evaluates the changes in the participants¶ skills. or attitudes as a result of the training: did they learn anything? ‡ Methods ± Pre.Level Two .

Performance (behavior) ‡ Measures and evaluates the transfer of the learning to the job or organization: can and do people use their newly acquired learning on the job? ‡ Methods ± ‡ Formal (testing or Manager interview) ‡ Informal (observation) ‡ Must take place after the learners have returned to their jobs Manager Level 3 Evaluation Employee Level 3 Evaluation Sierra Associates 2008 32 .Level Three .

Results ‡ Measures and evaluates the impact of the training on the productivity and profitability of the organization: what "return" has the organization received from the training? ‡ Methods ± ± Develop a "balanced scorecard³ with four perspectives ‡ Financial ‡ Customer Satisfaction ‡ Internal Perspective ‡ Innovation and Learning ± Linked to Vision/Mission statement on these aspects ± Allow time for the behavior change to take place ± Evaluate both before and after the program Sierra Associates 2008 33 .Level Four .

Balanced Score Card ‡ Financial ‡ A measurement. such as an ROI. and deepen relationships with its targeted customers ‡ Internal ‡ Achieve excellence by improving such processes as supplychain management. or support process ‡ Innovation and Learning ‡ Ensuring the learning package supports a climate for organizational change. innovation. retain. production process. or the impact itself.) ‡ Customer ‡ Improving an area in which the organization differentiates itself from competitors to attract. and the growth of individuals Sierra Associates 2008 VinFin Balanced Scorecard example 34 . such as how the output is affected ‡ (Can be either soft or hard results. that shows a monetary return.

Big Picture ‡ Sierra Associates 2008 35 .

tradoc. Multimedia Courseware Development Guide http://www. Donald.References ‡ Rossett. Inc.army. http://www. (1994).htm ‡ ‡ ‡ ‡ ‡ Sierra Associates 2008 36 .asat. San Francisco.doc Writing a learning objective standard that measures Learning Objective (LO) performance.html Design Training Needs Assessment Surveys http://adulted.doc Big Dog's Performance Coaching Page http://nwlink. Evaluating Training Programs ± The Four Levels. Training Needs Assessment (Techniques in Training and Performance Development Series. CA: Berrett-Koehler Publishers.mil/briefings/Learning%20objective%20standard%20w riting. (1987). Educational Technology Pubns (August 1987) Kirkpatrick.mil/tpubs/pams/p350-70-2.com/~donclark/perform/coach.army.about.com/od/trngneedsasst/a/TNAsurveys. Allison.

Resources ‡ First Things Fast: A Handbook for Performance Analysis by Allison Rossett (Nov 6. 1991) Beyond the Podium: Delivering Training and Performance to a Digital World by Allison Rossett and Kendra Sheldon (May 23. 2006) A Handbook of Job Aids by Allison Rossett and Jeannette Gautier-Downes (Jun 15. 2001) Article ± What is a Balanced Scorecard http://www.balancedscorecard.org/BSCResources/AbouttheBalancedScorecard/tabid /55/Default. 1998) Job Aids and Performance Support: Moving From Knowledge in the Classroom to Knowledge Everywhere (Essential Knowledge Resource) by Allison Rossett and Lisa Schafer (Nov 3.aspx ‡ ‡ ‡ ‡ Digi l T ls S vey Sierra Associates 2008 37 .

Training Requirements Sierra Associates 2008 38 .

academic institutions and the medical field. focus and productivity. Kristi received her Masters of Education from the University of Massachusetts Amherst and her Bachelors of Science in Chemistry from Worcester Polytechnic Institute.com sierra@samnet. MA 01057 413 267 3585 Cell: 508 789 4112 www. Conflict. service industry. Kristi works with groups on workplace and relationship processes to redirect the focus and perspective. Sierra Associates 8 Munn Road Monson.KristiThompson. She has presented on numerous topics such as Change. She has over 20 years of experience in the development of stronger work environments. Team Development and Quality Improvement.‡ Kristi Thompson is the Principal of Sierra Associates.net ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Sierra Associates 2008 39 . resulting in higher morale. She has applied her expertise in the high-performance manufacturing. hi-tech companies.

e.Data gathering methods 1 Method Questionnaire/ Survey population Pros ±Can customize per audience ±Can be anonymous ±Can and should ask different style questions ±Efficient in collecting lots of data ±The easiest way of collecting needs assessment data. job descriptions. -Provides management/HR perspective Cons ±% return is often less than 20% ±Takes time and skill to develop well ±Directed/limited answers ±Should be limited to 10 minute completion time ±What people tell you and what they really do may differ ±Can¶t probe deeper ±May be difficult or impossible to obtain desired documents ±May require interpretation/ explanation Time and resource intensive Documentation review (i.. performance appraisals) Interview managers/ supervisors ±Good for complex or undefined problems/areas ±Can be modified ³on the fly´ ±Personal touch Sierra Associates 2008 40 .

Data gathering methods 2 Method Observations Pros ±The most direct method of collecting needs assessment data ±Time efficient Cons ±Very resource consuming (time and people resources) ±Dependent upon the quality of documentation and honesty in documents ±Best used in conjunction with conversations with Mgr ±Scheduling issues ±Wide range of participants in attendance ±what people tell you and what they really do may differ ±Sometimes important to verify the results with observations and document analysis 41 Review performance appraisals of lesser performers Focus groups ±Good for complex or undefined problems/areas ±Can be modified ³on the fly´ ±Personal touch ±Ideas can build on each other ±Relatively effective and efficient Sierra Associates 2008 .

Conduct a Task Inventory* 3. Build performance measures for the trained tasks ‡ See next slides Task Inventory T k In en T k el c i n T k Pe f m nce Templ e ex mple Sierra Associates 2008        ‡ Review performance appraisals ‡ Interview manager/department head ‡ Interview manager/department head ‡Ask HR/EAP Task Performance Measure    © ¨§ ‡ Discuss with manager/department head/client Task Selection 42 ! ¦¥¤£  §  ¢  1.Steps for a full Analysis Step Method Toolbox Template Sys em e vie ex mple 2. Understand the client¶s business (Analyze system) ‡ Interview manager/department head System Overview Instrument ¡  . Select which tasks will be addressed 4.

Steps for a Needs Assessment 5. Estimate costs Sierra Associates 2008 Training Cost Worksheet 43 . Select method of instruction/ instruction setting Lockstep Classroom (In-house or Vendor) Boot Camp Video Self-Paced Personalized System of Instruction (PSI) Computer Based Training (CBT) Text Instruction e-learning or Internet Distance Learning (IDL) Job Job Performance Aid (JPA) On-The-Job (OJT) Training Method Decision Flow 6.

Sign up to vote on this title
UsefulNot useful