You are on page 1of 4

CORPORATE LEADERSHIP COUNCIL

FEBRUARY 2004
www.corporateleadershipcouncil.com

LITERATURE KEY FINDINGS

Assessing Training Effectiveness

METHODOLOGIES AND METRICS TO TRACK TRAINING EFFECTIVENESS
The Modified KirkPatrick Model

Key metrics for evaluating
training programs under the
Modified Kirkpatrick Models:
!
!
!
!

Amount of knowledge acquired
Change in employee on-the- job
behaviors
Change in profitability as
expressed through productivity,
quality of work, and sales
Trainee satisfaction

The Modified KirkPatrick Model assesses various metrics connected to training effectiveness
with each level progressively building on the previous. This section provides an overview of
1,2,3
the Modified Kirkpatrick Model.
!

Level one measures the following metrics—Trainee satisfaction with aspects of the
program including topic, speaker etc., and quality of program, and applicability of
materials taught.

!

Tools—Attendees completed surveys and evaluation forms assessing satisfaction with training,
baseline comparison for training program attendance, and hours of training
Between 85 to 89 percent of organizations assess training programs at the first level

Level two measures the following metric—The amount of knowledge acquired during
training including improved skills and professionalism.
• Tools—Exams, self-assessment, facilitator assessment, active testing (simulations, case
studies, skill practices etc.), and team assessments
• Between 37 to 41 percent of organizations assess training programs at the second level

!

Level three measures the following metric—The extent to which participants change
their on-the-job behavior.
• Tools—360 Degree evaluations, formal surveying of skills prior to training and following training,
on-the-job observation, follow-up focus groups, action planning, and program assignments
• Between 12 to 17 percent of organizations evaluate training programs at the third level

Case in Point: Level Three Evaluations
4
Conducted by Hutchinson Technology
Hutchinson Technology conducted level three Kirkpatrick evaluations by
assessing employees’ skill levels prior to training by surveying customers,
supervisors, and peers on how frequently employees displayed particular
behaviors. After the training, the company surveyed the same individuals
using a similar survey. The survey revealed that employees used the
selected skills 75 percent of the time, as opposed to 50 percent of the time
prior to undergoing training.

!

Level four measures the following metrics—The impact training has on profitability,
productivity, quality of work, sales, turnover, and expenses.
• Tools—Baseline and trend-line comparisons of hours in training, training session attendance,
employee turnover, sales rates, profits, expenses, customer satisfaction, and monitoring of
overall business performance
• Approximately seven percent of organizations utilize four of the five levels of the Modified
Kirkpatrick model4

 2004 Corporate Executive Board
CATALOG NUMBER: CLC11TQ4SZ

! Participants receive a questionnaire. they should keep the following CEO-type questions in mind: ! ! ! ! Does training work fast enough to make a difference? Does having a well-trained employee really make a difference in our industry? Does having a well-funded training function help the organization attract and retain the best people? If the training budget was doubled. “Nine Managers Reveal the Business Impact of their Training Efforts” March 2003 ! Southwest Airlines Balanced Scorecard Model—Southwest evaluates training programs prior to the program’s start. detailed below. that asks for feedback on how the training impacted participants’ behaviors. In this manner. which helps IBM understand which training components are most valuable to specific employee groups Merck & Company’s Training Effectiveness Metric—Merck & Company uses a unique formula. ! Verizon’s Return on Expectations Model—Verizon evaluates training programs prior 6. to measure the impact of specific training programs. • Prior to training—Translate value drivers.CORPORATE LEADERSHIP COUNCIL ASSESSING TRAINING EFFECTIVENESS PAGE 2 KEY FINDINGS METHODOLOGIES AND METRICS TO TRACK TRAINING EFFECTIVENESS (CONTINUED) Alternate metrics for evaluating training programs are the following: ! Alternate Methods and Metrics to Evaluate Training Effectiveness ! Customer service gains and operational efficiency Gain formula Internal promotions Return on expectations ! ! ! Compaq’s Success Case Model—Compaq evaluates its training programs after their 5 conclusion based on content learned as well as job-application as highlighted below: • Post-training survey—Employees fill out a survey testing them on the fundamentals of the content presented. and polling on their opinions regarding the applicability of the training to their job requirements • Post-training interviews—Compaq interviews the employees who provided the 15 “best” responses and the 15 “worst” responses to obtain information about the program’s strengths and weaknesses. such as customer service and operational efficiency into metrics.  2004 Corporate Executive Board . based on customer service gains and operational 8 efficiency. Analysts then compare the answer to the difference in productivity between the control group and the group that underwent training As organizations plan their own training metrics. With this model. Southwest implements training evaluation at the top of the organization and works down rather than the inverse method followed by most models • Post-training—Analyze metrics using tools such as baseline comparisons and surveys to determine if targeted goals are met ! IBM’s Web-based Assessment Tool—IBM uses a Web-based assessment tool to 9 evaluate its leadership development training: 1. The questionnaire asks the participants to assign a monetary value to the training they received The responses are correlated to the departments and functions of participants. This helps the company assess whether programs achieve the goal of producing successful leaders 11 that can be promoted. at least six months after completing the development training.7 to the program’s start based upon control groups and productivity outcomes: • Prior to training—Verizon divides employees into two groups: a control group that will not undergo training and a group that is trained. would productivity double? -Managing Training and Development. The controlling executive documents his/her expectations for the training’s goals • Post-training—The controlling executive indicates whether results met expectations. Merck determined that its average ROI for training programs is 10 84 percent and terminated 53 programs that were not producing high enough returns: Calculation—GAIN = d x SD$ x JSI x N • • • • ! d—Shift in performance by average individual undergoing training expressed in standard deviations from pre-training average SD$—The value in dollars of one standard deviation of performance shift JSI—Percentage of job skills impacted by training N—Number of participants who underwent training Ford’s Career Path Tracking System— measures the effectiveness of its Leadership Development Center by monitoring the career paths of alumni of the center. 2.

by asking the participants how much of the change in behavior is attributable to training. new hire. TACTIS TO DETERMINETRAINING SUCCESS Tactic # 1: Collect Data to Demonstrate Change in Behavior ! Collect date to demonstrate change in behavior Isolate the effect of training Lower turnover by spending on training To determine if training has been successful.15 Literature reveals the following strategies to isolate the effect of training: • • • • • • • 360-degree feedback—Firms can measure the change in behavior from different perspectives. Follow-up assignments—These can demonstrate to companies the application of training content. Customer input—Companies solicit input on service quality directly from customers. business initiatives. Tactic # 3: Lower Turnover Rates by Spending on Training ! Research reveals that companies that spend more than average amount on training have a higher placement of internal hires and lower annual turnover rates. new product. Dell Learning launched a Web-based global measurement system that provides online access to training statistics at global.  2004 Corporate Executive Board ..CORPORATE LEADERSHIP COUNCIL ASSESSING TRAINING EFFECTIVENESS PAGE 3 KEY FINDINGS METHODOLOGIES AND METRICS TO TRACK TRAINING EFFECTIVENESS (CONTINUED) Alternate Methods and Metrics to Evaluate Training Effectiveness (Continued) ! The tactics to determine training success are the following: ! ! ! Dell’s Web-based Global Training Measurement System—In 1998. Then performance after training is compared to predicted performance without training on a trend-line. The data can be collected in the 13 following ways: • • • • • • • Surveys Questionnaires On-the-job observation Post-program interviews Focus groups Performance monitoring Performance contracts in which a participant. 14. Control groups—Experimental group receives training while a control group does not. and segment. assuming the trend will continue without training. Surveys and questionnaires—These can capture participant’s accomplishments after training. Post-training measurements demonstrate performance differences between the two groups. The Training by Type Report documents training activity by category including management. 12 professional development and compliance. One-on-one interviews—Companies can glean specific details on the results of training i. Companies that spend $273 dollars per employee have less than seven percent annual voluntary turnover. Tactic # 2: Isolate the Effect of Training ! Another way to determine if training has been successful is by isolating its effect. customer service. sales. the Training Snapshot Report documents all training activity including classes taken. business. Trend-Line analysis—Companies draw a line from current performance to future performance.e. executive. For example. and the participant’s supervisor agree on specific outcomes from training. technical. business. the instructor. Companies that spend $218 dollars per employee in training have more than 16 16 percent annual voluntary turnover. organizations can collect data that demonstrate changes in behavior through evidence provided via employee observation in specific situations both before and after training. segment and department levels. total tuition and enrollment and total hours by region.

Quintin Jarrett." Training and Development (1 May 2001). “A Rational Approach to Evaluating Training Programs … Including Calculating ROI. “Leadership DNA: The Ford Motor Story. "Training Proves Its Worth. "American Society of Training and Development State of the Industry Report 2002. "Corporate Training -. 7 Pam Leigh. “How Dell Does It. (Obtained through ProQuest). [Accessed 30 January 2003]. (Obtained through Lexis-Nexis)." Financial Analysis. a copy of this report cannot be provided). (Obtained from Factiva). Washington: Corporate Executive Board (September 2001).businessinnovation. [Accessed Feb 23 2004] (Due to copyright restrictions. 14 Carroll Lachnit. 2002). "Systematic Strategies." 15 Jack J. "Training Proves Its Worth." 2  2004 Corporate Executive Board ." Managing Training & Development (July 1. (Obtained from Factiva). (Obtained from Factiva). Voice of the Leader. “Measuring Return on Training at Merck & Company. 4 Beverly Geber. 10 M. 11 Stewart D. "How Southwest Airlines Developed its Balanced Scorecard Analysis. (Obtained from Lexis-Nexis). (Obtained through Factiva). 13 Carroll Lachnit.” Ernst & Young Center for Business Innovation (Date Unknown). 3 Jack J." Workforce (September 2001).com). 5 Author Unknown.astd.” The Journal of Lending & Credit Risk Management (1 July 1997) (Obtained from Factiva). 6 John Berry. "Training ROI How Compaq Uses the Success Case Method to Prove ROI of Training. Phillips. (Obtained from Factiva). Planning & Reporting (1 July 2002). "Does Your Training Make a Difference? Prove It!" Training (March 1995). 16 Caroll Lachint. Mark Van Buren and William Erskine. 9 Corporate Leadership Council.” Training & Development (1 June 2000).The E-Learning Center. "Was it the Training?." CMA Management (1 December 2002). (Obtained through www. (Obtained from Factiva). 8 Author Unknown.” Training & Development (March 2001). "Training Proves Its Worth.org/ (February 2002)." InternetWeek (6 November 2000). "Training's New Guard 2001 (Profilles of New Employee Trainers). Phillips.CORPORATE LEADERSHIP COUNCIL ASSESSING TRAINING EFFECTIVENESS PAGE 4 KEY FINDINGS 1 Ajay Pangarkar and Teresa Kirkwood. Friedman." www. (Obtained from Lexis-Nexis)." Training and Development (March 1996). (This source is no longer available online). 12 John Cone.ey.