Professional Documents
Culture Documents
Participants
Certified
Instructional Designer
This document is confidential, it is provided to the recipient for the limited purpose of
supporting the participant for developing better understanding of the skill set delivered in the
program. Any commercial use of this document is prohibited. Copyrights are reserved with
Middle Earth HR and Carlton Advanced Management Institute.
Visit www.middleearthhr.com
Session 5
Implementation and Evaluation
I. Basics of
Instructional
Design.
II. Requirement
Overview Analysis
I. Implementat
ion of I. What Existing
Training – Knowledge do
integrating Implement
Analyse they have
with LMS & Evaluate (bars)
II. Evaluation &Bloom’s
of Training Taxonomy
II. Converting
the bar into
TLO
I. Understandin
g Story board
II. Creating E – Storyboard Design I. Creating Top
content using Level Design
video editing using
Reigeluth
Develop II. Creating
Motivational
I. Identify Different design using
Exercises ARCS
II. Making Gagne’s
Lesson Plan
III. Selection of Right
Exercises
Implementation:
How do you ensure the consistency and quality of all the programs?
Answer
The secret of quality and consistency is effective implementation.
Check List:
Implementation of the Training program is done by ensuring the entire task/ activities
in the checklist are completed.
The main task implementation starts up is to roll out the program, book venue and
manage facilities according to the requirements and schedule the date and time slots
for the program and book the faculty and resources required for the program in
advance. Care should also be taken to make sure that the technical glitches that may
arise are not overlooked.
Check Handout – 4 for sample checklists for a training program.
After the preliminary check list is verified, the implementation should also ensure that
the training standards if any is strictly adhered to and the legal & ethical laws and norms
are followed.
Sample checklist
Reiser & Dempsey (2002) enumerate the following issues as possible legal areas of
concern if the training program is hosted online or is a web based training program:
Copyright:
Taking material from the internet without getting consent, thereby
violating copyright laws or intellectual property rights.
Discrimination:
The training should abide to various antidiscrimination laws that apply include
Age Discrimination, Civil Rights Act, Gender discrimination.
Lawsuits resulting from comments made or material handed out during a
training that is discriminatory (based on age, sex, race, ethnicity, etc.).
Injuries:
The injury of a trainee or employee who received the injury during a training if
the training personnel were negligent in providing a safe training environment.
Injury due to missing or incorrect information received by training materials.
Other Issues:
Failure to develop and offer training which is mandated by the government can
result in litigation initiated by the government.
Possible lawsuits due to employees feeling that they were kept from
promotions because they were denied access to the training which is necessary
for the promotion.
An organization must monitor and audit for criminal activity, periodically
evaluate the effectiveness of the program, and create and communicate
procedures for employees and agents to report criminal activity without fear
of retaliation.
Self-Contained, Closed Products: products where end users cannot typically add or
connect their own assistive technologies, such as information kiosks, copiers, and fax
machines. This standard links to the other standards and generally requires that access
features be built into these systems.
Implementation of ELT
Selection of LMS
An LMS not only delivers content but also handles registering for courses,
course administration, skills gap analysis, tracking, and reporting.
Selection of LCMS:
Rather than developing entire courses and adapting them to multiple audiences, an
LCMS provides the ability for single course instances to be modified and republished for
various audiences maintaining versions and history. Some systems have tools to deliver
and manage instructor-led synchronous and asynchronous online training based on
learning object methodology.
LCMSs provide tools for authoring and reusing or re-purposing content (mutated learning
objects, or MLOs) as well as virtual spaces for student interaction (such as discussion
forums, live chat rooms and live web-conferences).
Authoring systems are authoring environments that have on-screen tools like menus,
prompts, icons etc. that let users enter text, graphics, branching logic etc. and that
generate underlying code
Editing:
Editing is a process which involves preparing a written, visual, audio and film media
to convey information through the processes of correction or modification to result in a
correct, accurate or consistent work to make it complete.
Interactivity
Ex: the interactivity of an iPod includes the way you move your finger on its input
wheel, the way this allows you to select a tune in the playlist etc.
Sharable. The goal is to make learning content readily available to virtually all
members of the learning community. That means the content should run on multiple
platforms and be launch able from any number of SCORM-conformant learning
management systems. It also means the content should carry information that enables
identification and search of the content (meta-data).
Content. The choice of the word "content" rather than course is important. A piece
of content can be as small as a single page, a single image, a single audio file, or a
word or character. This granularity provides great flexibility for learning developers.
Object. This term, from the world of information technology, implies that the
existence of learning chunks or objects containing data and behaviors will make it
easier to develop reusable content.
Reference Model. This term refers to SCORM's role as a roadmap to standards
work, like a bookshelf of reference materials. SCORM-based standards model the
learning content so that everyone needing to combine that content into larger
composites can understand it thanks to the SCORM framework.
SCORM conformant e-learning content can be packaged, deployed to, and delivered
via any SCORM conformant learning management system (LMS). E.g.: SumTotal's
authoring products, LCMS and Tool Book, are SCORM compliant.
Have you tried guessing on who would be the winner or loser of the game?
Solutions:
• You will closely watch the game and see how the players are performing through the game.
Types of evaluation
Training evaluation refers to the process of collecting the outcomes needed to determine
if training is effective. The logic of evaluation, as developed by Scriven, states there are
four steps to evaluation. These steps include selecting criteria of merit, setting standards
of performance, gathering data, and integrating the results to pass final judgment of
value. Based on this logic, then evaluation types are:
Progress evaluation may answer questions such as: participant progress toward certain
outlined goals, what activities and strategies aided the participants to reach
predetermined goals.
Summative evaluation is meant to evaluate the program at its conclusion. This type of
evaluation will attempt to determine: the success of the project, goals being met,
participant satisfaction and benefit, effectiveness, end results versus cost, and whether
the program should be repeated or replicated.
5.2.2Evaluation strategies
Methods and procedures for formative and summative course evaluation should
be carefully planned in the course design process.
Methods and procedures for evaluating student learning must be well articulated
and directly linked to the stated learner objective.
The content of course evaluation should closely link to the course objectives for
the purpose of course improvement.
In his 1975 book Evaluating Training Programs, Donald Kirkpatrick outlined a four-level
model of evaluation that here is addressed in the context of e-learning, e-training, or
blended instruction. Kirkpatrick's four levels are:
Reaction of student - what they thought and felt about the training
Learning - the resulting increase in knowledge or capability
Behavior - extent of behavior and capability improvement
and implementation/application
Results - the effects on the business or environment resulting from the
trainee's performance
This level measures how your trainees (the people being trained), reacted to the training.
Obviously, you want them to feel that the training was a valuable experience, and you
want them to feel good about the instructor, the topic, the material, its presentation, and
the venue.
Start by identifying how you'll measure reaction. Consider addressing these questions:
Did the trainees feel that the training was worth their
time? Did they think that it was successful?
What were the biggest strengths of the training, and the biggest weaknesses?
Did they like the venue and presentation style?
Did the training session accommodate their personal learning styles?
Did the trainees like and Verbal reaction which can be Feedback is not expensive
enjoy the training? noted and analyzed. to gather or to analyze for
groups.
Did they consider the Post-training surveys or
training relevant? questionnaires. Important to know that
people were not upset or
Was it a good use of their Online evaluation or grading disappointed.
time? by delegates.
Important that people give
Did they like the venue, Subsequent verbal or written a positive impression when
the style, timing, reports given by delegates to relating their experience to
domestics, etc.? managers back at their jobs. others who might be
deciding whether to
Level of participation. experience same.
Perceived practicability
and potential for applying
the learning.
Kirkpatrick defines learning as the extent to which participants change attitudes, increase
knowledge, and/or increase skill as a result of attending a program. So, to measure
learning we need to determine the following:
2. Learning Learning evaluation is the Typically, assessments or tests Relatively simple to set up,
measurement of the before and after the training. but more investment and
increase in knowledge or thought required than
intellectual capability from Interview or observation can be reaction evaluation.
before to after the learning used before and after although
experience: this is time-consuming and can Highly relevant and clear-
be inconsistent. cut for certain training such
Did the trainees learn what as quantifiable or technical
intended to be taught? Methods of assessment need to skills.
be closely related to the aims of
Did the trainee experience the learning. Less easy for more complex
what was intended for learning such as attitudinal
them to experience? Measurement and analysis development, this is
is possible and easy on a famously difficult to assess.
What is the extent of group scale.
advancement or change in Cost escalates if systems are
the trainees after the Reliable, clear scoring and poorly designed, which
training, in the direction or measurements need to be increases work required to
area that was intended? established, so as to limit the risk measure and analyze.
of inconsistent assessment.
Level three can be defined as the extent to which a change in behavior has occurred
because someone attended a training program. It can be challenging to measure behavior
effectively. This is a longer-term activity that should take place weeks or months after the
initial training. In order for change in behavior to occur, four conditions are necessary:
62 77 25 21 - 30% POSITIVE
John Julia
Skills as per BARS Level 1 Level 2 Level 3 Level 4
Emotion Control
Presentation Skills
Before Training
Sensitivity &
Response
John Julia
Skills as per BARS Level 1 Level 2 Level 3 Level 4
Emotion Control
Presentation Skills
After Training
Sensitivity &
Response
3. Behavior evaluation Observation and interview over time are Measurement of behavior
Behavior is the extent to which required to assess change, relevance of change is less easy to
the trainees applied change, and sustainability of change. quantify and interpret than
the learning and reaction and learning
changed their Arbitrary snapshot assessments are not evaluation.
behavior, and this reliable because people change in
can be immediately different ways at different times. Simple quick response
and several months systems unlikely to be
after the training, Assessments need to be subtle and adequate.
depending on the ongoing, and then transferred to a
Of all the levels, measuring the results of the training is likely to be the costliest and time
consuming. The biggest challenges are identifying which outcomes, benefits, or results
are most closely linked to the training and coming up with an effective way to measure
these outcomes over the long term. This can include increased production, improved
work quality, reduced turnover, etc.
Measuring outcomes:
o Sales volumes
o Customer retention
o Customer support
o Time for task completion
o Defect reduction
Increased employee retention.
Increased production.
Higher morale.
Reduced waste.
Increased sales.
Higher quality ratings.
Increased customer satisfaction.
Fewer staff complaints.
The above will vary widely depending on the business and product or service provided.
CASE:
Methods of evaluation used in each level and are they formative/ summative and criterion/
Normative evaluation types.
Level 1: Reaction
Use the 4 – levels of Kirkpatrick’s evaluation model to analyze, improve and evaluate a
training program.