Evaluating a Federally Funded Faculty Training Program

Members and roles:

Jackie Adams: Instructional Designer, MA in ID
Ray DeMilo, Jackie’s Boss
Ron Bentley, the dean of the school, client
Bruce Stingel, the vice president for academic affairs, client
Hank Lundstrom, an engineering technology professor and the lead evaluation team
member tasked with assessing the evaluation plan
Settings:
Development delivery & evaluation of in-service faculty training of a large university in
tandem with technical subject experts. The Advanced Manufacturing Technology Education
(ATME), which was aimed at improving science and engineering education at undergraduate
level by especially keeping the teachers up to date with the development in their fields, was
introduced by Advanced Technology Education (ATE) as a new project.

Instructional Desinger’s Main Mission:

As the main ID of the taskforce, Jackie was not only tasked with designing an optimal
approach to implementing ATME but also commissioned by her boss to develop a detailed
evaluation plan for ATME in line with the newly announced legislation with necessitated a
more meticulous performance evaluation for all federally funded projects. She,
consequently, reexamined the evaluation section of the grant documents and decided she
needed to highlight the significance of the evaluation itself in the light of the mission of the
project and clarify what exactly was going to be evaluated while delineating how and when
the evaluation was going to be conducted and by whom. She, then, created her evaluation
plan which consisted of an Implementation Survey (a questionnaire to evaluate integration
of key content areas into existing curricula), a Pre/Posttest to evaluate the knowledge gained
in training, and a Reaction Survey to collect feedback from instructors on the value of the
materials and other elements provided to the by the ATME. The major challenge of the
project for Jackie was an evaluation by a six-member team sent by the funding agency. This
was a high stake evaluation the results of which would be shared with the university and the
funding agency. As the lead member and the person responsible for assessing the evaluation
process of the project, Hank Lundstrom spent most of his visit with Jackie and reviewed her
work while asking her detailed questions. The case study does not inform us whether the
review team were happy with Jackie’s evaluation plan and wrote a positive report or not but
since samples of the plan documents are given as appendices to the case study, it seems that
the goal is to give the reader a glimpse of what an internal evaluation would look like.
Evaluation: Jackie is a novice instructional designer and obviously the weight of the new
Evaluation Plan is heavy on her shoulders. To her misfortune, her boss, Ray, is not supportive
enough, either. She prepares some tools to measure the learning outcomes but she seems
to have forgotten that the major goal of the project is to enhance the quality of instruction
as she fails to develop tools to measure this reliably. She relies solely on some survey
questionnaires that mostly focus on the implementation of the learning goals into the
curricula. To demonstrate instructors have improved by mastering state-of-the-art topics
and incorporating those in their teaching agenda, she needs more reliable methods of
measuring this.