Evaluation of Online Professional Development (OPD) program

at Norther Illinois University (NIU)

Evaluation project proposal

Fahad Alenezi

ETR531

1
 Introduction

Evaluation context

Online professional development (OPD) is designed and applied by The Faculty Development

and Instructional Design Center at Northern Illinois University. The aim of this center is aims to

support faculty, academic supportive professional staff, and graduate teaching assistants through

a variety of programs, resources, and services in fulfilling Northern Illinois University's mission

on teaching and scholarly activities. The target population of this program are faculty, academic

supportive professional staff, and graduate teaching assistants who are working in all the colleges

at NIU.

OPD is a program which designed to prepare faculty members at NIU to teach online through

providing opportunities for effective online teaching and providing training to Facilitate the use of

instructional technologies in teaching. Specifically, this program is designed to serve those who

are interested in teaching online. This program provides online workshop, seminars and online

teaching consultations to faculty to provide them a basic information and skills in how to integrate

technology to teach online and to design courses that combatable with online teaching. The

program covers different topic that help faculty to teach online such as workshops in how to apply

the Quality Matter Rubric when designing online courses, how to design online courses using

different software i.e., captivate, articulate and lectora. The program run through the semesters and

sometimes they have a short and extensive program during the summer semester.

2
Summary of Need Assessment

As the goal of the OPD is to prepare faculty to teach online, the clients are worry and not really

sure if OPD which they offer meets the faculty needs. In addition to that, clients are wondering if

the efforts they put on this program is worthwhile. According to the Assistant Director and the

Online Teaching Coordinator, they would like to know if the OPD they offer are really improve

the faculty skills to teach online.

Data Collection of Need Assessment Phase

The evaluator sent an email to the stakeholder at Faculty Development and Instructional

Design Center to see if they have concern or needs for an evaluation in their program. Response

was received from Stephanie Richter the assistant director. The evaluator met with both the

assistant director and the Online Teaching Coordinator several times and provided a report about

the need of the evaluation. The result from this report indicated that the stakeholders mentioned

some concerns about their OPD if it really improves faculty skills to teach online and if this

program meets the faculty expectations. They mentioned that they do have different evaluation for

their workshop and face to face training, however, they do not have an overall evaluation for the

OPD they designed. Both the assistant director and the Online Teaching Coordinator asserted that

an overall evaluation will provide reliable information about if their programs are really fits the

faculty members needs. An evaluation of OPD is needed to improve these professional

developments on a way the meet the stakeholders needs.

Purpose of Evaluation

Based on the finding of the needs assessment, the purpose of this evaluation is to evaluate the

overall effectiveness of the online professional development (OPD) that prepare faculty to teach

online at Northern Illinois University.

3
Evaluation Model

There are a wide range of educational program evaluation that assess the overall effectiveness.

Kirkpatricks (1994) and Guskeys (2000) evaluation models are the most common models that

evaluate the overall effectiveness of professional development/training program. Although

Kirkpatricks model is designed to evaluate program in the business and industry context. While,

Guskeys (2000) model is designed for evaluating educational professional development program.

For the purpose of this study, Guskeys model will be used to guide the evaluation of OPD at NIU.

In Guskeys Model there are five levels of evaluating (1) reactions (satisfaction with the training),

(2) learning (gains in knowledge, attitudes, or skills demonstrable in training), (3) organization

support (institutional commitment), (4) transfer (behavior change), and (5) results (impact). This

program will be evaluated based on these five levels. OPD program will be evaluated based on

these five levels using survey, interview and documents analysis.

Evaluation Questions

Based on Guskeys Model, the following evaluation questions will guide this evaluation:

What are instructors feelings about the OPD materials, content, and delivery of training?

What knowledge and skills did participants gain from the training?

Did the university support the training program?

Do instructors change their instructional practices based on what was learned at the OPD?

Did the OPD impact instructors online teaching?

4
Data Collection and Analysis

In this phase a quantitative and qualitative data will be collected via semi-structured interview

and questionnaire. The procedure of the data collection will be as follow:

Survey will be sent to faculty who already participated in OPD by email. And at the end of

the survey, participants will be asked to be interviewed to gain more information.

After receiving the participants responses, the data will be analyzed using SPSS. And then,

the evaluator will choose four of the participants who showed an interested to be

interviewed.

Interviews will take place at the participants convenience time and place and will last

between 20 to 40 minutes.

SPSS software will be used to analyze that quantitative data, and Nvivo software used to

identify themes from the interviews.

Recorded previous online training will be reviewed and evaluated based on Quality Matter

for online courses.

The following table show the levels that will be evaluated and how this data will be used.

5
 Table1. Guskeys Five Critical Levels of Professional Development Evaluation

Education Gathering What is Measured or How Will Information Be

Data Analysis Plan
Level Methods Assessed? Used?

1. Participants Questionnaires Initial satisfaction To improve program Statistical

Reactions with experience. delivery and design Analysis: Means.

Questionnaires New knowledge To improve program Statistical
2. Participants
and/or skills of content, format, and Analysis: Means.
Learning participants. organization.

Questionnaires The organizations To document and Statistical

3. Organization
advocacy, support, improve Analysis: Means.
support and accommodations, organizational support
facilitation and
change recognition.

Questionnaires Degree and quality of To document and Statistical

4. Participants
information. improve the Analysis: Means.
use of new implementation of
program content
knowledge or
skills

Questionnaires affective (attitudes To focus and improve Statistical

5. Participants
and dispositions). all aspects of program Analysis: Means.
learning Interview Overall effective of design, Nvivo software
the program. implementation, and used to identify
outcomes. follow-up. themes and their
To demonstrate the frequency.
overall impact of
professional
development

6
Benchmark or Standard for evaluation

The quantitative data will be measured by calculating the mean of the composite score for

participants on Likert scale. Regarding the evaluation of the previous OPD, the evaluator will use

Quality Matter (QM) rubric to evaluate the OPD to determine if the program meets the standard.

QM is a process designed to assure the quality of online course. QM consists of six standards

namely Course Overview and Introduction (16 points values), Learning objectives (15 points

values), Assessments and Measurements (13 points values), Instructional Materials (13 points

values), Learner Interaction and Engagement (11 points values), Course Technology (10 points

values), Learner Support (9 points values), Accessibility (12 points values) Standards from the

Quality Matters Higher Education Rubric, 5th Edition

Resources

Different kind of resources should be available for the evaluator like faculty emails who

participated in OPD, access to previous recorded online session and demographic data about the

participants.

7
Budget

The proposed cost of this evaluation is $8,000. For more details, see the following table.

Table 2

Item cost Note

Designing, distributing the survey. Analyzing $2,000,00

the quantitative data
Interview, interview transcription and analysis. $2,000,00

Document (online course) review based on $3,000,00

QM
Writing and delivering report $,1000,00

Total $8,000,00

Timeline

Stage Date

Designing, distributing the survey. Analyzing the February 1, 2017

quantitative data
Interview, interview transcription and analysis. March 1, 2017

Document (online course) review based on QM April 1, 2017

Writing and delivering the final report May 1, 2017

8
References

Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin.

Standards from the Quality Matters Higher Education Rubric, 5th Edition. Quality Matters.
Retrieved from
https://www.qualitymatters.org/node/2305/download/QM%20Standards%20with%
20Point%20Values%20Fifth%20Edition.pdf

9
 Appendix A

Survey items

Strongly Agree Neutral Disagree Strongly

Statement Agree Disagree

1. The approach of OPD is one which

matches my preferred style of learning.

1. The OPD materials are well matched to

my needs and relevant to the subject or
phase in which I teach.

2. The trainers/mentors are well informed

about the subject/phase in which I teach
and are able to offer sound advice and
guidelines.
3. The OPD has taken account of what I
already know and is matched to my
identified needs.
4. I am aware of the progress I have made
and any further training that I may
require.
5. High speed Internet available at My
University.

6. The OPD helps me to understand when

and how I should employ ICT to teach
my subject.
7. The OPD has enabled me to apply ICT in
my teaching and to be more critical of
when ICT is an appropriate teaching
resource.
8. After the OPD, I felt more confident in
teaching online.
9. The OPD enhanced my knowledge and
skills.

10
 Appendix B

Interview Question

What type of resources and support are made available to you to support OPD to teach

online?

What impact did the OPD you receive regarding teaching online have on your ability to

teach online?

Do you think the knowledge you gained from the OPD has increased your students

achievement in any way? If so, how? If not, why do you think this is the case?

11