MDDE 604: Instructional Design and Program Evaluation in Distance Education

MDDE 604: Instructional Design and Program
Evaluation in Distance Education
Introduction
Welcome to MDDE 604: Instructional Design and Program Evaluation in
Distance Education. In this course, you will be introduced to the concept
of individualized or flexible instruction, and consider the attributes of
instructional systems that can be adapted to meet the needs of
individual learners. Learning objects will be discussed and you will
have the opportunity to use and even design learning objects of your
own.
In the first part of the course, you will work through the various stages
of the instructional systems design (ISD) process, designing and
developing flexible learning materials that meet an educational need or
solve a performance problem that you have identified. In the latter part
of the course, you will explore the process of program evaluation in
general, and distance education in particular, concluding with the
design or analysis of an evaluation system for distance education.
MDDE 604 has been designed to be more applied in focus than the
other core courses in the Master of Distance Education program. The
distance education concepts introduced in MDDE 601: Introduction to
Distance Education and Training provide the foundation for
understanding the context in which your instructional materials will
be delivered. Although it is not a prerequisite, MDDE 602: Methods of
Inquiry and Decision Making covers concepts that will be useful as you
study the program evaluation part of this course. In addition, as you
design and develop the instructional unit that forms the basis of this
course, you will apply the instructional design skills, knowledge, and
concepts you were introduced to in MDDE 603: Systems Design in
Distance Education.
Course Assessment
Your final course mark will be based on three assignments and on
your participation in the computer conferences. The assignments for
the course are briefly described below.
In Assignment #1, you will complete the analysis stage of the

instructional systems design process. This includes a real or
Master of Distance Education 604 / Study Guide 1

hypothetical needs assessment, in which you will identify an
educational need or performance problem that could be resolved by
instruction. Assignment #1 is worth 10% of your course grade. Further
details on this assignment are located at the end of Unit 2.
In Assignment #2, you will develop a unit of instruction and discuss

the instructional design, performance assessment, and delivery
considerations associated with the course that includes your
instructional unit. This assignment is worth 45% of your course mark.
Further details on Assignment #2 are located in Unit 3.
Assignment #3 involves the application or course of program

evaluation principles in a distance education context. Two options are
available. You may complete a group activity where you evaluate a set
of learning objects, or create a program evaluation system following
the Calder (1994) framework, This assignment is worth 30% of your
course mark. Further details on Assignment #3 are located in Unit 5.
Computer Conferences: There are four required computer

conferences, which together make up the remaining 15% of the course
mark. The topics for the conferences are specified on the course web
page.
Course Materials
The course package for MDDE 604 contains the following materials:
? This Study Guide for MDDE 604
? A set of articles in the Course Readings:
Fletcher, J. D. (1992). Individualized systems of instruction. (American

Forces Information Services, Report # IDA-D-1190.) Washington,
DC: Office of the Assistant Secretary of Defense for Force
Management and Personnel (DOD).
Romiszowski, A. (1994). Individualization of teaching and learning:

Where have we been? Where are we going? Journal of Special
Education Technology, 12 (3), 182-194.
Nesbit, J., Belfer, J., & Vargo, J. (2002). A convergent participation

model for evaluation oflearning objects. Canadian Journal of
Learning and Technology, 28 (3), 105-120.
Texts:
2 Instructional Design and Program Evaluation in Distance Education (Fall 2003)

Morrison, G., Ross, S., & Kemp, J. (2003). Designing effective instruction
(4th ed.). New York: Wiley.
Calder, J. (1994). Programme Evaluation and Quality. London: Kogan

Page.
NOTE: In addition to the materials listed above, you will need to use
the following text, which was part of the course materials for Systems
Design in Distance Education (MDDE 603):
Smith, P. L. & Ragan, T. J. (1999). Instructional design (2nd ed.). Upper

Saddle River, NJ: Prentice-Hall.

Unit 1 Individualized Instruction
Introduction
In education and distance education settings alike, courses and
programs are more effective when the unique characteristics of
learners are taken into consideration. Like many service providers
today, distance educators need to be consumer oriented, taking
student needs and preferences into account in order to deliver a
quality product. To achieve this goal, the concepts and theories
associated with individualized instruction (also called flexible learning
or instruction, or individualized learning) are valuable tools for those
involved in distance education.
This unit examines various attributes of instructional systems and

explores how they may be adapted to meet the needs of individual
learners. The readings in this unit provide a foundation for
understanding basic concepts pertaining to individualized instruction
and the application of these concepts within distance education
settings.
Objectives
After completing this unit, you should be able to do the following.
4. Discuss how the following factors associated with individualized

instruction can be varied to adjust to the needs of learners:
a. pace;
b. sequence;
c. content;
d. mode of instruction and learning style;
e. level and type of interactivity.
5. Discuss the strengths and limitations of applying the principles of

individualized instruction to education in general, and to distance
education, in particular.

Resources
In addition to reading this Study Guide, you should read the following
articles from the course Readings:
? Individualized Systems of Instruction by J.D. Fletcher
? Individualization of Teaching and Learning: Where Have We Been; Where

Are We Going? by Alexander Romiszowski.
Also read the following online articles:
? An Updated and Theoretical Rationale for Interaction by Terry

Anderson. This article is found at the following address:
http://it.coe.uga.edu/itforum/paper63/paper63.htm
? Using students’ learning styles to provide support in distance education

by Mohamed Ally and Pat Fahy. This article can be downloaded
from the Faculty Showcase on the CDE home page.
Optional Activity: Complete the Learning Styles Questionnaire at

http://www.ncsu.edu/felder-public/ILSdir/ilsweb.html. There you
will find resources to help you determine your own learning style and
learn how to accommodate it.
Study Questions
To guide you to important material in these articles, Study Questions
are provided below for each of these readings.
Study Questions (Fletcher Article)
1. Define the term individualized instruction.
2. Explain the following statement: “Individualized systems of

instruction may be viewed as attempts to recapture the power of
individualized instruction without the encumbrance of its
expense” (Fletcher, 1992, p. 8). How does this statement relate to
what Bloom calls the 2-sigma problem?
3. Fletcher identifies four components of an instructional system that

can be adjusted to individualize instruction – pace, sequencing,
content, and mode of instruction. Give an example of each
component and explain how it can be varied in a course or
program.

4. Briefly describe two different individualized systems of instruction
for both print-based and computer-based approaches.
Study Questions (Romiszowski Article)
1. What factors have led to the adoption of individualized approaches

to teaching and lea rning?
2. Give three examples of different approaches to individualized

instruction.
3. Will new technological initiatives in individualized instruction

succeed, or is the route to individual treatment of learners more by
way of
a. improving the skills of the teaching profession in the area of
“making learning personal, empowering, and successful,” or
b. improving the skills of the learners themselves, empowering
them to exercise their options as autonomous participants in
the teaching-learning process?
Explain your choice.
4. What are the four principal components of a teaching-learning

system? What does the future hold for the improvement and future
development of these components?
Study Questions (Ally and Fahy Article)

1. What are the various types of learning style inventories?. What do
they measure?
2. Describe the following types of learning styles: diverger;

assimilator; converger; accommodator. What kind and/or level of
support strategies does each require?
Study Questions (Anderson Article)
1. Identify and describe the various modes of interaction.
2. Consider the following statement: “After years of sometimes

acrimonious debate, it seems clear that there is no single media that
supports the educational experience in a manner that is superior in
all ways to that supported via other media.” Do you agree with this
statement? Why or why not?

3. Using an example from your own experience, apply the
equivalency theorem to illustrate how interactivity can be used to
enhance the effectiveness of distance education.

Commentary
What is individualized or flexible instruction?
Individualized instruction is the use of a variety of methods and

materials to adjust to differences in learners. Usually, individualized
instruction facilitates learning in the absence of a teacher, i.e., through
self-instructional materials. A wide variety of individualized
instructional methods exist, including the use of any number of media
and materials, and ranging in sophistication of design to include
contract learning, independent study, computer-based instruction, and
online or e-learning.
Individualized instruction promotes learner autonomy. For example, it

permits students to learn when they want to by allowing for differences
in timing, frequency, and duration of study. It allows students to learn
how they want by varying modes of learning as well as amount and
type of interaction. Individualized instruction allows learners to learn
what they want by providing choices regarding subject matter, learning
expectations, and resources.
Where did individualized instruction originate?
Individualized instruction is by no means new. The first example was

probably the Socratic educational method advocated by Plato and the
ancient Greeks, using one-to-one tutorials in which teachers adapted
their behaviour to the needs of students. Similar tutorial-based
instruction characterized the elitist educational models prior to the
Industrial Revolution.
After the Industrial Revolution and throughout the 19th and 20th
centuries, increased demands on educational and training systems
resulted in many different approaches to make learning more
egalitarian and more individualized. In Canada, correspondence
schools, both public and private, have offered a version of
individualized instruction in elementary and secondary schools, post-
secondary education, and professional education since the early 1860s.
For the most part, these offerings consisted of print-based materials
with assignments sent through the mail, and supervised exams set by
the institution or agency at specified locations.
Movements toward individualizing and personalizing education have

occurred throughout the modern history of education. In the 1920s and
30s, for example, initiatives such as the Dalton Plan and the Winnetka
Plan attempted to make conventional basic education responsive to

individual needs. These kinds of initiatives continue to evolve and
develop, and examples of them abound throughout educational
systems in Canada and abroad.
The growth of instructional systems design, which began in the 1960s,

can be attributed to the effective learning it evokes as a result of the
emphasis placed on meeting the needs of individual learners. This
approach, which originated in the military and training sectors,
quickly moved to secondary and post-secondary education where its
influence continues today.
In the last decade, two additional factors have had a significant impact
on individualized or flexible learning. The first was the introduction of
constructivism and its emphasis on personally meaningful learning. In
a constructivist approach, each learner creates a unique array of
knowledge and skills as a result of their education, experiences, and
environment. The second was the growth of web-based information
and communication capabilities. These developments have had a
tremendous impact on individualized learning allowing almost
limitless possibilities for educators.
What attributes can be individualized?

In the design of flexible learning materials, there are several attributes
that can be individualized and made personal to learners. As you
learned in the Fletcher (1992) article, individualization may occur in
terms of pace of instruction, content of instruction, sequence of
instructional activities, and style of instructional presentation.
The pace of instruction can be placed entirely under the learner’s

control, it can be directed by an external source (e.g., the instructor or a
device like a computer in response to the learner’s activities), or it can
be controlled by some combination of learner and other control.
Similarly, with respect to content and sequence, these can be
determined by the learner, the instructor, or some combination of the
two.
Much is said and argued in the literature about the importance of

matching the student’s learning style to the style of instruction. As you
learned from the readings for this Unit, Fletcher (1992) contends that
students should generally be given the opportunity to develop the
skills necessary to use materials and instruction in a wide variety of
styles, rather than have the materials matched to their preferences.
More recently, Ally and Fahy (2002) expand on this area, concluding
that certain learning styles require more support than others in online
distance education and recommending that courses accommodate such
learner preferences.

Interaction is another element that can be individualized. As Anderson
(2002) notes, the types and levels of interactivity can vary. He proposes
the equivalency theorem:
Sufficient levels of deep and meaningful learning can be developed as long as

one of the three forms of interaction (student-teacher; student-student;
student-content) are at very high levels. The other two may be offered at
minimal levels or even eliminated without degrading the educational
experience. High levels of more than one of these three modes will likely
deliver a more satisfying educational experience, though these experiences
may not be as cost or time effective as less interactive learning sequences. (p.
4)
An important issue associated with individualization of any of these

elements is the management of the instructional process. Generally, the
greater the variability of individualization of instruction, the greater
the complexity of the system and the greater the requirements for
management and monitoring.
Many educational applications involve group-based instruction

including, for example, common scheduling, pre-established times for
testing and examinations, and shared break times. In the situations,
record-keeping is also based on group activity.
On the other hand, in an individualized system, students may progress

at different rates, work on different activities for variable lengths of
time, enter and exit the system at different times for different
durations, and complete instructional activities at different times, in
different places, and take variable lengths of time to do so. So the
management, monitoring, and record-keeping for an individualized
system is usually complex. In most educational applications today,
record-keeping is usually conducted through the use of a learner
management system.
Concepts Underlying Individualized Instruction

The underlying philosophy of individualized instruction presumes
that most learners can (and will) learn effectively if they receive the
appropriate opportunities and support. To make this happen,
circumstances must be arranged to take advantage of what students
already know and to make use of their own learning strategies. This
point is presented in the Fletcher (1992) article in the discussion of the
2-sigma problem.
In most classroom or group-based learning situations, curriculum and

instructional strategies are determined by the instructor or
organization and are the same for all students. Further, the amount of
time available for the learning is determined by the organization and
based upon the needs of the average learner in the group. Although

students may be grouped by age or level of achievement, their
personal attitudes, knowledge, beliefs, and ways of learning are
essentially considered less important than the needs of the
organization and the group.
In contrast, individualized instructional systems assume that students

have different needs and circumstances. To learn effectively, some
students will probably need more time or different approaches than
others. Therefore, to plan and prepare individualized instruction,
certain information is essential, such as students’ existing knowledge,
skills, and attitudes; students’ preferences and skills in learning; the
goals and objectives of the curriculum; and the tests and evaluation
instruments that will certify the achievement of the specified goals and
objectives. These important areas will be studied in detail in Unit 2:
Conducting an Analysis and Preparing a Proposal.
Unit 2 Conducting an Analysis and Preparing a

Proposal
Introduction
In Units 2 and 3 of this course, you will develop a unit of instruction to
be delivered at a distance. The flexible learning materials you create
should be digital-based (i.e., can be accessed by computer) and self-
instructional (unless you receive prior permission from your instructor
to do otherwise). Your instructional unit should include one or more
learning objects, or it could be a learning object itself.
However, before you begin to develop your learning materials, you

first need to conduct a (real or hypothetical) analysis to ascertain the
purpose of the instructional unit or learning object, the characteristics
of the learners that it will serve, and the course, program, and
organizational context in which it will be situated.
As you learned in MDDE 603: Systems Design in Distance Education, an

instructional system involves many types and levels of sub-systems.
These include the organizational context, the educational or training
program, and learners themselves. As you also learned in that course,
the creation of an instructional system is based on the principles,
processes, and procedures of the instructional systems design (ISD)
approach. Analysis, the first phase in ISD, is the focus of this unit and
the basis of the work you will be completing in this part of the course.

Objectives
1. Explain the importance of conducting a needs assessment for

identifying educational needs or performance problems, and give
an example of how this process can be applied.
2. Identify a unit of instruction or learning object that meets an

educational need or addresses a performance problem.
3. Identify learning objects that may be suitable for inclusion in your

instructional unit.
4. Identify the characteristics of learners who will be completing your

instructional unit.
5. Describe the program/course and organizational context of your

instructional unit.
Resources
In addition to reading this Study Guide, you will need to refer to the
following texts.
? Morrison, G., Ross, S., & Kemp, J. (2003). Designing effective

instruction (4th ed.). New York: Wiley. Chapters 1 – 4.
? Smith, P. L. & Ragan, T. J. (1999). Instructional design (2nd.) Upper

Saddle River, NJ: Prentice-Hall. Review Chapters 3 and 4.
? Calder, J. (1994). Programme Evaluation and Quality: A Comprehensive

Guide to Setting up an Evaluation System. London: Kogan Page. Note:
This text forms the basis of the second part of the course where you
will study program evaluation. However, Chapter 4: Programme
and Curriculum Development (particularly pages 67-72) discusses
needs assessment and may be useful at this stage.
Commentary
The commentary below provides additional information that will help
you complete the assignment for this unit.
The Analysis Phase

As you learned in the previous unit, individualized instructional
systems assume that students have varying needs and circumstances.
To learn effectively, some students will need more time or different
approaches than others. Therefore, to plan and prepare individualized
instruction, certain information is essential, such as students’ existing
knowledge, skills, and attitudes; their preferences and skills in
learning; the goals and objectives of the instruction; and the
assessment that will determine the achievement of the specified goals
and objectives.
Needs Assessment Methods
The Instructional Systems Design approach was developed originally

for the training sector, and later adopted by the education sector. As a
result, the emphasis of this approach is on providing an instructional
solution to a performance problem or closing a performance gap.
While this approach can be used to produce instructional material for
skills-based educational programs, most programs in the education
sector are content or knowledge based. As a result, the processes and
questions you would ask in the analysis phase of ISD for developing
materials for the education sector would differ from those used in the
training sector. The main differences would primarily occur in the
analysis phase, particularly in needs assessment.
For example, how can you determine if your educational institution

should offer a program in a certain area? How would you determine
the curriculum for that program? How would the curriculum be
articulated into courses? What content would each course cover? The
answer to these questions and others are not straightforward, and will
often vary as a result of your beliefs and values, as well as the
institution’s mission and mandate.
Numerous approaches could be used to address these questions. The

course texts discuss several approaches for needs assessment, such as
the Discrepancy-based Approach, the Innovation approach, and the
Problem-finding, Problem-solving Approach. In addition, a number of
alternative approaches are discussed below, which also could answer
these needs assessment questions.
In the academic arm-chair approach, which is actually quite common,

no needs analysis is conducted. Typical statements that one might hear
from an academic using this approach would be “Learners need what
universities have to offer or they wouldn’t attend university,” or
“Students need a liberal education and they are in no position to define
what that would consist of.”
The job-skills approach is more common in vocational/technical

schools and community colleges than in universities. In this approach,
a needs assessment is conducted, and several groups (often referred to

as stakeholders) are assessed. Major stakeholders would likely be
members of professional organizations or actual or potential
employers of the graduates. The job-skills approach normally does not
ask whether a program is really needed, but focuses more on the types
of skills and knowledge required to do the job.
The labour market approach involves determining what types of jobs

are either currently in demand or forecasted to be in demand in the
future. Programs are then developed to meet these demands. Once the
fields of study have been determined, the curriculum is designed to
meet the needs of the job market.
The revenue generating approach considers what types of programs

and courses are in demand and thus might generate revenue. Revenue
generation is not the sole criteria for developing a new program;
however, in times of rising costs and shrinking budgets, new programs
are being developed to generate revenue or are developed on the basis
of a cost-recovery model.
In the mixed approach, a combination of one or more of the above

approaches is used to determine whether a new program should be
developed and to identify the curriculum for that program.
Using Existing Materials to Develop Instruction

Most distance learning materials that are produced for the educational
sector is not produced “from scratch” or “from the ground up.” The
most common development approach is to take existing resources and
to develop a study guide around them that provides directions and
any “missing ingredients” required to make the materials effective for
distance learning.
This approach has historically been referred to as the “wrap-around”

model. The use of existing materials and wrap-around models is often
adopted as a means to save time and money. More recently, the use of
learning objects has become a new way to produce instruction in a
wrap-around manner. This is the topic of the section that follows.
Learning Objects
With the advent of web-based instruction and e-learning, learning
objects have emerged as a way to achieve efficiencies in instructional
design and course development. Learning objects are very likely the
means by which instruction will be created in the future. Distance
education and instructional design will no doubt change greatly as the
use of learning objects becomes more commonplace.
Learning objects are the product of several diverse disciplines:

? computer science, which offered the concept of objects, which can be
constructed, shared, and assembled to serve different purposes;
? library science, which provided the concept of cataloguing and

using labels or tags to allow materials to be easily identified and
located; and
? instructional design, which provided a means by which flexible

instructional materials are created to achieve specific learning
outcomes.
Many definitions have been offered for the term “learning object,”
some so broad that they encompass almost anything related to
instruction. In the next reading, Wiley (2002) defines a learning object
as “any digital resource that can be reused to support learning.”
The critical features of learning objects are that they are digital,
reusable, and based on the attainment of one or more learning
outcomes.
The following reading identifies essential features of learning objects

and also discusses some instructional design issues associated with
these use. Go to the online book, The Instructional Use of Learning
Objects, located at http://www.reusability.org/read. Read the chapter,
“Connecting learning objects to instructional design theory: A
definition, a metaphor, and a taxonomy” by David Wiley.
Then, for a more in-depth look at how learning objects can be used,
and where they may be located, read the article “Learning Objects:
Resources for Distance education,” by Stephen Downes. This article is
located at http://www.irrodl.org/content/v2.1/downes.html
Finally, peruse the Special Issue on Learning Objects in the Canadian

Journal of Learning and Technology. This journal is located at
http://www.cjlt.ca/content/vol28.3.
Additional Resources
Learning Objects Resources page – This site contains papers,

templates, and a learning object on “What is a learning object?”
(http://www.alivetek.com/learningobjects)
“Learning Solutions – Learning Objects: Behind the Buzz” by Richard

Clark and Allison Rossett
(http://www.clomedia.com/content/templates/clo_feature.asp?articl
eid=24&zoneid=30)
”International E-learning specifications” by Norm Friesen and Rory

McGreal (http://www.irrodl.org/content/v3.2/tech11.html)

Assignment #1
Assignment #1 involves completing the analysis phase of the
instructional design process and developing a proposal for your
instructional unit or learning object. This assignment is worth 10% of
your course grade and should be submitted to your instructor for
grading and feedback after you have completed Unit 2.
Your answers to the questions below should be a total of about 4 to 5

pages in length (1,000 to 1,250 words).
Part 1: Needs Assessment (10 marks)
Completing an actual needs assessment is beyond the scope of this

course. However, for the purposes of this assignment, assume you
have completed a proper needs assessment.
1. Why is it important to do a needs assessment? What can a needs

assessment reveal? (Commentary for this unit; Morrison, Ross, &
Kemp, Chapter 2; Calder, pp. 67-72; Smith & Ragan, Chapter 3)
2. Which needs assessment method(s) did you use? Provide a

rationale for your selection. (Use the approaches described in the
Commentary for this unit, the approaches described in the
Morrison, Ross, & Kemp text, Chapter 2; the Calder text, pp. 67-72,
(particularly Figure 4.1); or the approaches described in the Smith
& Ragan text, pp. 32-36).
3. Based on the results of your (hypothetical) needs assessment,

identify the course or program that will meet the need(s)
expressed. Describe its components, i.e., the topics it will cover, the
major activities it will involve.
4. Describe the characteristics of the learner population for whom the

program, course, or learning object will be designed. (Morrison,
Ross, & Kemp, Chapter 3; Smith & Ragan, Chapter 4).
5. Describe the context in which the program, course, or learning

object will be delivered. (Morrison, Ross, & Kemp, Chapter 3;
Smith & Ragan, Chapter 4).
Part 2: Proposal

Following the guidelines listed below, identify a topic or subject area
for your unit of instruction or learning object. Clearly identify where
this unit fits within the course or program described in Question 3
above.
Guidelines
? Your unit of instruction or learning object may cover any topic
or subject area, but you should be quite familiar with the
content.
? The materials you develop should be digital (i.e., able to be
accessed by a computer), self-instructional, and designed to be
delivered at a distance.
? The unit of instruction should take a learner approximately one
hour to complete. If you are developing a learning object, you
should be aiming at a medium level of granularity.
If you are using a wrap-around model of development, provide a

complete bibliographic citation any existing materials you will use to
wrap-around in the development of your unit of instruction. It is
required that you incorporate at least one learning object in your
instructional unit.

Unit 3 Developing Your Instructional Unit
Introduction
Unit 3 involves completing the design and development phases of the
ISD process and creating the unit of instruction or learning object you
proposed in the previous unit. Throughout this process, you will keep
a Learning Journal to record your experience as you develop and hone
your instructional design skills, particularly in regard to the use and
development of learning objects.
In this unit, you will create the instructional materials that form the
basis of the course. You will examine various aspects of the design
phase in the instructional design process. Topics such as the 5 P’s of an
instructional strategy will be discussed, as well as instructional
elements (e.g., learning outcomes or objectives, advance organizers),
planning for interaction, maintaining motivation, student assessment,
and learner support.
Objectives
1. Apply the processes, procedures, and principles of instructional

systems design (ISD) in the design of a unit of instruction or
learning object.
2. Develop your instructional unit or learning object to include the 5

P’s of instruction: preparation; presentation; planned interaction;
practice with feedback; performance assessment.
Resources
In addition to reading this Study Guide, you should refer to the
following texts.

instruction (4th ed.). New York: Wiley. Chapters 5 – 11.

? Smith, P. L. & Ragan, T. J. (1999). Instructional design (2nd ed.).
Upper Saddle River, NJ: Prentice-Hall. Review chapters 5 through
18.
Commentary
The commentary below provides additional information that will help
you complete the assignment for this unit.
The Design Phase

Good instruction is cohesive. It has a beginning and an end. In
between, performance objectives are sequenced, information is
presented in a logical progression, and instructional activities are
orchestrated so that learning occurs in an effective and efficient
manner.
Underlying the instruction you create is your instructional strategy. An

instructional strategy is a plan. It consists of the instructional
components and procedures you will use to bring about the
predetermined learning outcomes.
As you begin developing your instructional unit or learning object,

start with the learning outcomes or objectives that you have identified.
Never lose sight of them throughout the remainder of the process (i.e.,
establishing design specifications and developing your instruction).
These predetermined goals are your measuring points; they determine
whether or not learners have achieved what you intended. To ensure
that these end points are achieved, you must develop an instructional
strategy that ultimately leads learners to this destination.
The instructional strategy you create provides the structure for your
course. It is constructed using several types of instructional
components. The basic building blocks of an instructional strategy
consist of the following;
? preparation,
? presentation
? planned interaction
? practice, with feedback;
? performance assessment.
These “5 P’s” are elaborated upon below.

Preparation
The first P – preparation – sets the stage for learning. It readies

learners for the instruction and prepares them for the new learning
they will achieve through the training program. In general, the
preparation component serves the following purposes in your
instructional strategy:
? To gain learners’ attention:: When presented early in the instruction,

questions, problem situations, dilemmas, and humorous or novel
situations help gain interest in the material to be covered.
? To establish a meaningful context for new learning: Early in the

instructional unit or learning object, it is important to emphasize the
relevance of the instruction in order to emphasize its value and
usefulness.
? To help recall previous learning: Stimulating recall allows students to

establish a base on which new learning will take place. It activates
the ”hooks” in the existing knowledge structure, so that the ”cups”
of new knowledge will ”hang” on them.
? To identify explicit outcomes that learners are expected to achieve as a

result of the instruction: Provide explicit, easily understood learning
objectives. This not only indicates learners’ destination, but your
expectations as well.
Presentation
The presentation component of your instructional strategy is the core

of your instructional product. This component presents the
information that students will learn.
Information should be presented in a clear, well-organized manner

that identifies distinctive features and that uses appropriate examples
and/or demonstrations.
Information should be presented in a simple-to-complex progression.

For example, simple explanations should proceed to more difficult
ones; clear-cut examples are followed by more complex examples.
Examples help learners understand concepts, principles, and

procedures better. An example should
? illustrate the ideas or concepts with real objects, situations, or

events;
? contain all the important characteristics of the idea or concept that

make it unique;

? vary in terms of difficulty in comparison with other examples (i.e.,
start with simple examples and progress to more difficult ones).
Clear presentations of basic information and examples form the

primary presentation component of your instructional strategy, as well
as other elements, such as the following:
? introductions;
? statements of facts;
? definitions;
? discussions;
? statements of principles;
? elaborative or background information;
? examples and non-examples;
? graphics and illustrations;
? procedure lists;
? case studies;
? summaries.
Presentations may be made using a variety of materials or media, for

example, through links to web-based resources, to selected articles in
periodicals or chapters in texts, to audio or video presentations, or to
computer simulations. Learning objects may also be selected and
assembled to present content in a coherent fashion.
Planned Interaction
The planned interaction component of an instructional strategy helps

learners process the newly acquired information in order to involve
them more actively in the material and to deepen their understanding.
It also involves activities that create interactivity between learners and
between learners and the instructor.
Too often, instructional designers adopt the ”sponge model” of

instruction. They expect learners to study passively, soaking up the
information being presented. A better strategy, however, requires
learners to actively participate in the instruction.
Active involvement is achieved by including the following elements in

your instructional strategy:

? synchronous or asynchronous audio or video conferencing;
? group activities;
? application or observation exercises;
? embedded questions;
? simulations or case studies.
Practice with Feedback
The fourth primary component of your instructional strategy –

practice with feedback – is based on a basic principle of behaviour:
People learn from their mistakes.
The practice with feedback component permits learners to practise

their newly acquired skills or apply their newly acquired knowledge.
In this way, they can determine whether or not they have achieved the
learning outcomes or objectives. Corrective feedback is provided to
remediate incorrect or inadequate learning. Feedback must be
informative; a simple correct or incorrect is not enough.
Performance Assessment
The final component of your instructional strategy–performance
assessment – provides an external measure and verification of the
learner’s capabilities in terms of the stated learning outcomes or
objectives. This component sometimes includes ongoing performance
assessment in the workplace to ensure that skills are being retained
and/or generalized.
Instructional elements often incorporated in this part of the

instructional strategy include written examinations or tests. These may
involve objective and subjective measures, e.g., short answer (written
or verbal response); multiple choice; matching; true/false; essay
(written or verbal response). Other types of performance assessment
strategies may involve projects, performance tests, or observation
checklists.
The Development Phase

At this point, you have reached the home stretch in the ISD process.
The time has now come to put pencil to paper (or fingers to the
keyboard) and begin developing your instructional unit or learning
object.

With your design specifications close at hand, you are finally ready to
begin developing your instructional unit or learning object – the
embodiment of your thorough analysis, well-designed instructional
strategy, and carefully considered presentation techniques, as well as
your unbridled creativity.
The development phase requires knowledge and skills in effective

writing, formatting, and production techniques to create a quality
learning object or instructional unit. Production may range in
complexity from simple print-based instructional materials, to complex
multimedia presentations.
Moisey, Susan D. (2001). An integrated instructional design model to

foster lasting behavior change. Educational Technology, 41(2), 60-62. You
may access this article from the Faculty Showcase page on the CDE
web site.
Imel, S. (1998). Technology and adult Learning: Current perspectives.

This article is located at the following URL:
http://ericacve.org/docgen.asp?tbl=digests&ID=50
Wilson, B. & Ryder, M. “Dynamic Learning Communities: An

alternative to designed instructional systems. You can access this
resource at: http://carbon.cudenver.edu/~mryder/dlc.html
Siemens, G. Instructional Design in E-learning. This article is located

at: http://www.elearnspace.org/Articles/InstructionalDesign.htm
Instructional Design for Online Course Development. This excellent

resource is located at:
http://www.ion.illinois.edu/IONresources/instructionalDesign/inde
x.html
Robler, M. D. (2000). How interactive are your distance courses? A

rubric for assessing interaction in distance learning. You can access this
article at: http://www.westga.edu/robler32.htm
Print in Distance Education This article is available at the following

URL: http://www.uidaho.edu/eo/dist7.html

Assignment #2
Assignment #2 involves developing a unit of instruction and
answering questions about applying instructional design principles
and using learning objects. This assignment is worth 45% of your
course grade and should be submitted to your instructor for grading
and feedback, after you have completed Unit 3.
Part 1: Unit of Instruction (30 marks)
Your instructional unit should be designed using the strategies

outlined in Designing effective instruction (2nd ed.) (Morrison, Ross, &
Kemp, 2003) and Instructional Design (2nd ed.) (Smith & Ragan, 1999).
The length of your instructional unit will vary, depending upon the
formatting, amount of original content, the media you incorporate, and
the mode in which it is presented. In a print, paper-based format, it
would be about 15 to 25 pages long; in a web-based format, it would
involve approximately 15-25 screens. Your unit of instruction or
learning object will be evaluated on the following basis.
Introduction Considerations (5 marks)
? Use preparatory elements, such as introductions, overviews,

advance organizers, and other elements effectively. These elements
should activate attention, establish instructional purpose, arouse
interest and motivation, and preview the lesson.
? The unit should include appropriately stated learning objectives or

outcomes.
Body Considerations (15 marks)
? Use appropriate instructional strategies to enhance instructional

effectiveness and foster learning.
? Organise and sequence the content effectively.
? Present content clearly and use examples effectively.
? Use a level and style of writing appropriate to the learner

population.

? Format pages to promote clarity of the presentation.
? Include a summary or conclusion to provide closure to the unit.
Interactivity (5 marks)
? Provide activities to foster interaction and active learner

involvement.
? Ensure that activities are engaging and interesting, promote

learning, and correspond to the objectives.
Learner Assessment (5 marks)
? Provide opportunities for practice and provide appropriate

assessment to ensure that learning outcomes have been attained.
? Provide useful feedback.
Part 2: Instructional Design Strategy (5 marks)

Describe the instructional strategy you used to structure the unit or
learning object and organize activities within it, as well as to foster the
type and level of learning identified in the learning outcomes or
objectives.
Your answer should be about 2 to 3 pages in length (500 to 700 words).

Integrate material from the texts or other resources to support your
answer.
Part 3: Learning Journal (10 marks)
As you develop the learning materials that comprise Assignment 2,

keep a journal to record and reflect on the process and experience of
using learning objects to produce instruction.
Write a paper (4-6 pages or 1000 – 1500 words in length) that reflects
on your experience with learning objects. Discuss the barriers and
facilitating factors that influenced you as you developed instruction
using learning objects. Identify problems or barriers you encountered
and how you solved or overcame them. Discuss what assisted you in
your instructional development process. Use your journal entries as
examples to illustrate your experience.

Unit 4 Introduction to Program Evaluation
Introduction
For the first time in the history of Athabasca University, 1994 marked
the year when the Government of Alberta reduced the University’s
operating grant. In fact, the entire post-secondary system experienced
major cut backs that year.
Beginning in 1994, all Alberta post-secondary institutions were cut by

21% over a three-year period. Athabasca University’s operating grant
was cut an additional 10%, a total of 31% over the same time period.
The primary reason given for why AU was cut more than the other
institutions was the government’s view that the University did not
produce enough graduates.
Over that same period, there was a marked change requiring

government-funded organizations to become more accountable. If
organizations did not meet the standards set in mandatory
performance indicators, further cuts were likely. Similarly, post-
secondary organizations that met or exceeded certain standards,
would receive “bonuses” in the form of performance grants or awards.
Although Athabasca University has been the recipient of increased
funding and numerous performance grants since 1994, it has still not
regained the level of funding it had prior to 1994. As you can see,
government influences have played (and continue to play) a major role
in determining the evaluation processes and priorities of post-
secondary educational institutions.
Thus, the subject of evaluation is a timely and important topic. In this

unit, you will begin your study of evaluation in distance education by
looking at the purposes of evaluation, the stages in the evaluation
process, and the approaches that can be used when conducting a
formal evaluation. The importance of evaluation for distance education
will also be examined. The unit concludes with an examination of the
types of data that most educational organizations collect and the
various ways in which such data can be used.
Objectives
1. Define the term evaluation and describe the purposes it serves.

2. Describe and provide examples of various types of program
evaluation.
3. Discuss the role of program evaluation in terms of achieving

effectiveness, efficiency, accountability, and quality.
4. Describe the role that performance indicators play in evaluation,

and give examples of types of data that may be gathered to reflect
program performance.
Resources
In addition to reading this Study Guide, you should refer to the
following texts.

instruction (4th ed.). New York: Wiley. Chapter 12.
? Chapters 1-3 in Programme Evaluation and Quality (Calder, 1994).
Study questions are provided below to guide your reading of the

Calder text.
Study Questions (Calder, Chapter 1)

1. List and briefly describe the eight stages of evaluation.
2. What is the aim of evaluation?
3. What is the relationship between being a “learning organization”

and evaluation?
4. What does Calder mean by the term programme evaluation?
5. Calder presents four groupings of types of goals that learning

providers can hold. Give an example of a goal from each of the four
groupings.
6. How does Calder differentiate between evaluation and

assessment?
7. Give examples of how the interest of the group sponsoring the

evaluation is important in determining the purpose of the
evaluation activity.
8. Briefly describe the difference between formative and summative

evaluation. Give an example which illustrates this difference.

9. As a formative evaluation methodology, what are some of the
problems with the pretest-posttest approach?
10. What advantages do approaches such as illuminative evaluation,

have over a pretest-posttest approach?
11. The final phase of evaluation is its utilization. Why is this phase
essential if an institution wishes to become a learning organization?
12. What does it mean to say that evaluation (regardless of the stage) is
not a value-free activity?
1. List five decision-making levels that evaluation activities need to

consider.
2. Briefly discuss the problems that you might encounter if you were
charged with the responsibility of developing performance
indicators for a distance learning organization.
3. List and describe the three types of organizational learning.
4. What is the difference between quality assurance, quality

assessment, and quality control?
1. Why do you think the importance of basic administrative records

(e.g., student records) has been overlooked as useful information
for evaluation?
2. Why is it important to know the types of problems and situations

that you would like your basic statistical data to inform before you
determine the types of statistics you will collect and analyze?
3. Outline the various methods and their related trade-offs for

collecting student demographic data. Which method do you
prefer? Explain your choice.
4. Why is monitoring student progress in distance education difficult,

yet important? How would you define drop-out and non-
completion at your institution or at Athabasca University?
5. Outline three ways that assessment data can be used as an

evaluation measure.

6. What milestones would you use to monitor student progress either
at your institution or at Athabasca University?
Commentary
Program evaluation, as a field of academic study, emerged during the
1960s — a time of significant educational reform and major social
service initiatives. Government funding over this period came with a
proviso that programs must be evaluated and that a certain portion of
budgets be allocated toward evaluation activities. Evaluation was soon
seen as an understudied area that lacked formal models, theory, and
skilled practitioners.
Certainly, evaluation as a concept and practice was not new. Accounts

of Chinese history show that workers were evaluated using a form of
standardized testing more than 3,000 years ago. Ancient Greek
scholars, such as Socrates, used a question-and-answer format to
ascertain knowledge of students. In the late 1800s, widespread
standardized testing of school students began to emerge. At about the
same time, there was increased interest in certification, and
accreditation of schools and colleges began to occur using formal
evaluation procedures. The first evaluation of students in terms of
curricular objectives took place in the 1930s by E. Smith and R. Tyler,
who later published their findings in Appraising and recording student
progress (1942). Two other important developments in evaluation also
occurred at that time. In the late 1940s, the Educational Testing Service
was established, which provided large-scale administration of
standardized tests in a variety of academic areas. Also, the
introduction of Bloom’s taxonomy of educational objectives in the mid-
1950s provided a systematic means of determining student outcomes.
But the systematic, formal evaluation of programs — the collection of

activities and materials plus the environment in which they exist,
which are designed to meet particular goals — was a new area, and the
1960s witnessed its genesis. The focus of evaluation expanded to
include process, the means by which student or organizational
performance was achieved.
In 1963, Lee Cronbach from Stanford University published the article

“Course improvement through evaluation,” which defined evaluation
as the collection and use of information to make decisions about an
educational program. He identified three types of decisions for which
evaluation could be used:
? decisions about course improvement, i.e., what instructional

materials and method are satisfactory and where change is needed;

? decisions about individuals, i.e., identifying the needs of students in
order to plan instruction, judging student merit for purposes of
selection and grouping, and acquainting students with their
progress and deficiencies; and
? administrative regulation, e.g., judging how good the school system

is and how good individual teachers are.
This broader concept of evaluation influenced the development of

several evaluation models. Three notable ones were the Stake Model,
Scriven’s Goal-Free Evaluation model, and the Stufflebeam (CIPP)
Model.
The Stake Model
Robert Stake’s (1967) article, “The countenance of educational

evaluation,” introduced a model of program evaluation based on
comprehensive observation and description. He urged evaluators to
provide a full description of the educational program, reporting data
on antecedents, transactions, and outcomes.
Antecedents were defined as conditions existing prior to teaching and

learning. Transactions were the encounters that students or clients have
with those involved in program – the succession of activities that
comprise the process of education. Outcomes were the consequences of
the program; for example, they include the abilities, achievements,
aspirations, and attitudes of students that result from the program.
Stake provided a framework for collecting these data and presenting

them in a description matrix (3 rows x 2 columns), allowing
comparisons to be made between intended and observed antecedents,
transactions, and outcomes. With this format, discrepancies or
congruence were readily apparent.
The model also included a similar format for constructing a judgement

matrix, where the evaluator reported information pertaining to
standards and judgements, as they related to antecedents, transactions,
and outcomes. Standards pertained to criteria that indicated acceptable
and meritorious levels for antecedents, transactions, and outcomes.
Judgement indicated the degree to which the program had achieved
these standards.
Stake’s model involved two major activities: description and

judgement of the program being evaluated. Due to the emphasis on
full description, judgements about the worth or value of the program
were well supported and useful for guiding future practice. Stake
concluded the article by cautioning that the “countenance of

evaluation should be one of data gathering that leads to decision-
making, not to trouble-making.”
Over the next several decades, Robert Stake’s contributions to the field
of program evaluation continued, as he further developed the theory
and methodology associated with producing “rich descriptions” of
programs being evaluated. In the 1980s, he introduced the concept of
“responsive evaluation,” which underscored the nature of evaluation
as a service for stakeholders, not a critical analysis judging the worth
or merit of the program. He continues to contribute to the field,
currently publishing on what he terms the “case study approach” to
program evaluation.
Recent articles on evaluation by Robert Stake are located under the

Resources section of the course web site.
Goal-Free Evaluation
In sharp contrast to Stake’s descriptive approach, Michael Scriven

advocated for (and continues to advocate for) judgement and the
determination of merit as the primary goal of program evaluation. In
“The Methodology of Evaluation” (1967), he distinguished between
the roles and goals of evaluation, and introduced two primary roles of
evaluation — formative and summative. He acknowledged the
importance of formative evaluation for the development of effective
educational programs, but emphasized that evaluators should not
shirk the responsibility of making judgements about the merit of the
program in their summative role. He recommends that only external
evaluators be used for summative evaluation.
Scriven introduced the concept of “goal-free” evaluation, asserting that

programs should not be evaluated according to their stated goals. He
argued that such goals may not be worth achieving and that it is the
evaluator’s responsibility to determine and make a judgement about
the merit of the goals of a program as well as its process and outcomes.
The Stufflebeam (CIPP) Model
Daniel Stufflebeam defined evaluation as the process of delineating,

obtaining, and providing useful information for judging decision
alternatives (Stufflebeam et al., 1971). His model, often termed the
CIPP (Context, Input, Process, Product) model, represented a decision-
management–oriented approach to educational evaluation. Four types
of evaluation were identified.
? Context evaluation provides a rationale for determining the

objectives of a program. It defines the relevant environment,

describes the desired and actual conditions pertaining to that
environment, identifies unmet needs, and identifies problems that
prevent needs from being met.
? Input evaluation provides information for determining how to use

resources to achieve the program objectives. It involves identifying
and assessing relevant organizational resources and capabilities, as
well as strategies and designs for implementing strategies.
? Process evaluation occurs when a program has been approved and

implementation has begun. It provides periodic feedback to those
responsible for implementing program plans and procedures.
Process evaluation serves to detect or predict defects in the design or
its implementation, to provide information for program decisions,
and to maintain documentation of the process as it occurs.
? Product evaluation involves the measurement and interpretation of

attainments at the end of the program cycle, as well as during the
program cycle if necessary. It involves developing operational
definitions of objectives, measuring them against criteria and pre-
determined standards, and interpreting outcomes using the context,
input, and process information.
In addition to identifying these four types of evaluation, Stufflebeam’s

model also described the following.
? Four decision-making settings for evaluation (homoeostasis,

metamorphism, incrementalism, and neomobilism).
? Four types of educational decisions (planning decisions to

determine objectives, structuring decisions to design procedures,
implementing decisions to utilize, control, and refine procedures,
and recycling decisions to judge and react to outcomes achieved).
Within this framework, evaluation activities are focused to provide the

best means of fostering effective decision making within the
organization.
Daniel Stufflebeam continues to contribute to the field of program

evaluation. In a recent article (Stufflebeam, 1994), he emphasizes the
importance of adhering to standards for evaluation studies and
practice. He cautions evaluators not to neglect their role as judges of
merit or worth, in favour of the self-determination role advocated by
newer approaches to program evaluation, particularly empowerment
evaluation.
Empowerment Evaluation

Empowerment evaluation is a form of self-evaluation that fosters
program improvement and self-determination. It is a collaborative
activity where program participants engage in program evaluation
activities. The role of the external evaluator (if there is one) is to coach
or facilitate the process.
Fetterman (1994; Fetterman, Kaftarian, & Wandersman, 1996) asserts

that merit and worth are not static entities, and therefore should not be
the focus of program evaluation. Rather, he argues, that by
internalizing self-evaluation processes within the organization, a
dynamic and responsive approach to evaluation can be developed so
that organizations can accommodate to shifts in the environment that
affect their programs. In this way, those involved in a program may
reshape plans and strategies to ensure viable, effective programs.
For further information, review the information on the web site located
at http://www.stanford.edu/~davidf/empowermentevaluation.html.
Schuemer, R. (eds.) (1998). Evaluation Concepts and Practice in

Selected Distance Education Institutions. Hagen, FRG: ZIFF,
FernUniversitat. This monograph may be downloaded from:
http://www.fernuni-hagen.de/ziff/108cont.htm
Levine, S. J. Evaluation in distance education. This chapter is located

at: http://www.learnerassociates.net/debook/evaluate.pdf
Kirkpatrick’s Four Levels of Evaluation. This resource is located at:

http://www.astd.org/virtual_community/comm_evaluation/focus_a
rchive/kirkpatrick.html
References
Cronbach, L. (1963). Course improvement through evaluation. Teachers
College Record, 64, 672-683.
Fetterman, D. (1994). Empowerment evaluation. Evaluation Practice,

15(1).
Fetterman, D. M., Kaftarian, S. J. & Wandersman, A. (Eds.) (1996).

Empowerment evaluation: Knowledge and tools for self-assessment and
accountability. Thousand Oaks, CA: Sage.
Scriven, M. (1967). The Methodology of Evaluation. In: R. W. Tyler, R.

M. Gagne, & M. Scriven (Eds.), Perspectives of curriculum evaluation
(pp. 39-83). Chicago, IL: Rand McNally.

Smith, E. & Tyler, R. (1942). Appraising and recording student progress.
New York: Harper and Row.
Stake, R. (1967). The countenance of educational evaluation. Teachers

College Record, 68, 523-540.
Stufflebeam, D. (1994). Empowerment evaluation, objectivist

evaluation, and evaluation standards: Where the future of
evaluation should not go and where it needs to go. Evaluation
Practice, 15(3), 321-338.
Stufflebeam, D., Foley, W., Gephart, W., Guba, E., Hammond, R.,
Merriman, H., & Provus, M. (1971). Educational evaluation and
decision-making. Itasca, IL: F. E. Peacock Publishers, Inc.

Unit 5 Program Evaluation Applications
Introduction
This unit examines the role that evaluation plays in program and
curriculum development, as well as in course design, development,
and delivery. The contribution of evaluation to the improvement of
student support services is also examined.
Throughout this unit, you will focus on program evaluation in terms of

the following key questions:
? Are the range and type of courses in the program adequate?
? How viable are the courses in the program?
? What is the quality of service that students experience in total within

the program?
The answers to these questions can contribute significantly to the

quality of organizational decision making.
Objectives
1. Define the term program, and give examples of programs in

distance education.
2. Discuss various ways in which the need for a program and its
viability may be ascertained.
3. Describe the stages in the program development cycle.
4. Discuss the role that evaluation can play in relation to the quality
of distance education courses at the design, development, and
delivery stages.
5. Define the term stakeholders, and describe the role they play in
program evaluation.

6. Explain why student support services are important aspects to
consider in program evaluation.
7. Apply an evaluation framework to assess instructional materials.
Resources
In addition to this Study Guide, you should read Chapters 4-6
(pp. 66-115) in the following text.
Calder J. (1994). Programme Evaluation and Quality. London: Kogan

Page.
1. How does Calder use the term program?
2. Identify the major stakeholders for program development and

evaluation.
3. List and describe five methods presented by Calder for identifying

learning/training needs.
4. The Commentary for Unit 2 of this course describes several

approaches for identifying learning needs for the purpose of
program and curriculum development. What parallels do you see
between the information provided in the Commentary and the
approaches listed in Calder’s text in Figure 4.1?
5. Briefly describe the issues to address and the questions to ask when
conducting a needs assessment for program development.
6. What does Calder mean by the term program viability?
7. What type of questions would you want to ask if you were in

charge of carrying out a program viability study?
8. Briefly describe what is meant by the terms piloting and pump-

priming . How do they relate to program viability?
9. What are the three strategic curriculum planning questions that all
learning providers should ask about program development?
1. Briefly describe Calder’s framework for a course evaluation design

(Figure 5.1).

2. Briefly describe the issues one would most likely address in an
evaluation project at the course design stage and at the course
development stage.
3. Briefly describe the six evaluation approaches presented by Calder

for evaluating a course at the development stage.
4. What are the three basic questions to ask in an evaluation study of

a course at the presentation stage?
5. List and briefly describe the five categories of topics on which

students could be asked to provide feedback.
6. Give an example that illustrates why a particular evaluation

question might be relevant for one study but not for another.
7. What are some of the concerns with relying on self-completion

questionnaire data for evaluation purposes? What other methods
might be used to collect feedback from students and other
stakeholders?
8. What are the issues surrounding timing, frequency, and duration

of feedback requests to students?
Study Questions (Chapter 6)

1. If you were in charge of an evaluation study of an institution’s
information and marketing services, what types of questions
would you want to ask to evaluate these services?
2. What type of evaluation questions would you ask if you were in

charge of evaluating admission services such as recruitment,
application and selection procedures for admission, and admission
procedures and processes?
3. Some distance education organizations have both central and

regional operations. What are the advantages and disadvantages of
local (regional) versus central evaluation studies?

Assignment #3
This assignment is worth 30% of your course grade and should be
submitted to your instructor for grading and feedback after you have
completed Unit 5 of the course.
In this assignment, you will design a program evaluation system

(Option A) using the three-stage process described in Chapters 7 to 9
of the Calder text, or you will participate in the collaborative
evaluation of a set of learning objects (Option B), using the Convergent
Participation Model (Nesbit, Belfer, & Vargo, 2002).
Complete either Option A or Option B.
Option A – Creating an Evaluation System

In this assignment you will design an evaluation system for the
program that includes the unit of instruction you prepared in
Assignment #2. In developing the evaluation system, follow Calder’s
three-stage process as outlined in Chapters 7 to 9 of the Calder text.
The final product of your efforts will be a set of answers to the

questions listed below. Address each question separately. Try to keep
your assignment to 20 (double-spaced) pages in length. The page
references listed beside each question direct you to the appropriate
part of Calder’s framework for addressing this item.
1. Describe the program and its components that will be the focus of
the evaluation system. In your answer, refer to Calder’s definition
of a program. (5 marks)
2. What is the purpose of your evaluation? Who are the major clients
of the system? Provide a rationale for your selection. (Calder, pp.
117-119) (5 marks)
3. What is the focus of the evaluation? Provide a rationale for your

decision (formative, summative, or some combination). (pp. 22-23)
(2 marks)
4. Who will be designing and carrying out the work of the

evaluation? Provide a rationale for your selection. (p. 120) (2
marks)

5. What approach or combination of approaches will be used (e.g.,
participative evaluation, self-evaluation, etc.)? Provide a rationale
for your selection. (pp. 121-122) (2 marks)
6. Discuss the need for policy development and the issues that need
to be addressed by these policies. (pp. 123-124) (3 marks)
7. What model will you follow to determine the priorities of your

evaluation system (e.g., empowered individual, advisory
committee, etc.)? Provide a rationale for your selection. (pp. 124-
127) (2 marks)
8. Briefly describe the type of data that you would want to collect on
a routine basis. (pp. 132-134) (3 marks)
9. What type of monitoring studies would you want to carry out on a

regular basis? Give examples of these studies. (pp. 134-137)
(3 marks)
10. Describe additional ad hoc studies that you might want to conduct
specifically to evaluate the “new” program. (pp. 137-138) (2 marks)
11. Briefly outline the management, staffing, and resource issues you
will need to address before implementing your system. (pp. 138-
142) (3 marks)
12. Briefly outline the issues and questions that need to be addressed
in setting up a self-evaluation of your evaluation system. (pp. 143-
154) (3 marks)

Option B – Learning Objects Evaluation
In this assignment you will apply the process described in the article,
“A Convergent Participation Model for Evaluation of Learning
Objects” (Nesbit, Belfer, & Vargo, 2002), which is located in the
Readings as well as at the following address:
http://www.cjlt.ca/content/vol28.3/nesbit_etal.html
To complete this assignment, follow the procedure outlined below.

a. Assemble a panel of four participants.
b. Each participant will select a learning object to be evaluated
(and will later serve as the moderator for that object’s
evaluation and publish an integrated review of that object).
Note: Inform the instructor so that appropriate computer
conferences can be established.
c. In the first cycle, each participant will individually and
asynchronously evaluate each learning object using the LORI
evaluation format presented in the article. The evaluations will
then be submitted to the appropriate moderator (i.e., the panel
member who originally selected the learning object).
d. In the second cycle, the moderator will integrate the four
reviews and provide them to the panel members. He or she will
then moderate a discussion, over a one-week period, during
which time points of disagreement will be discussed.
e. At the end of the discussion period, the moderator will bring
the conference to a close and create an integrated panel review
consisting of the integrated ratings and comments. This review
will then be submitted to the instructor. It may also be
submitted to the learning object repository as a review of that
learning object.
Each participant should submit the following materials to the

instructor for review and grading. Try to keep your assignment to 10 -
15 pages in length.
1. Cycle 1 Reviews -- Provide a copy of each review you wrote in the

first cycle (i.e., for your own learning object and for the others).
There should be four reviews in total. (10 marks)
2. Cycle 2 Discussion -- Reflect on your experience as a moderator.

Review the conference discussion of the evaluation of your
learning object in the second cycle of the review. What were the
major areas of difference? How did you resolve them? What went

well? What could have been improved? Include examples from the
conference discussion to illustrate your answers. (10 marks)
3. Integrated Review – Using the LORI evaluation format presented

in the article, write an integrated review based upon the input of
the other panel members. Include the range and central tendency
of individual ratings, with comments included within the different
evaluative categories. (10 marks)


MDDE 604: Instructional Design and Program Evaluation in Distance Education

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MDDE 604: Instructional Design and Program Evaluation in Distance Education

Uploaded by

Copyright:

Available Formats

MDDE 604: Instructional Design and Program

Evaluation in Distance Education

In Assignment #1, you will complete the analysis stage of the

Master of Distance Education 604 / Study Guide 1

In Assignment #2, you will develop a unit of instruction and discuss

Assignment #3 involves the application or course of program

Computer Conferences: There are four required computer

? This Study Guide for MDDE 604

? A set of articles in the Course Readings:

Fletcher, J. D. (1992). Individualized systems of instruction. (American

Romiszowski, A. (1994). Individualization of teaching and learning:

Nesbit, J., Belfer, J., & Vargo, J. (2002). A convergent participation

2 Instructional Design and Program Evaluation in Distance Education (Fall 2003)

Calder, J. (1994). Programme Evaluation and Quality. London: Kogan

Smith, P. L. & Ragan, T. J. (1999). Instructional design (2nd ed.). Upper

Master of Distance Education 604 / Study Guide 3

This unit examines various attributes of instructional systems and

4. Discuss how the following factors associated with individualized

5. Discuss the strengths and limitations of applying the principles of

4 Instructional Design and Program Evaluation in Distance Education (Fall 2003)

? Individualized Systems of Instruction by J.D. Fletcher

? Individualization of Teaching and Learning: Where Have We Been; Where

Also read the following online articles:

? An Updated and Theoretical Rationale for Interaction by Terry

? Using students’ learning styles to provide support in distance education

Optional Activity: Complete the Learning Styles Questionnaire at

Study Questions (Fletcher Article)

1. Define the term individualized instruction.

2. Explain the following statement: “Individualized systems of

3. Fletcher identifies four components of an instructional system that

Master of Distance Education 604 / Study Guide 5

Study Questions (Romiszowski Article)

1. What factors have led to the adoption of individualized approaches

2. Give three examples of different approaches to individualized

3. Will new technological initiatives in individualized instruction

4. What are the four principal components of a teaching-learning

Study Questions (Ally and Fahy Article)

2. Describe the following types of learning styles: diverger;

Study Questions (Anderson Article)

1. Identify and describe the various modes of interaction.

2. Consider the following statement: “After years of sometimes

6 Instructional Design and Program Evaluation in Distance Education (Fall 2003)

Master of Distance Education 604 / Study Guide 7

What is individualized or flexible instruction?

Individualized instruction is the use of a variety of methods and

Individualized instruction promotes learner autonomy. For example, it

Where did individualized instruction originate?

Individualized instruction is by no means new. The first example was

Movements toward individualizing and personalizing education have

8 Instructional Design and Program Evaluation in Distance Education (Fall 2003)

The growth of instructional systems design, which began in the 1960s,

What attributes can be individualized?

The pace of instruction can be placed entirely under the learner’s

Much is said and argued in the literature about the importance of

Master of Distance Education 604 / Study Guide 9

Sufficient levels of deep and meaningful learning can be developed as long as

An important issue associated with individualization of any of these

Many educational applications involve group-based instruction

On the other hand, in an individualized system, students may progress

Concepts Underlying Individualized Instruction

In most classroom or group-based learning situations, curriculum and

10 Instructional Design and Program Evaluation in Distance Education (Fall 2003)

In contrast, individualized instructional systems assume that students