INSTITUTE OF DISTANCE AND LEARNING

UNIVERSITY OF MUMBAI, SANTACRUZE

MA Education Programme
Assignment Details

Full Name of the Student: MARIYA BANO MOHAMMED RIZWAN

sssssssssssssss

Academic Year: _2022-2023 _sss

Admission Application No.: 152648 sss

Email ID: S.Kmariya101010@gmail.com sssssssssssssssssssssssssssssssssssssssssssss

Paper Name: OPEN AND DISTANCE LEARNING

sssssssssssss

Paper Code: EC 6

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss

Assignment Total No. of Pages:

16_______________________________ssssssssssssss

Language of Writing Assignment: ENGLISH

sssssssssssssssssssssssssssssssssssssssss
Assignment Submission Date: 28/10/2022

ssssssssssssssssssssssssssssssssssssssssssssss

Signature of the Student:

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
Q.1 What is SLM? How it is different from a textbook. Explain the stages of
development. of course in SLM.

INTRODUCTION

Structure:

The self learning material is to tackle the subject related difficulties faced by you, so
that you may not have to face any problem, while teaching in the class. Although these
hard facts have been identified by the teachers themselves, it may be possible that you
might have faced during teaching the subject may not have been included in this
material. SLM is an important component of distance education. Each unit begins with
an introduction. The objectives tell us about the knowledge that we shall acquire after
the study of the unit and what effective changes would be there in the learning process.
After every section and subsection, "self check exercises" including provision for
answering the questions in the units are given where in questions have been asked about
the text of the unit. One should expect to go ahead only after attempting the small
exercises. Appropriate margin or space has been provided in each page for writing
notes. One may use it, so that knowledge or skills are enhanced.

Efforts have been made to provide appropriate pictures, diagrams etc in the booklet so
that one may be able to teach the students effectively. At the end of each unit, summary
of the unit has been included in the form of main points under the title 'we have learnt'
and after that exercises and answers to "self check exercises" are given. One should
expect to check the answers only after writing the answers of these questions.

1
 Suggested
reading
SLM Self-Exercise
Reading
comprehension
Progress
overview
We have Writing
learnt

Summary

Diagram 1: Study Skills in SLM

Study Skills in SLM :

Depending upon various needs, different thinkers give different labels or definitions to
the term study skill. The labels commonly associated with study skills are: a tool kit, the
best ways of acquiring knowledge, a set of organization skills, micro skills and
systematic study habits and so on. For some study skills, strategies are to be developed
by a learner to derive the greatest possible benefits from activities like listening,
speaking, reading and writing. For other study skills are "advanced skills" which are not
purely mechanical but essentially involved some amount of creative thinking. These
skills acquire active mental exercise on the part of the learner. The essential skills that a
learner requires in order to gain maximum benefits from his/her studies. Study skills are
the skills on strategies that a learner employs to study and come to grip with the study
materials independently and efficiently. It is however difficult to categorically specify
all the skills that a learner uses either to get across his/her message or to decode some
one's message.

The term 'Study' refers to -

2
  Follow a course of lectures and taking notes.
 Being acquainted with and being taught all that is necessary to know about the
subject.
 Cramming chunks out of or the whole of subject matter.

The dedicating and systematic pursuit of understanding and dedicating one's thoughts
and energies to learning study involves in setting up goals and choosing methods,
solving problems, performing experiments, going through tests, collecting information,
segregating facts from opinions or suggestions, comparing facts, weighing up opinions
and considering suggestions and finally looking for proof and truth. 'Study' thus
demands you to analyze and criticize not only your own ideas but also those of other
people - whether you hear them in lectures or discussions or read them on books. You
will have to further make brief but efficient notes and summarizes to help yourself
remember what you have read and to clarify your thinking.

Study Skills and Distance Teaching/Learning:

It should be highlighted here how essential study skills are for a system in which
teachers and learners are at a distance for most of the teaching learning process. The
means used in distance education to improve the study skills of distance learners. Study
strategies play a vital role in distance education. The learner being away from the
teacher and the Institution, depends mainly upon his / her course materials for her/his
studies. If he/she has not developed proper study skills to tackle the text she/he is
certainly at a disadvantage. A distance learner's study depends also on audio-video
material. Again how efficiently he / she deduces meaning out of them primarily depends
on how well one develop study strategies.

Significance of Study Skills :

Learners should be helped to acquire good study strategies as early as possible certainly
before they develop either poor study habits or wrong concepts concerning studying.
Teaching study skills would help the learners spend less time in studying and learn

3
more. Studying requires a certain amount of tension, concentration and in a specific
direction. Of course, the amount of tension varies with different individuals. Three vital
concerns in building good study habits. These are when to study?, where to study? How
much time to spend on study?

When to Study?

Some learners study only just before the terminal examination or an announced test.
Some may even study the whole night and cram answers. All of use have probably done
this at least once or twice. Rote learning, however does not bring about any real
learning.

A good learner must plan one's study time and spread over a period of time. Of course
the ability to study and more so, the ability to plan our studies is not something we are
born with. It is a set of skills that must be learned. Teaching the learners to plan their
study is therefore a major responsibility of the teacher. Learners should be made to
realize that regular planning would prevent confusion and help them to retain and
organize what they are studying.

In a class room situation, for example, the learner may be advised to study each subject
as close as possible to the class, period for the subject before or after the class period.
This chiefly depends on the form that, class period usually takes. That is, if it is a
straight forward informational lecture a study session right after the lecture will be
useful to review notes and check whether it has been understood. A study session just
before a seminar / discussion gives learners a chance to read upon the background
information that will help them make an effective contribution in the classroom.

Where to Study?

Ideally you should be able to study any where in a quiet library or on a crowded bus or
train. But let us be realistic. Most of us can not entirely shut off our minds to
distractions. By implication, the place of study, should be as free as possible from such
surroundings as might steal away the attention of the learner. As for the conditions for
4
study, the place should be well lit and properly ventilated and neither too hot or too
cold. We should also mention here that it can be difficult to get every thing the learner
wants, for example, if staying in a hostel or some one else's house he/she can not find
everything at his/her own choice.

Additional Features of self-learning materials

 Learning
 Individual Self-paced
 Private learning
 Any time
 Any number
 Standardized
 Expert content
 Updatable content
 Structured teaching
 Active Learning
 Frequent feedback

Comparison of textbooks with self-learning materials

Textbook Self-learning material

Assume interest Arouse interest

Written mainly for teacher use Written primarily for learner use

Do not indicate study time Give estimates of study time

Designed for a wider market Designed for a particular learner group

Rarely state aims and objectives Always gives aims and objectives

Structured for teachers and specialist Structured according to the need of

learners

Little or no self-assessment Major emphasis on self-assessment

5
 Modes for Development of Self-Learning Material

Self-learning materials: the concept

The success and effectiveness of the ODL system largely depends on self-learning
materials (SLMs). Developing SLMS is a challenging task, and is quite different from
that of face-to-face teaching or writing a textbook and/or journal article. It is pertinent
for you to understand the concept of self-learning materials which are based on the
principles of use of the means and ways of communication.

Concept of learning at a distance

As you have studied in the preceding block, the distance learners pursue their study on
their own for they are away from their teachers, institution, and fellow learners. They
learn independently with minimum external support from teachers and peers. Moreover,
a majority of the distance learners is usually adults; and they need learning materials,
which facilitate their independent learning. Normally they are motivated enough and
also possess certain study skills which enable them to pursue their study independently
and at their own pace.

As you know, in the ODL system the scope for personal contact with the learners is
limited. Most learning takes place not through the mediation of a teacher but primarily
through the mediation of the learning materials. Therefore, learning at a distance
demands study skills on the part of the learners to enable them to gain optimally from
the learning materials. Usually learning at a distance takes place and gets reinforced by:

 interacting with learning materials;

 receiving feedback on various academic activities, such as assignment
responses, project work, term-end examinations, etc.;,.

6
  attending personal contact communication during counseling sessions and
tutorials;
 using multiple media and multimedia components (cassettes, broadcast,
interactive CD-ROMs, teleconference, internet, etc.);
 participating in face-to-face academic activities (seminars, counseling sessions,
tutorials, laboratory work, etc.).

Self-learning materials (SLMs) perform the functions of an effective classroom teacher

by providing learning experiences similar to the classroom-based teaching learning
process. Thus, the invisible teacher built in the learning materials facilitates the learners
in their studies in the same way as the classroom teacher does within the face-to-face
classes:

Instructional design versus learning

Instructional design is a process comprising analysis of learning needs and goals and the
development of a delivery system to meet these needs. It is concerned with research
about instructional strategies and the processes for developing and implementing those
strategies. It is the science of creating strategies for development, implementation,
evaluation, and maintenance of environment situations that can facilitate learning.
Instructional design involves systematic development of instructional specifications
using learning and instructional theory to ensure quality teaching and learning. SLMs
are based on instructional design involving analysis of learning needs, development of
learning materials, and placement of a delivery system and the mechanism of
assessment to know whether the learners have achieved the course objectives. The
quality of instructional design has a direct link with learning on the part of learners. The
more innovative an instructional design is the more effective and learner-oriented the
SLM shall be. This is so because there is a direct link between quality of instructional
design and learning.

Let us now examine this relationship between instructional design and learning, and
hence the implications of instructional design for developing effective self-learning

7
materials. In the conventional system of education, the learners get most of their
instruction through the face-to-face interaction with the teacher and the peer groups.
But, the learners pursuing their study through the ODL system use the specially
designed learning/training materials. Here, their learning depends on the
effectiveness/quality of materials prepared or transformed for a particular target group.
In this case, the learners interact with the content presented in the forms of printed text,
audio-video materials, computer-enabled instruction, etc. and the quality of these
learning materials do enhance student learning provided they are developed on the
principles of effective learning and the needs of learners. Therefore, learning materials
must be based on effective instructional design and needs and characteristics of the
targeted learners.

Characteristics of self-learning materials

Self-learning materials perform the functions of an effective teacher who guides,

motivates, explains, discusses, asks questions, assesses progress, suggests appropriate
remedial measures, and provides advice to learners. Self-learning materials emphasize
pedagogical dialogues with the learner. While reading the course units, the learners
interact with an invisible teacher (in a simulation situation since the teacher is simulated
in the materials) and feel as if they are being taught by him/ her. The materials also
provide questions for self-check and thus increase curiosity of the learners. The success
of self-learning depends on the quality of the learning materials.

Self-learning materials follow learner-centered approaches. They are designed and

developed as per the needs of the learners. The following are some of the characteristics
of self-learning materials:

Self-contained

Efforts are made to make the text self-sufficient so that a learner does not hunt for the
additional sources, or even a teacher. For this the scope of the content of the unit 'is
visualized in detail. While avoiding what is redundant only the essential details are

8
presented so that the unit can cover information required by the learners and keep away
all that is superfluous or redundant.

Self-explanatory

The content is presented in a style so that a learner can learn from the material without
much external support. The concepts are explained to the extent such that majority of
the learners are able to comprehend them. Therefore the content is self-explanatory and
conceptually clear. To make concepts self-explanatory the content is analyzed and
presented logically considering the mental and lingual background of the learners1
target group. A few may not be able to comprehend the contents fully and may therefore
need additional help and guidance by teachers through correspondence and at study
centre tutorials.

Self-directed

The learning materials aim at providing necessary guidance, hints and suggestions to the
learners at each stage of learning. The self-directed material is presented in the form of
easy explanations, sequential development, illustrations, learning activities, etc., thus
performing the role of a teacher who for instance guide, instruct, moderate and regulate
the learning process in classroom situations.

Self-motivating

In distance education systems, the learners remain off the campus for most of their
study time. The study materials, like a teacher in the classroom, should be highly
encouraging for the learners. The materials should arouse curiosity, raise problems,
relate knowledge to familiar situations and make the entire learning meaningful for
them, providing reinforcement and feedback at every stage of learning.

Self-evaluating

As the learners remain separated from the distance learning institution as well as the
teachers, the study materials should make provisions for feedback as well. To ensure

9
optimum learning, the learners should know whether they are on the right track. Self
evaluation in the form of self check questions, exercises, activities, etc. provides the
learners with the much needed feedback about their progress, reinforces learning, and
motivates them for self-learning. The course writer should develop a built-in evaluation
system by giving an appropriate number of self-check exercises, activities and questions
in the course units.

Self-learning

Self-instructional materials are based on the principles of self-learning. So, a unit,

besides information, provides the learners study guide directions, hints, references, etc.
to facilitate their independent learning. To make the content comprehensible, it is
supported by simple explanations, examples, illustrations, activities and so on. In other
words, the materials are designed and, developed in such a way that the learners can
undertake learning by oneself

10
Q.2 Explain Need of Distance Education with INDIAN context and Enlist the role
of Distance Teacher, Editor, Course Writer in Distance.

WHAT IS DISTANCE EDUCATION

What, then, is distance education? There is no one meaning of the term distance
education. it is known by variety of names, viz. a 'Correspondence Education', 'Home
Study', Independent study', 'External study', 'Off-campus Study', 'Open Learning', 'Open
education, etc. In Australia, its official name is External system. "This description is not
very much appreciated because it carries vibes of old London external system which
usually provides examination but not teaching." This system is prevalent in India also.
In some countries, the term 'Correspondence Education' is widely used. But it has
increasingly been replaced by the term

'Independent study' in North America. 'Home study' is sometimes used to describe

correspondence programmes of private schools both in North America and Europe. This
is also disliked by many mainly because some of the correspondence institutions have
only profit motive, particularly the private ones. No doubt, some of them enjoy very
good reputation, but several of them are far from it. Further, several correspondence
institutions depend only on one medium, viz. the print material. The distance education
today lays emphasis not only on print medium but also on other media. In fact, multi-
media approach is the corner-stone of distance education systems.

In France, it is known as 'tele-enseignement'. In Germany, it is described as

'Frenstudium' or 'Fernunterricht'. In Spanish speaking countries, it is described as
'education a distancia'. 'Off- campus' is a term which is popularly used for distance
education in Australia. 'Extra mural' refers to distance education in New Zealand. These
terms have come into vogue because of the historical circumstance in various countries.
They have peculiar characteristics of their own, most of which are similar to distance
education. for instance, all of them emphasise separation of the student from the teacher.

11
In India, we have been using three terms: 'External Appearance' (private appearance),
'Correspondence Education', and 'Distance Education'. I would not regard the first as
distance education, because, here a university permits a student to take the examination
as private candidate, and if he passes, he is given the degree. The university does not
take any responsibility to impart education to the student. As a result, he is one his own
and very often at the mercy of the mercenary tutorial institutions. Since university does
not give any education, what is being done under external appearance can best be called
'private study' but not distance education, the second, correspondence education, has
been quite popular in this country. Lately some of them have designated themselves as
Distance Education and Open University System.

In this connection, I would like to invite your attention to the two terms which are used
inter-changeably and about which there has been some controversy. They are 'Open
learning' and 'Distance Education". "Open Learning' covers "a wide range of
innovations and reforms in the educational sector." Included are changes that aim to
improve such things as the participation of learners, instructional design, methods of
transmitting information and support of learners. A very comprehensive document
entitled "Open Learning" by Mackenzie, Postgate and Scupham which was brought out
by the UNESCO in 1975 describes open learning as follows: "Such systems are
designed to offer opportunities for part-time study, for learning at a distance and for
innovations in the curriculum. They are intended to allow access to wider section of
adult population, to enable students to compensate for lost opportunities in the past or to
acquire new skills and qualifications for the future. Open learning systems aim to
redress social or educational inequality and to offer opportunities not provided by
conventional colleges or universities." In this system, restrictions on learning are few
than those in formal educational institutions. Educational opportunities are planned
deliberately to that access to knowledge is available to individuals in spite of barriers
such as geographical distance. It is argued by some that while distance education is
accommodated by an open learning system, the opposite premise may not be true. As
Ruggles and his associates argue: "Not all the distance education programmes have the
characteristics of an open learning system. Some are very rigid and inflexible." In their

12
view, this term may symbolise a new and emerging philosophy that will become more
closely associated with learning at a distance.

"The term 'open' generally refers to four aspects: (i) People, where it would not debar
applicants on account of their lack of educational qualifications; (ii) Place, in the sense
that learning would be home based and not restricted to class rooms or a campus: (iii)
The use of new methods of teaching: and (iv) Ideas. However, writers on the subject
find the term 'open learning' or 'distance education' unsatisfactory.

Doubtless, there are several similarities between 'open learning' and 'distance education'.
However the term 'open learning' conveys certain amount of vagueness about it.
Moreover, when we look at the open universities and open learning institutions, we find
all that is conveyed by 'openness' is not to be found in quite a few of these institutions.
Open learning, therefore, is always synonymous with Open University or distance
education. Several open universities and open learning institutions prescribe certain
entry qualifications and they debar those who do not fulfil the conditions laid down by
them. Therefore, if the idea is to provide educational facilities to larger number of
people scattered all over, i.e. providing greater access to education, the term 'distance
education' would be more appropriate. This term basically emphasises separation of the
teacher and the learner, and planning of educational programmes and material by an
educational organisation and use of technical media on a large scale.

Distance education has the following important characteristics:

 The quasi-permanent separation of teacher and learner throughout the length of

the learning process; this distinguishes it from conventional face-to-face
education.
 The influence of an educational organisation both in planning and preparation of
learning materials and in the provision of student support services; this
distinguishes it from private study and teach yourself programmes.
 The use of technical media; print, audio, video or computer, to unite teachers and
learners and carry the content of the course.

13
  The provision of two-way communication so that the student may benefit from or
even initiate a dialogue; this distinguishes it from other uses of technology in
education.
 The quasi-permanent absence of a learning group throughout the length of the
learning process so that people are usually taught as individuals and not in groups,
with the possibility of occasional meetings for both didactic and socialisation
purposes.

Need of Distance Education

Generally individuals take distance mode for advanced education while they are
working. Their primary thought process behind picking the distance education is to
redesign abilities and incorporated the equivalent at work. Yet, it doesn't imply that
understudy who isn't presently working can't take the distance course. Distance
education is cheap and gives freedoms to applicants to upgrade their capability and
development in profession choices.

Advantages of Distance Learning

 Inexpensive
 No age bar
 Opted from any spot and from any foundation.
 Working class can likewise take distance learning for upgrading their profession
possibilities.
 As a distance education understudy, you are liberated from study burden and
pressure of going to full-time classes.

Here are a few different ways where distance education can help the country:

1. Take up an Important Job: Distance learning gives you a relative bit of leeway
over the long haul. Occupations that make an ideal commitment towards the country are
consistently a matter of need. Distance learning assists the understudies with adaptable

14
examining hours which can assist you with continueing the work of your premium with
no deterrent and can contribute towards the country's general turn of events.

2. More Skilled Individuals Lead to More Development: Distance learning causes

you to redesign your abilities as well as encourages you to procure more. This, thusly,
will assist you with duplicating the nations gifted labor force which in the lieu makes a
more noteworthy commitment to the GDP of the country.

3. Enjoy a Sport: Many drop their games vocation as they need to proceed with their
investigations. Indeed, distance learning can assist you with satisfying your fantasies
with no deterrent. You can rehearse your game and can address your country at the
global level without dropping your education. This will assist the country with
delivering worldwide athlete.

4. Advantageous Education Medium for Women: Some Indian families have a thin
mentality with respect to ladies' education. Accordingly a few ladies are compelled to
drop their education in the midway and are made to wed without finishing their
education. Open education gives ladies a brilliant opportunity to concentrate again and
accomplish their fantasies and add to the development of the country.

5. Multi-Talented Professionals: Correspondence Education causes understudies to

take up different courses that are not open in the schools from their nearby
topographical region. For instance, an understudy from Chennai can apply in any Delhi
schools that give Correspondence Education. The understudy can finish his course by
sitting in Chennai. This can assist with making multi-gifted experts.

6. Pocket-Friendly Fee Structure: Distance education is consistently affordable in

nature as it doesn't trouble the understudies and their folks. These assist the understudy
with putting their assets into some other beneficial medium that could help the country
in without a doubt.

15
What is your role as an editor of distance learning materials?

Your role as an editor of distance learning materials is very important, as it covers all
stages of course development: initiation, development, production and even post-
production activities. You are expected to produce materials which are self-instructional
and learner friendly. But you cannot do this alone. You have to work with teams of
course writers with different abilities, educational technologists, media specialists and
institutional administrators. You are responsible for the academic and pedagogical
quality of the material; which means, you have to check the content, language and the
instructional design of the materials.

You should be able to perform all the above mentioned functions yourself, as an editor.
You ought to have the necessary competence, skills and involvement to do that, or you
must manage to get the tasks done through others. Of course, you need the necessary
resources and infrastructure from the distance education institution which asks you to
work as an editor.

16
 INSTITUTE OF DISTANCE AND LEARNING
UNIVERSITY OF MUMBAI, SANTACRUZE

MA Education Programme
Assignment Details

Full Name of the Student: MARIYA BANO MOHAMMED RIZWAN

sssssssssssssss

Academic Year: _2022-2023 _sss

Admission Application No.: 152648 sss

Email ID: S.Kmariya101010@gmail.com sssssssssssssssssssssssssssssssssssssssssssss

Paper Name: OPEN AND DISTANCE LEARNING

sssssssssssss

Paper Code: EC 6

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss

Assignment Total No. of Pages:

14_______________________________ssssssssssssss

Language of Writing Assignment: ENGLISH

sssssssssssssssssssssssssssssssssssssssss
Assignment Submission Date: 28/10/2022

ssssssssssssssssssssssssssssssssssssssssssssss

Signature of the Student:

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
Q.1 Evaluate the types of Open and Distance Learning Evaluation and Explain
each evaluation type critically.

INTRODUCTION

The drive for transparancy and public accountability in the UK's public sector has had a
far-reaching impact on Higher Education. Part of this impact has been an increased
emphasis on evaluation (Oliver, 2000). However, the drive to evaluate has not been
matched by support and training for the practitioners who are supposed to carry out
these processes (see, e.g., Phelps et al., 1999).

In response to this, several initiatives have been implemented to provide practitioners

with support, such as the development of toolkits (Conole et al., 2000), cookbooks
(Harvey, 1998) or manuals of advice and guidance (Phillips et al., 1999). What these
resources lack, however, is specific advice on evaluating online learning and teaching.
Consequently, this article will include a review of the issues specific to this domain,
supplemented by illustrative cases. To structure this, however, it is necessary to
elaborate the reasons for evaluating online learning and teaching.

Background

The characteristics of distance learning and of online learning and teaching

As noted above, many discussions in the literature address generic issues of evaluating
learning technology rather than concentrating on the particular characteristics of online
learning and teaching. However, it is important to take these into account when
designing and implementing an evaluation. In order to do this, the charactersitics of
distance learning will be outlined, and then extended by a consideration of online
learning and teaching.

Peters (1998) identifies several distinctions betwen distance learning and traditional
forms of study. These include:

1
  A shift from an elitist model to mass higher education

 A move towards increasingly structured and planned programmes of study

 The industrialisation of course development, including the division of labour

amongst teams of specialists

 The challenge of maintaining dialogue as a central component of distance

courses

 The loss of informal opportunities for learning, for example in social settings

Importantly, although physical distance is taken into account in his analysis, Peters is
primarily concerned with "distance pedagogics". Such techniques as broadcast lectures,
because they add nothing to traditional forms of teaching, are grouped with traditional
teaching techniques. By contrast, open learning self-instructional materials are
considered alongside distance learning techniques, even when the specific packs are
distributed to campus-based students. In this paper, however, physical distance will be
considered, since it introduces a range of pragmatic problems to evaluation

Online learning and teaching is harder to characterise, due to its relatively short history
and diverse forms of implementation. Clearly, most online programmes will mirror the
characterisation of distance learning provided above. Importantly, however, several new
characteristics may also be added. These include:

 Technical requirements

 Skills requirements

 The breadth and use of different media

These characteristics are equally important for staff and students involved in online
learning (See, eg, Salmon, 2000.)

2
Another important aspect of online learning and teaching is that many systems
automatically log use. Particularly important is the way in which online discussion -
compared closely to traditional correspondence learning by Peters (1998) - is recorded
in full. As will be seen, this is of considerable use to evaluators.

Evaluating learning technology

The term 'evaluation' refers to a wide-ranging collection of methodologies, and is also

used to cover review processes such as checklists as well as empirical judgements. It is
worth noting that the term is sometimes confused with assessment (Phillips et al., 1999);
however, in this article, it is taken to cover processes that support judgements of value
and worth of programmes (Guba & Lincoln, 1981).

Reviews of evaluation methodologies have stressed the importance of determining the

purpose that the process will serve. Numerous distinctions have been made, but the
following set (from Oliver, 1997) has been adopted as a useful summary within the
learning technology community

 Formative evaluation

 Summative evaluation

 Illuminative evaluation

 Integrative evaluation

 Evaluation for Quality Assurance

These five purposes will be adopted to structure the following discussion. In addition,
special attention has been given in the literature to the evaluation of costs and to
comparative evaluations. As they represent special cases, these two categories will also
be considered.

3
Summary

In this section, the scope of this review has been established. The key characterstics of
online learning and teaching and reasons for evaluation have been identified. In the next
section, the evaluation of online learning and teaching will be considered, starting with
general issues before moving on to consider each of the specific reasons for evaluating a
programme in turn

Evaluating online learning and teaching

General issues for evaluation

In the previous section, the characteristics of online learning and teaching were
identified. Of all of these, the most immediate impact on evaluation arises as a
consequence of the physical distance involved. Even a cursory glance at lists of methods
for gathering data (eg. Oliver & Conole, 1998) will reveal that most involve contact
with the students Unless considerable effort and expense can be made to arrange
meetings, methods such as focus groups, interviews and observation are rendered
impractical

Many suggestions have been made that re-create these methods by proxy Cousin &
Deepwell (1998), for example, have discussed the feasibility of virtual focus groups.
They demonstrated that these can be an effective substitute for a real meeting and offer
all the benefits often advocated for computer-mediated communication such as allowing
space for reflection when responding. However, they also noted several limitations,
such as participants' reluctance to contribute messages on sensitive topics and the need
for a skilled facilitator of online discussions (see, eg, Salmon, 2000).

An alternative approach is to focus the evaluation on the types of data that online
systems are good at gathering Phelps & Reynolds (1998), for example, combined web-
based questionnaires (allowing immediate responses without the subsequent need for
lengthy data entry) with system usage data such as the time and frequency of page
access. This was achieved by using Javascript to create a tracking log via a CGI script
4
on the host server each time a page was requested. These method provided very rich
data on patterns of usage and on users' motivation and satisfaction. However, once
again, the methods had their limitations Usage logs are difficult to interpret, since they
cannot reveal why a leamer accessed a particular page or what they did with it once they
had gained access. Whilst this type of data is valuable and easy to collect, it remains
important to triangulate it with other sources as part of the interpretative process

Additionally, in this case, the rate of return for the online questionnaire was low. This
leaves the evaluation open to the criticism that the opinions recorded will be from a self-
selecting sample, and thus unrepresentative of the wider group of users. In particular, it
seems probable that less confident users of technology are those least likely to respond.
Other evaluations of online learning (eg. Taylor et al., 2000) have complemented this
method using a paper-based survey distributed to non-respondents of the online
questionnaire, leading to a much better overall response rate

Another common approach to data gathering in an online environment involves the

creation of a "feedback" discussion area Again, this offers the opportunity for
continuous feedback from participants and also provides a full transcript of responses in
an electronic format, ready for analysis. However, as Taylor et al. (2000) note, this can
open the floodgates to an unstructured wash of criticism, much of which may come
from a small but vocal minority. These views, which may be unrepresentative, can
cause considerable problems if used during formative assessment if designers feel that
they need to take all criticisms into account. Since it is impossible to please all of the
learners all of the time, the value of feedback forums may be greatest when an evaluator
is able to act as an intermediary between the data and the course team.

Many of these issues can be summarised by noting that evaluation in this context raises
two general problems for evaluators. Firstly, with many of these methods, the process
can no longer be controlled Opportunities for contributing data can be provided, but
what the student does with this opportunity is up to them. Taking a more extreme
position, it is also impossible to tell who is actually contributing the data. Secondly,
methods for interpreting these types of data are still being developed Whether the data
5
be from system logs or bulletin boards, lessons are still being learnt about the most
useful and appropriate ways of drawing conclusions

Finally, it is worth identifying methods of evaluation that are not affected by the move
to distance education Essentially, these will be either those designed to operate at a
distance, such as postal or telephone surveys, or those that do not require empirical data.
Surveys are clearly subject to the same issues as other distance methods, as discussed
above. Methods such as checklists and conceptual maps, which fall into the latter
category, are also subject to criticism. Whilst these are relatively easy to implement,
significant questions have been raised about their value (eg. Tergan 1998), not least
because of their highly subjective nature

Formative evaluation

Having highlighted some pragmatic problems for the evaluation of online learning, it is
worth considering the purposes of evaluation. In the previous section, five purposes
were identified, the first of these is formative evaluation. This refers to evaluations that
are intended to provide information that allows revisions and improvements to be made.
Its primary audience usually consists of the project or course team

Several features characterise formative evaluations. Firstly, they are usually camed out
by a member of the project or course team; in this regard, they are internal evaluations.
In order to be useful, they must provide timely information in a format that is readily
accessible to the course team. In this respect, utility is a higher priority than validity
(Patton, 1997).

Within small, self-contained teams, immediately accessible evaluations (where the

results need little or no subsequent analysis) such as focus groups are often useful. As
noted above, however, such techniques are often inapplicable in a distance context. In
addition, any source of data that relies on an input from the students will introduce
delays into the evaluation when nothing can be done but to wait for responses. The
implication of this is that scheduling becomes extremely important. Ironically, although

6
formative evaluations may be what is referred to as "quick and dirty", they are at their
most useful when carefully prepared for. The economy of effort must come in the
collection and analysis of data, rather than in the planning of the study.

A good illustration of the issues involved in formative evaluation is provided by the

EuroMET project (Phelps & Reynolds, 1999). This involved the development and
delivery of web-based courses in Meteorology by a consortium of 22 partners. Given
the complex structure of the project, it was important to ensure that appropriate
information was gathered and communicated in a timely manner. The evaluation that
was carried out included two strands. The first was a survey of users' views on ease of
use, pedagogy (including scientific integrity) and value as a replacement for traditional
teaching methods. The second involved usability trials, carried out with a sample of five
users under controlled conditions. This approach allowed the project team to identify
elements of the course that worked and those that needed revision The strong use of
visual material was welcomed by users, for example, whilst inconsistencies in the
material (such as variations in style and symbol use from section to section, reflecting
the different contributing authors) were identified as an area for attention. Both strands
of evaluation contributed to the re-design of the system's navigation. The evaluation
showed that users found that some icons were too similar to each other, that users had
no sense of where they were in the material, and that students wanted clear learning
objectives and end-of-unit summaries to be added.

The timing of this evaluation allowed these points to be fed back to the project team and
suitable revisions incorporated. The structure had added value in that one strand helped
to validate the revisions that were proposed

The formative evaluation has been extremely useful in producing modules

which are suitable for their target audience, are easy to use, and are
instructive. The fact that the evaluation was embedded into the development
work meant that it was relatively easy for the developers to modify the modules
according to the recommendations of the evaluators and, in turn, for these
modifications to be tested during the next evaluation phase. In particular the
7
 usability study showed that the modifications made after the first evaluation
phase were effective.

(Phelps & Reynolds, 1999, p. 192.)

Summative evaluation

In contrast to formative evaluation, summative evaluation is often an external process

concerned with judgement rather than improvement. It often involves assessment of a
project against its aims or in the case of online education, of a course in terms of
learning outcomes. It is often asserted that such evaluation ought to be carried out by an
evaluator outside of the project team in order to assure objectivity (e.g. Bradbeer 1999).

However, recent critiques of evaluation have made the point that evaluation is
inherently political (Patton, 1997), objectivity is, in many ways, a myth. Many of the
proposed advantages of scientific methods, designed for use in controlled conditions,
such as transferability and replicability, simply do not apply in the ill-defined, authentic
world of education practice

Such critiques have led to a division between experimental designs for summative
evaluation and those that are primarily exploratory (Oliver & Conole, 1998). This
section will focus on experimental approaches; the exploratory approaches will be
explored further in the following two sections.

Experimental approaches face several challenges. One of the most significant is that it is
effectively impossible to prevent 'contamination, where some factor external to the
experiment influences outcomes. An obvious example of this would be if an online
course broke down, and students passed the final exam because they had all formed
self-help groups and taught themselves from textbooks instead. Since it is impractical to
control all the extraneous factors in any educational setting, particularly when it
involves learning at a distance, it becomes extremely difficult to attribute causality to
the teaching intervention

8
More subtle problems arise in the context of comparative evaluations. These are often
popular with managers or funders, since it is assumed that the comparison will
demonstrate whether the innovation adds any value to the leaming experience. Here,
however, cross-condition contamination is even harder to prevent Even if online courses
are password-protected, it is quite possible for students to share IDs, download materials
or even just share notes.

Other problems also anise for comparative experiments. Experiments are predicated on
the ability to control the context in which they take place; this is necessary in order to
isolate the variables to be studied. In an educational setting, it is often impossible to do
this on pragmatic and ethical grounds. If different teachers are involved, another
important factor is introduced. The same is true if the materials change, the students are
different, the subjects covered vary or the way in which they are taught alters. From an
ethical point of view, it is difficult to justify allocating extra resources or opportunities
to only a sub-set of a student body, particularly when the course carries credit towards
an award.

A final criticism is aimed at comparisons of traditional and computer-based courses

This is particularly relevant for courses that are subjected to a comparison of learning
outcomes "before and after" adaptation to an online format. The argument is that,
because the methods used differ so radically, these experiences are so different that they
cannot be compared in any meaningful way. The analogy used is that it is like
comparing apples and oranges

It has been argued that such comparisons can be drawn, but that this must be done with
care (Oliver & Conole, 1998a) It is an easy matter to gather data on student preferences,
for example, or to compare performance on an end of year exam. What must be asked,
however, is what such a comparison means. If the change from traditional to distance
learning (for example) really does represent a completely new educational experience, it
is inappropriate for the assessment used to remain the same - a point often neglected

9
when designing online courses. This raises serious questions about the validity of
assessment methods which is beyond the scope of this paper. However, if the
assessment does remain the same, then a comparison of performance - irrespective of
the measure's validity can clearly be made. If what is being evaluated is simply student
performance against some assessment yardstick, then it is appropriate to compare their
net experience in reaching this. In such cases, comparison of courses becomes a sensible
option

Given the problems noted above about contamination, control and transfer, what the
experimental approach may permit is a firm conclusion about one particular comparison
(albeit with the proviso that the measurement's reliability should be considered
critically). Claims about transferability, however, are more difficult to justify. The
implications of this are that experimental evaluations (and comparative evaluations) are
possible. However, they must be designed with care, reported in a way that
acknowledges the limitations of the method in an educational context, and interpreted
with the same criteria as any qualitative case study.

Hiltz et al. (2000) provide a good example of an evaluation of online course that adopts
a critical approach to experimental methods. The evaluation concentrates on the Virtual
Classroom system, and involved three separate studies. These considered hypotheses
such as, "[Online communication and leaming] can improve quality of learning as
measured by grades or similar assessments of quality of student mastery of course
material". Importantly, the proviso made above about the validity of assessment as a
measure of learning is explicitly acknowledged here

Careful attention is paid to general experimental evaluation issues, and explicitly

discusses the limitations of the experimental method outside of a laboratory setting
Moreover, the limitations of studies are also made clear. For example, considering the
first study of Asynchronous Learning Networks (ALNs) described in isolation, Hiltz et
al. acknowledge that, "The longitudinal field study does not allow us to conclude
whether better educational outcomes in ALN-supported courses are the result of
collaborative learning techniques, ALN use, or both" This problem was tackled by
10
triangulating the three studies. This allows Hiltz et al. to conclude that, "though any one
measure or method might be legitimately questioned in terms of its validity, reliability,
or generalizability, the weight of several different kinds of studies over a period of five
years, is convincing."

Illuminative evaluation

The problems of employing experimental methods in educational settings are not new.
In 1987, Parlett & Hamilton proposed an alternative model based on a 'social
anthropological' approach to evaluation. Rather than attempting to quantify impact,
these studies seek to discover the factors that are important to the participants. This is
achieved through phases of observation, inquiry and explanation, with analytical
methods adapted pragmatically and triangulation used to improve the reliability of
findings.

In contrast to experimental evaluations, which seek to control the factors that might
influence learning and teaching, illuminative evaluation seeks to describe and interpret
them. The educational context becomes the focus of the study, rather than the measure
of learning that is used for assessment purposes

This freedom clearly avoids the problems that faced experimental approaches. However,
the very responsive flexibility that allows illuminative evaluation to achieve this
prevents its conclusions from being objective or transferable. Conclusions are
interpretations constructed by the evaluators. Confidence in them can be increased if
methods are triangulated, but they remain interpretations of specific events. Although
this may be perfectly adequate for summative evaluations of single programmes, it will
pose problems if the lessons learnt from this are to be applied elsewhere. Experimental
studies make the claim that their results are generalisable; however, as noted above, this
claim is problematic in an educational setting

11
In summary, illuminative evaluation accepts the criticisms levelled at experimental
studies and, rather than trying to overcome them, works within the constraints that they
represent. No attempt is made to generalise, for example. ***Work on this!!!

Wegerif's study of the development of communities in online discussion provides an

example of the illuminative approach to evaluation. The study involved participant
observation, in-depth interview together with surveys and a transcript of discussion,
then analysis. This allowed a deeper understanding of the process through which
students succeeded or failed in joining an online community, and the implications of
this on their achievements. The conclusions that were drawn were specific to this
situation, but recommendations were put forward as a starting point for discussion -
including comparison with other studies of situations like this.

"As well as its more specific conclusions and recommendations, this study has
illustrated a method for researching the social dimension of ALNS and put
forward the beginnings of a conceptual framework, including the concept of
the difficult threshold between insider and outsider status, which may prove of
general value in understanding the impact of the social dimension on learning
on ALNs and how this impact can be taken into account in course design."

Wegerif, 1998

Integrative evaluation

Experimental and illuminative approaches to evaluation can be seen as two extremes,

each of which has limitations. Several evaluators have attempted to create compromises
that incorporate elements of both approaches. Integrative evaluation is one such
approach

The term integrative evaluation is used in several contexts, but in the field of learning
technology research is usually associated with the approach devised by the Teaching
with Independent Learning Technologies (TILT) project (Draper et al., 1994). This
combined the structured approach of experimental evaluations with the values and
12
flexibility of illuminative studies. In addition to the study itself, phases of work took
place that addressed the context of the course, addressing issues such as policy,
resources and the tacit teaching objectives of the staff involved. Integrative studies
incorporate multiple methods, including within-group experimental studies of
performance, surveys, interviews and confidence logs.

Inherent in the approach, however, is the assumption that the evaluation's findings will
be situationally-specific. The term "integrative" reveals the central motivation for the
project team, which was to improve the way that computer-based resources (including
online materials) were incorporated into the course. As with illuminative approaches, no
attempt is made to generalise the findings.

Draper & Brown (1998) used the integrative approach in their study of remote
collaborative tutorial teaching. This involved around 20 different studies, each of which
adopted a similar approach, and which were then summarised and synthesised in order
to make claims about the project as a whole. This process allowed the evaluators to
argue that the collaborative tutorial teaching process were at least as effective as
traditional methods, were received with mixed levels of enthusiasm, and were primarily
of benefit in enriching the curriculum and in staff development. There was no need to
generalise the conclusions beyond this point, since the approach was unique to this
project; however, the synthesis of so many individual integrative studies did provide an
adequate basis for summative judgement and for recommendations for others attempting
to adopt a similar approach.

Evaluation for Quality Assurance

13
REFERENCES

Draper, S. & Brown, M. (1998) Evaluating remote collaborative tutorial teaching in

MANTCHI. In Oliver, M. (Ed) Innovation in the Evaluation of Learning Technology,
65-86. London: University of North London Press.

Draper, S., Brown, M., Edgerton, E., Henderson, F., McAteer, E, Smith, E. & Watt, H.
(1994) Observing and measuring the performance of educational technology. TILT
project report, University of Glasgow.

Wegerif, R. (1998) The Social Dimension of Asynchronous Learning Networks. Journal

of Asynchronous Learning Networks,2 (1).
http://www.aln.org/alnweb/journal/vol2_issuel/wegerif.htm

Parlett, M. & Hamilton, D. (1972) Evaluation as Illumination: a new approach to the

study of innovatory programmes. In Murphy, R. & Torrance, H. (Eds) Evaluating
Education: Issues and Methods. London: Harper & Row.

Hiltz, S., Coppola, N., Rotter, N., Turoff, M. & Benbunan-Fich, R. (2000) Measuring
the Importance of Collaborative Learning for the Effectiveness of ALN: A Multi
Measure, Multi-Method Approach. The Journal of Asynchronous Learning Networks, 4
(2). http://www.aln.org/alnweb/journal/Vol4_issue2/le hiltz/le-hiltz htm

Patton, M. (1997) Utilization-focused evaluation. London: Sage.

14
Taylor, J., Woodman, M., Sumner, T. & Blake, C. (2000) Peering Through a Glass
Darkly: Integrative evaluation of an on-line course. Educational Technology & Society,
3 (4). http://ifets ieee.org/periodical/vol 4_2000/v_4_2000.html

Phelps, J. & Reynolds, R. (1999) Formative Evaluation of a Web-based Course in

Meteorology. Computers & Education, 32, 181-193.

15