BritishJournalof Educational Technology Vol 39 No 6 2008 1037-1056

doi:1O.1111/j.1467-8535.2007.00777.x

Measuring the quality of e-learning

David B. Hay, Caroline Kehoe, Marc E. Miquel,

Stylianos Hatzipanagos, Ian M. Kinchin, Steve F Keevil
and Simon Lygo-Baker

David B. Hay, Stylianos Hatzipanagos,Ian Kinchin, and Simon Lygo-Baker arepart of King's Institute
of Learning and Teaching at King's College London. Caroline Kehoe and Steve Keevil belong to the
Division of Imaging Sciences at King's College London. Marc Miquel is part of the Department of
MedicalPhysics at Guys and St Thomas' NHS FoundationTrust, St Thomas' Hospital London. Address
for correspondence: David B. Hay, King's Institute of Learning and Teaching (KILT), King's College
London, James Clerk Maxwell Building, 57 Waterloo Road, London, S11 8WA, UK. Tel: 020-7848-
3265;fax: 020-7847-3253,email: david.2.hay@kcl.ac.uk

Abstract
This paper shows how concept mapping can be used to measure the quality of
e-learning. Six volunteers (all of them 3rd-year medical students) took part in
a programme of e-learning designed to teach the principles of magnetic reso-
nance imaging (MRI). Their understanding of MRI was measured before and
after the course by the use of concept mapping. The quality of change in
individuals' maps was assessed using criteria developed to distinguish between
meaningful and rote-learning outcomes. Student maps were also scored for
evidence of conceptual richness and understanding. Finally, each map was
compared directly with the content of the electronic teaching material.
The results show that many of the student misconceptions were put right in
the course of their learning but that many of the key concepts introduced in
the teaching were ignored (or sometimes learnt by rote) by the students. This
was because the teaching material locked these new ideas in structures and
terminology that precluded meaning-making among non-experts. Our data
suggest that students' prior knowledge is a key determinant of meaningful
learning. We suggest that this must be acknowledged if the design and use of
electronic teaching material is also to be meaningful. Ultimately, measures of
student learning are the only authentic indicators of the quality of teaching
through technology.

Introduction
The literature on teaching with technology comprises many and varied frameworks
for evaluation (eg, Britain & Liber, 2000; Jones et al 1999; Laurillard, 2002). None of
02008The Authors. Journal compilation Ollecta 2008. Published by Blackwell Publishing. 9600 Garsington Road. Oxford OX4 2DQ.
UK and 350 Main Street. Malden, MA 02148. USA.
1038 British Journalof Educational Technology Vol139 No 6 2008

these, however, set out to infer teaching quality from the quality of student learning
that they engender. This is an important oversight. Essentially, all evaluation depends
on measures of 'fitness for purpose', and because teaching has purpose only where it
supports learning, learning is the only authentic measure of teaching. That this has not
been substantively pursued in teaching evaluation is attributable to two important
issues. The first is that the learning process is commonly deemed too complex for
empirical measurement (see for example, Bowden & Marton, 1998; Knight, 2002).
There is, however, a growing literature on student learning quality (Daley, 2002; Hay,
2007; luli &Helld6n, 2004; Novak, 1998; Novak &Musonda, 1991). The second issue
depends on the obvious truism that while teaching can lead to learning, learning is not
a necessary consequence of teaching. Indeed there is a persistent strand of the litera-
ture that suggests that learning is ultimately a consequence of student behaviour
(rather than any direct consequence of the teaching they experience) (eg, Biggs, 1996,
2003; Shuell, 1986; Tyler, 1949). This argument is more intractable, but we do not
believe that it should prohibit attempts to assess teaching through measures of learn-
ing. Suffice that while student learning can never be more than a proxy for teaching
quality, it is still the only authentic measure of the effectiveness of teaching.

Learning quality
There is now a small but significant literature on the quality of student learning in
higher education (see Hay, Kinchin &Lygo-Baker, 2008 for a review). That learning can
be reduced to measurable criteria (as these authors attest) is attributable to Novak's
work towards a common definition of learning (see Novak, 1998 for example). Briefly,
Novak (1998) states that the quality of change can be located within a continuum
from rote to meaningful learning and that meaningful learning can be defined in the
following ways:
1. Relevant prior knowledge: that is, the learner must know some information that
relates to the new information to be learned in some non-trivial way.
2. Meaningful material: that is, the knowledge to be learned must be relevant to other
knowledge and must contain significant concepts and propositions.
3. The learner must choose to learn meaningfully: that is, the learner must consciously
and deliberately choose to relate new knowledge to knowledge the learner already
knows in some non-trivial way (p. 19).
This definition is important because it provides a common framework against which
learning quality can be measured. But the contribution of Novak goes beyond this
episteinologicalstance alone; it also provides a method (of concept mapping) in order
that measurement of learning quality can be assessed (Novak, 1998, 2005; Novak &
Mudona, 1991). The approach is explained below.

The concept mapping method

Concept mapping is one of a broad family of graphic organising tools that includes
mind mapping (Buzan & Buzan, 2000) and spider diagramming. Yet Novak's method
(Novak, 1998) has some very specific rules that set it apart from other techniques and
facilitates the measurement of learning quality. First, a concept map is a hierarchical

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1039

map of concept labels. Big and inclusive ideas are placed at the top, and exemplary or
subordinate ones are placed below. Second, concepts are linked with arrows and the
arrows are labelled to explain the nature of the association. Thus, concept maps com-
prise any number of propositional statements, each of them made up of paired and
linked concepts. Each proposition is a statement of understanding and the validity of
each assertion is laid bare. This is a powerful teaching tool because it facilitates the
declaration of understanding among teachers and st6dents. Where the same person
maps the same topic repeatedly, then a comparison of two or more such 'snap-shots'
facilitates measurement of cognitive change and the quality of change can be assessed
against Novak's original definition of meaningful learning (ibid.). Those parts of the
knowledge structure that are new can be readily differentiated from those that are old,
and the degree of integration of the new ideas with the extant parts of the prior
knowledge structure can be measured. This is shown in Figure 1.

This study
The data we report here pertain to the evaluation of a single e-learning programme in
medical imaging. The development of this programme as a whole was begun in 2005,
but here we describe evaluation of only one part of the teaching 'product': specifically
a series of case studies in medical diagnosis using magnetic resonance imaging (MRI)
and a tutorial on the theory of MRI technology. Testing is often part of the e-learning
materials development cycle (eg, Valkanos et al, 2005; Oliver &Herrington, 1995), but
our approach was designed to measure the quality of knowledge change that was a
consequence of the student's e-learning experience. Because this is a new approach for
e-learning design, we also developed and used three different ways of assessing student
learning quality. All of these depended on the collection of data through concept
mapping but used different means of attributing quality scores. The data collection
methods and the three different approaches to analysis are described below.

Methods
Six 3rd-year medical students of King's College London volunteered to take part in the
evaluation. Their identities have been disguised by the use of pseudonyms.

Data collection
The first author of this paper taught the student volunteers how to use the concept
mapping method (as described by Novak, 1998). The lesson was given to the group as
a whole, but afterwards, each student was asked to make his or her own map to describe
their prior knowledge of MRI. These maps were made without reference to notes or
instructional materials, and the students had no knowledge of the focus of the learning
task that was to follow. The concept mapping lesson and the map-making activity took
about an hour, and the students' maps were collected for analysis at the end of
the session. Each student then received a CD-ROM of the teaching materials. This
comprised instruction of two types. First it provided a series of interactive diagnostic
simulations. Here the students were presented with case studies that required them
to ask questions about patient histories and to order diagnostic tests. The programme
rewarded their enquiries with medical data and information that led progressively

0 2008 The Authors. Journal compilation O Becta 2008.

1040 BritishJournalof Educational Technology Vol139 No 6 2008

BEFORE AFTER

NON- knowledge structure

LEARNING remains unchanged

some prior-concepts
are rejected and new
SURFACE ( ones are added, but no
new links are made between
LEARNING retained concepts
and the newly added concepts
are not linked to the
the prior knowledge structure

new concepts
are linked to the retained
DEEP knowledge structure and new
LEARNING p links are made between those
parts of the prior knowledge
structure that are retained

top (organising) concepts a rejected concepts .. ,retained concepts 0 added concepts 0

Figure 1: Empiricalmeasures of learningquality (after Hay, 2007)

Conceptual structuresare shown before and after learning.Concepts are shown as circlesand the
explanatory links between them are drawn as lines. Non-learning (top) comprises the simple repetition
of priorknowledge and constitutes reinforcement without change. Rote learning (middle) is the
addition (or deletion) of concepts in the absence of integration with extant parts of the prior
knowledge structure. Meaningful learning (bottom) comprises the integrationof prior knowledge and
newly acquired information in ways that demonstrably increase understandingof the topic

towards a diagnosis. All of these case-study simulations were grouped around the use of
MRI in cardiology.

The second part of the package was a tutorial on MRI. Figure 2 is a summary of its
structure and content. The tutorial was text-based and annotated with graphics,
images and diagrams. It was written to teach the basic principles of MRI and was the
'core content' of this part of the e-learning package as a whole.

In all, the simulations and the tutorial were expected to take 6-8 hours of study-time,
and the students were asked to complete the package in the following week; the volun-
teer group then reconvened and the students were reminded briefly of the concept

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1041

Figure2: A summary of content in the e-learning material

The e-tutorialon MRI comprised six sections and was designed to introducestudents to the topic

mapping method. Then they were asked to draw their maps of the MRI topic again. This
was done without recourse to their first maps or to any other prompts.

Data analysis
Three different approaches were used to score the quality of change in the students'
maps.

Method one: analysis of structural change and learning quality

This approach was based on methods previously described by Kinchin, Hay and Adams
(2000) and Hay (2007). First, the basic typologies of students' maps were classified as
spokes, chains and networks (see Figure 3) and compared before and after learning to

0 2008 The Authors. Journal compilation 0 Becta 2008.

1042 British Journalof EducationalTechnology Vol 39 No 6 2008

SPOKE CHAIN NET

Structure
0-R-0 0_1hJý

Many levels, Several

lierarchy Single level but often justifiable
inappropriate levels

Additions to central Cannot cope with Additions I

the deletions may
kiditions interfere do
concept not
with beginning near
additions of the have varying
associated concepts sequence influence as 'othe
routes' are often
Have no available through
effect on Disrupt the the map
Deletions overall sequence below
structure the deletion

Often
Icompound', only Often employ
Links Often simple making sense technical
when viewed in terminology to
the context of the enhance meaning
previous link

Figure 3: Concept map typologies

Kinchin, Hay and Adams (2000) have shown that there are three basic concept map structures
(spokes, chains and netivorks). Structural change (from one basic structuraltype to another) is an
indicatorof learningquality (good or bad) (Hay & Kinchin, 2006)

detect gross structural change (see Hay & Kinchin, 2006 for a detailed description of
the classification system and discussion of the implication of knowledge structure for
student learning). Second, the prior-knowledge maps and the maps made after learning
were compared for qualitative measures of change (Hay, 2007). The criteria used for
learning quality measures were as follows:
(1) Non-learning: defined by an absence of cognitive change. Non-learning was there-
fore measured by the lack of new concepts in the second map and by an absence of
new links in the extant prior-knowledge structure.
(2) Rote learning: defined in two ways: first, by the addition of new knowledge, and
second, by the absence of links between the newly acquired concepts and those
parts of the prior knowledge repeated in the second map.

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1043

(3) Meaningful learning: defined by a substantial change in the knowledge structure.

Thus, evidence of deep learning comprised the emergence of new links in parts of
the prior-knowledge structure developed in the course of learning or the meaning-
ful linkage of new concepts to parts of the pre-existing understanding.

Method two: use of expert terms

This approach comprised a simple analysis of the frequency of concept labels used in
the teaching material and in the students' concept maps after learning. We used the
concept mapping method to reduce the text of the e-learning tutorial to a concept map,
and the concept labels were compared with those used by the students. Figure 2 is a
summary of the tutorial (and Appendix 1 shows the concept map of all of the taught
content).

Method three: expert scores of student maps

All of the students' maps were blind-marked by a consultant clinical scientist special-
ising in MRI. This was done to measure both the degree of 'fit' between student and
teacher views, and to give credence to the richness and individuality of student under-
standings (even where these might not correspond exactly to the content of the teach-
ing). The scoring framework is shown in Appendix 2. It used 5-point Likert scales (from
strongly agree to strongly disagree) and measured the following constructs:
"*conceptual richness (the richness and relevance of ideas used to describe the topic);
"*linkage and linkage quality (the richness and appropriateness of concept linkage);
"*evidence of understanding (the degree to which the propositions describe
understanding);
"*hierarchy and structure (the complexity and validity of the structural representation
of knowledge).
There were two statements designed to measure each construct, and each of these were
asked twice: once in the positive and once in the negative form. Negative scores were
inverted (so that each construct could be given a score from 4 to 20) and a pair-wise test
of reliability was done to measure the instrument validity (below). All 12 of the stu-
dents' maps (six prior-knowledge maps and six maps after learning) were given to the
assessor in random order, and all of the statements were ordered at random in each
individual score sheet. The assessor was also asked to identify any significant miscon-
ceptions in the maps and to list any concepts or propositions that were conspicuous by
their absence.

The face validity of the instrument was tested by six researchers. All of them agreed
that the constructs were appropriately defined and were parsimoniousdefinitions of an
individual's understanding, irrespective of the topic or discipline of study. The method
chosen to track change judged by a constant (the expert) was a simple comparative
analysis against time. The responses were grouped under the four constructs and no
outliers were found in the resulting analysis. Each construct saw a simple positive
change over time, as did each student (ie, all of the students' second maps were evalu-

a 2008 The Authors. Journal compilation 0 Becta 2008.

1044 British Journalof Educational Technology Vol139 No 6 2008

ated as better than their first, and overall, the improvements were distributed evenly
among the four constructs: concept richness, linkage, propositions and hierarchy).

Results and discussion

Table I shows a summary of the students' learning measures obtained using each of
the methods. All of them showed that the students' knowledge structures were richer
and more explanatory after e-learning. Nevertheless, some students had clearly learnt
more than others. Three satisfied the criteria for meaningful learning and three did not
(method one). Among those who did not, two showed rote learning and one showed
rote-learning outcomes mixed with non-learning. These differences in outcome are
illustrated by the case studies described later in this paper.

The results obtained from method two offered less substantial evidence for the quality of
the students' e-learning. In all, the teaching material comprised 78 different concepts
(Appendix 1). In aggregate, however, only 28 of these were represented in the student's
post-learning maps. Furthermore, all of these were derived from just three of the six
parts of the tutorial (Figure 2; part 1: the introduction, part 4: gradient echo and part
6: interactive real-time imaging). Not one term used in parts 2, 3 or 5 of the tutorial
found a way into any of the students' maps. Moreover, learning directly attributable to
parts 4 and 6 of the tutorial was confined to only three students (Mark, Simon and
Amy). The other three (Tom, Sally and Peter) used 'taught' concepts from part I alone.
Even Simon and Amy (who used more newly taught ideas than anyone else) did so
through surface-level addition. In their second maps, for example, both of them used
the terms coronal, sagittal and axial (introduced by part 6 of the e-learning material),
but neither Simon nor Amy offered any explanation of these terms, nor did they explain
why these different image planes might be important in MRI.

Among all the student maps after learning, Mark's was deemed the best by method
three, but Mark's prior-knowledge structure was also notably superior to the others.
Previously, Hay et al (2008) have suggested that prior-knowledge structure is the
best predictor of the meaning-making that can occur in the course of learning. This
view is corroborated here and is illustrated by Figure 4, which shows the increase of
individuals' knowledge and understanding as a function of the quality of their prior-
knowledge structure. Nevertheless, two students (Sally and Amy) did notably better
than the average correlation between achievement and prior knowledge. Both Sally and
Amy revised the structure of their knowledge in the course of their learning (from a
spoke to a network in both cases). All the other students changed merely by the addition
of concepts or links without reorganisation. Kinchin et al (2000) suggest that cognitive
restructure is a key component of meaningful learning, and Hay and Kinchin (2006)
have shown how spoke structures in particular are amenable to change. While we
suggest here that the quality of students' prior knowledge is likely to determine the
meaning-making that they can make from the teaching they experience, it is also likely
that that the different ways that students go about organising their knowledge can be
more or less amenable to learning. Sally's concept maps before and after e-learning are
presented among the case studies below to illustrate this. Finally, while Sally and Amy

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1045

0--

tCn

0 si

to to

.- Si E Z-

ZU01-~~~~t
C)0 co ý Ow

cz

10 42 0

u - - cz

too
to Co
E o

0
EZ~t D U o

CD c
ZZZ o:. 0-
5.4:
cnM /IzzE-I4

0 2008 The Authors. Journal compilation 0 Becta 2008.

 1046 British Journalof EducationalTechnology Vol 39 No6 2008

0
tn CN

rZI

t-4 CU
'ZI
r-4
-4 0 <)
"04 r-.i
to0
r-
-O + * D r.0
,.O
in i'n

-4 Ln

-4
+
00 00 -Me
r-4 C. +
Z e 0Cz

-0
+€ +
.CjC

0- --4 -4
00
en oN
00
C)Ot
00q
-4% C)-"
U +'€ ±t
N€
"-4
C)

C)
-4
0 '-

r--t
+
tnM
emr

to-Z
02 'J-
0 0

0 c0

U
0 0

to0CU0L O-CU:

0)-

0U r'L4C

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1047

35
30
25 -Sally
lAmy Mark
o 20
in Tom
S15
S10 Peter

5 o * Simon

0
0 20 40 60

Prior-knowledge score

Figure 4: Change in student learning scores as a function of the quality of priorknowledge

Students' pre- and post-test scores were correlatedwith the measures of quality of their
prior knowledge

did better than might have been expected from their prior-knowledge structures, Simon
did worse. This, too, is probably best explained by analysis of his prior-knowledge struc-
ture. Simon started with a chain, and as Hay and Kinchin (2006) suggest, chains are
resistant to change. Simon is presented first among the case studies below.

Case-study analysis
In order to explore the learning of our volunteers in more detail, we show the concept
maps made before and after learning for three of them. These case studies are indicative
of all of the types of change that we observed among the group as a whole.

Simon: rote-learning outcomes

Simon's maps of MRI before and after the e-learning are shown in Figure 5.

His first map was a chain-like structure with only one significant branch. The map
focused on issues of image storage, and although it did not comprise any significant
misconceptions, the specific focus of the map was deemed inappropriate to the broader
context of an understanding of MRI (expert analysis: method three).

At first glance, Simon's second map was notably different from his first. Closer scrutiny
suggests, however, that the apparent changes were superficial and indicative of rote
learning. The second map described MRI in ways broadly synonymous with the prior-
knowledge structure, and most of new knowledge was clearly derived verbatim from the
tutorial (by rote learning rather than meaning-making). Simon's second map was
improved by an understanding of magnetism and radio waves, but in developing this
understanding he also acquired a new misconception (that whole molecules of water
become polarised in MRI). After learning, Simon described ideas relating to the genera-
tion of three-dimensional images and the use of different image planes in MRI. But

0 2008 The Authors. Journal compilation 0 Becta 2008.

1048 BritishJournalof EducationalTechnology Vol 39 No 6 2008

AI uses

[big magnet
to
Spolarlse
water
causing a

whposition of

which Is detected by

Figure 5: Simon's concept maps before and after e-learning

Simon's prior-knowledge map (A)focused on issues of image storage and display (1) and did not offer
a good explanation of MRIfunctionality. This was improved in the second map although the second
map also showed a significant misconception of the topic (2). Issues of storagewere persistent (3)
and two sets of newly acquired knowledge were included in the second map (4a and 4b). These were
added superficially and without linkage to the prior-knowledge structure. Overall, Simon's pattern of
knowledge change was indicative of rote learning

0 2008 The Authors. journal compilation 0 Becta 2008.

 e-Learning quality 1049

neither of these topics was described to show evidence of meaning, nor even that the
newly acquired terms were actually understood.

Peter: non-learning and rote-learning outcomes

Figure 6 shows Peter's knowledge structures (before and after learning).

The core structure of both maps was homologous. Peter merely bolted two new sets
of concepts (one about medical testing and one about medical histories) to a prior-
knowledge structure that remained largely unchanged. Moreover, the newly intro-
duced concepts were described in ways that precluded evidence of understanding. None
of the new terms were linked in ways that suggested that they might have been under-
stood, nor were any of them linked to the prior-knowledge structure.

Sally: meaningful learning

Sally's prior knowledge was a simple spoke structure (Figure 7A). More than half of it
comprised propositions that 'experts' would deem wrong. Nevertheless, Sally's second
map (Figure 7B) combined new knowledge with parts of her prior-knowledge structure
in ways that indicate meaningful learning. All but one of her earlier misconceptions
were also changed, but another was also acquired.

Not least, Sally acquired new ways of describing the topic of MRI and these went
beyond the content of the teaching material. Sally's acquired interest in the work of
'academic laboratory studies', for example, was a particularly insightful commentary
on her learning, and evidence of thinking about the subject that was beyond the
content and instruction of the e-learning material. This theme was not part of the
e-learning materials (Figure 2 and Appendix 1) but was something that Sally had dis-
covered for herself.

Towards an authentic pedagogy of e-learning

Concept mapping has a unique role to play in the pedagogy of higher education (see
Hay et al, 2008 for a review). On the one hand, it provides an epistemological frame-
work for the practice of teaching and, on the other, it is grounded in a theory of
learning (eg, Ausubel, 1963, 1968; Ausubel, Novak & Hanessan, 1978). This paper,
however, is probably the first to show that the method can also be used to measure the
quality of e-learning. Our results lead us to suggest that measures of student prior
knowledge should be a key step in e-learning materials design. It is only out of prior
knowledge that students are able to make meaning of the new topics that they are
taught. If content is 'locked up' in the e-learning material in ways that are not acces-
sible to students, then non-learning or rote-learning outcomes must be expected. Our
data also suggest that where learning is meaningful, it is a consequence of individual
thinking, development and synthesis. It is not a clear and attributable outcome of
teaching. This means that good teaching will always be difficult to measure directly, but
it also has significant implications for teaching practice. If meaning-making is depen-
dent on the behaviours of learners, then teachers should do at least as much to encour-
age and support student-centred learning activities as they do to develop and deliver

© 2008 The Authors. )ournal compilation O Becta 2008.

1050 British Journalof Educational Technology Vol 39 No 6 2008

A mnedi!cal
prodiUce
produce po

emit emit

rmes harmful
radation

used for -_used for usedfr usedfor

e.g. e.g. e.g. e.g

- 5ou d E5 Eý

Figure 6: Peter'sconcept maps before and after e-learning

Peter's understandingof the topic changed little before (A) and after (B) learning.Hisfirst map was a
branching structurecomprising one significant misconception (1). This was addressed in the second
map but the basic structureand content of the second map was unchanged (2). 7Tvo new idea sets
were also added (3a and 3b), but this was done superficially. Overall, Peter'smaps showed evidence of
non-learningand rote-learningoutcomes

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1051

isi

high s Ah
resolution ha toconeped beor
agn"'in

denaty nicra tote

aae
ese a rdthepo d eto tnos
is ustt ckarnial
a f e ll
medical rr
imaging
because
ticire cludes kxwasde kAtnides krAkies O ft
iMY useof

~ ~ ~ ~ onIR 5
all~fhrmsocpin fu
eeadese.tion getdagei al' apatrlann
it tansU$aS 1,430

COam ,teteud
togivepiturs oaf

but C iito=im

Figure 7: Sally's concept maps before and after c-learning

Sally's map before learning (A) was a spoke structure and comprised many different misconceptions
(1-4). Her map after learning (B) was much improved (both in structuralrichness and in content,), and
all but one of her misconceptions (5) were addressed. The biggest change in Sally's map after learning
was a newly acquiredappreciationof the role of researchin the development of MRI technology. This
was not a specifiefocusfor the e-l earning that Sally had done but was of her own making

0 2008 Tite Authors. Journal compilation 0 Becta 2008.

1052 British Journalof EducationalTechnology V1ol 39 No 6 2008

content. Indeed, Novak (1998) states that teachers should endeavour to present new
learning materials in meaningful ways. Ultimately, it is the learner who must choose to
learn meaningfully, but teachers can help them to do this by ensuring that the new
material is introduced in ways that are appropriate to the pre-existing knowledge of
their students. Consequently, good teaching is likely to comprise measures of student
prior knowledge and repeated measurement of newly emerging understanding. If there
is a criticism of our own e-learning material, it is that it fails to correspond to the
prior-knowledge structures of the students for whom it was designed. That we can say
this, however, is a consequence of the measurement of student knowledge that we have
now done.

The duration of this study was short in which to expect meaningful learning to occur.
This is an important point and it serves to emphasise the need for longitudinal studies
of student e-learning. Nevertheless, teaching in higher education is commonly done
through short and intense coverage of specific topics before moving on to other themes.
Many courses are actually designed so that a different topic is taught each week.
Because of this, our study design is appropriate to the context in which students usually
learn at higher education level. University student learning remains a largely neglected
topic (Laurillard, 2002). Addressing this is a clear priority for research. In the context
of e-learning, in particular, this will require more than documenting issues of technol-
ogy use alone but must be underpinned by empirical studies measuring and defining
learning. We hope that the methods and approach described here will lead others to
continue to explore the quality of e-learning.

Conclusion
This paper provides empirical evidence for change that is indicative of different qualities
of learning. It suggests that student learning quality is a product of student activities
and behaviours (rather than any direct consequence of taught content). Nevertheless,
we also suggest that the ways in which teachers present new knowledge to students can
have an important bearing on the ways that their students learn. Our data suggest that
measurement of student prior knowledge is key to effective e-learning materials design.
It is only out of existing knowledge and understanding that students are able to make
sense of any new teaching they receive, and if they are to be able to learn meaningfully,
new material must be presented in meaningful ways. To be truly effective, e-learning
materials design should involve teachers and students in the research of learning.

Used in the ways described here, concept mapping is a relatively new approach to the
study of learning quality, and it is therefore important that future work explores some
of the limits and potential weaknesses of the method in this context. In particular,
concept mapping studies should be compared with the outcomes of other knowledge
representing tools and approaches (rich picture building, or interviewing for example).
Some methods may prove to be more or less appropriate for different people or topics.
But we also suggest that concept mapping is likely to have a very wide-ranging utility.
The proposition (the linking of two concepts to make meaning) is the basic unit of
concept mapping and propositions are widely acknowledged to be the most basic units

0•2008 The Authors. Journal compilation 0 Becta 2008.

 e-Learning quality 1053

of language. As Kinchin et al (2000) have shown, concept maps can be linear and
causal (chains) or associative (simple spokes). Sometimes they combine these different
ways of knowing in complex networks too (see Hay &Kinchin, 2006). Because maps
are made step-by-step, however (from basic propositions upwards), these structures
emerge in the course of concept mapping and need not constrain the author of a map
towards any particular type of knowledge structure.

Above all, however, concept mapping makes learning visible (Hay, 2007; Hay et al,
2008), and can be used to document the phenomena of learning directly (Hay, 2007).
Concept mapping and other approaches like it (rich picture building, for example) have
a lot to contribute to future e-learning design in the broader frames of situated learning
and constructivism, but longitudinal studies of student knowledge-change in the
course of university-level learning remain an important priority for research.

Aclnowledgements
The work reported here was funded by a grant for the research of pedagogy from the
Society for Educational Studies and by the Higher Education Funding Council for
England (HEFCE) monies disbursed by King's College Teaching Innovation Fund. Initial
development of the e-learning materials was funded by the KCL Development Trust.

The work reported here is a collaboration between the staff of King' Institute of Learn-
ing and Teaching (KILT) and the Division of Imaging Sciences at King's College London
School of Medicine. KILT provides teaching, support and research for the enhancement
of teaching and learning quality. The Division of Imaging Sciences is a centre of excel-
lence for research and teaching in medical imaging technologies, with a particular
interest in cardiovascular magnetic resonance imaging.

References
Ausubel, D. P. (1963). The psychology of meaningful verbal learning. New York, NY: Grune and
Stratton.
Ausubel, D.P. (1968). Educationalpsychology: a cognitive view. NewYork, NY: Holt, Rinehart and
Winston.
Ausubel, D.P., Novak, J. D.&Hanessan, H. (1978). Educationalpsychology: a cognitive view. New
York, NY: Holt, Rinehart and Winston.
Biggs, J. B. (1996). Enhancing teaching through constructive alignment. Higher Education, 32,
1-118.
Biggs, J. B. (2003). Teaching for quality learning at university (2nd ed.). Buckingham, UK: The
Society for Research into Higher Education and Open University Press.
Bowden, J. &Marton, R (1998). The university of learning.Abbingdon, Oxon: Routeledge Falmer.
Britain, S. &Liber, 0. (2000). Aframeworkfor the pedagogicalevaluation of virtual learningenviron-
ments. JTAP Report no. 041. Retrieved 17 April, 2007, from http://www.leeds.ac.uk/educol/
documents/00001237.htm.
Buzan, T. &Buzan, B. (2000). The mind map book. London: BBC Worldwide Ltd.
Daley, B. J. (2002). The scholarship of teaching and learning: facilitating adult learning. Journal
of Scholarship of Teaching and Learning, 3, 1, 14-23.
Hay, D. B. (2007). Using concept maps to measure deep, surface and non-learning outcomes.
Studies in HigherEducation, 32, 1, 39-5 7.

0 2008 The Authors. Journal compilation 0 Becta 2008.

1054 British Journalof EducationalTechnology Vol 39 No 6 2008

Hay, D. B. &Kinchin, I. M. (2006). Using concept maps to reveal conceptual typologies. Education
and Training,48, 79-83.
Hay, D. B., Kinchin, I. M. & Lygo-Baker, S. (2008). Making learning visible: the role of concept
mapping in higher education. Studies in Higher Education, 33, 3. in press.
luli, R. J. & Helld6n, G. (2004). Using concept maps as a research tool in science education
research. In A. J.Canas, J.D. Novak &E M. GonzAlez (Eds.), Proceedingsof the First International
Conference on Concept Mapping, Pamplona, Spain. Retrieved 29 August, 07 from http://
cmc.ihmc.us/papers/cmc2004-223.pdf.
Jones, A., Scanlon, E., Tosunoglu, C., Morris, E., Ross, S., Buthcher, E et al (1999). Contexts for
evaluating educational software. Interacting with Computers, 11, 499-516.
Kinchin, I. M., Hay, D. B. & Adams, A. (2000). How a qualitative approach to concept map
analysis can be used to aid learning by illustrating patterns of conceptual development.
EducationalResearch, 42, 1, 43-57.
Knight, P T. (2002). Being a teacher in highereducation. Birmingham, UK: SRHE and OU Press.
Laurillard, D. (2002). Rethinking university teaching: a framework for the effective use of learning
technologies (2nd ed.). London and New York: Routledge/Falmer.
Novak, J. D. (1998). Learning creatingandusing knowledge: concept mnaps asfacilitativetools in schools
and corporations.Mahaw, New Jersey and London: Lawrence Erlbaum.
Novak, J. D. (2005). Results and implications of a 12-year longitudinal study of science concept
learning. Research in Science Education, 35, 23-40.
Novak, J. D. &Musonda, D. (1991). A twelve-year longitudinal study of science concept learning.
Anierican EducationalResearch Journal,28, 1. 117-153.
Oliver, R. & Herrington, I. (1995). Developing effective hypermedia instructed materials. Austra-
lianJournal of EducationalTechnology, 11, 2, 8-22.
Shuell, T. J. (1986). Cognitive conceptions of learning. Review of EducationalResearch, 56, 411-
436.
Tyler, R. W. (1949). Basicprinciplesof curriculum and instruction.Chicago: University of Chicago
Press.
Valkanos, N., Georgiadou, E., Hatzipanagos, S., Berki, E., Siakas, K. (2005). ADDURI: a
framework for the development and support of resource-based learning environments and
Materials, in Proceedings of International Workshop On E-Learning Online Communities Eloc
2005, January 3, 2005, Cairo, Egypt, 129-133, London: Middlesex University Press.

0 2008 The Authors. Journal compilation 0 Becta 2008.

 e-L earning quality 1055

Appendix 1: A concept map of the MRI tutorial

rr*odi
t~io o tt
W-*'Od

ito

ms do is used

Da0 tol flt'QM

- ~~~~~00~~~~~
hXWO two_

terd
W-lgg

bodef.0d k
In fgmiomoe
OtosMSd
______ . mootndb ititootiopykeomt*by

0 2008 The Authors. Journal compilation 0 Becta 2008.

1056 BritishJournalof EducationalTechnology Vol139 No 6 2008

Appendix 2: The assessment framework for 'expert' scoring of student

concept maps

8 -

CONCEPTUAL RICHNESS
1. The map uses all of the concepts that are important to the topic DOOMM
DWLID
2. The map fails to include most of the important concepts

3. All of the concepts used in the map are appropriate to the topic

4. Most of the concepts used are actually irrelevant to the topic WWLI:
LINK QUALITY

5. All of the possible links are identified & explained WWDDI

EIMOMW
6. Most of the important links between the concepts are neglected wDwmm
7. All of the links in the map are relevant and correctly labelled WWDTD
8. Most of the links are irrelevant or inadequately explained
EVIDENCE of UNDERSTANDING

9. All of the propositions in the map show evidence of understanding

MMMOD
10. Most of the propositions in the map are actually misconceptions

11. The map is a clear definition of the topic

12. The map fails to provide a clear definition for the topic
HIERARCHY and STRUCTURE
ENDDE
13. The map shows ajustifiable hierarchy

14. There is no justifiable hierarchy to the map

EDDEE
15. The over-arching structure of the map is a complex network
3DDEE
16. The map is simple (it is a spoke or a chain but not a network)

0 2008 The Authors. Journal compilation 0 Becta 2008.

COPYRIGHT INFORMATION

TITLE: Measuring the quality of e-learning

SOURCE: Br J Educ Technol 39 no6 N 2008

The magazine publisher is the copyright holder of this article and it

is reproduced with permission. Further reproduction of this article in
violation of the copyright is prohibited. To contact the publisher:
http://www.blackwellpublishers.co.uk/asp/