Assessment
Evaluating assessment: the missing link?
S L Fowell,1 L J Southgate2 and J G Bligh1
Background Methods chosen for assessment and the
manner in which they are applied are so intimately as-
sociated with how individuals learn that developing
appropriate assessment strategies is a key part of ef-
fective curriculum development.
The assessment cycle We describe a four-stage assess-
ment cycle identifying important steps in assessment.
Each step is described in detail, stressing its key aspects,
including: the need for clear assessment policy and
strategy, the importance of assessment blueprints in
planning assessment, the need for effective feedback
when presenting results, and the essential, but often
overlooked, need to evaluate the assessment process
itself.
Evaluating assessment This ®nal evaluation stage is the
most important part of the assessment cycle and can be
divided into four levels. The ®rst level includes evalu-
ating each question in the assessment, the second level
is concerned with establishing validity and reliability,
the third level centres on the assessment process and
Introduction
Most medical educators involved in curriculum plan-
ning and development recognize the interplay between
assessment and learning, and that to a large extent as-
sessment drives learning. Developing an appropriate
assessment strategy is a key part of effective and sus-
tainable curriculum development. Devising suitable
assessment systems is not a one-off process. Four dis-
tinct steps make up the assessment cycle:
1. planning and preparation;
2. developing and carrying out assessment;
1
Department of Health Care Education, The University of Liverpool,
3rd Floor, University Clinical Department, Duncan Building, Liver-
pool L69 3GA, UK
2
University College London Medical School, 4th Floor, Archway
Wing, Whittington Hospital Campus, Highgate Hill, London W19
SNF, UK
Correspondence: S L Fowell, Department of Health Care Education,
The University of Liverpool, 3rd Floor, University Clinical Depart-
ment, Duncan Building, Liverpool. L69 3GA, UK
276 Ó Blackwell Science Ltd M E D I C AL ED U C AT I ON 1999;33:276±281
review of assessments by external examiners and the
fourth level involves evaluation over several assess-
ments.
Relating assessment to the curriculum This long-term
evaluation should examine whether existing assess-
ments are congruent with the curriculum and relate to
all facets of the students' learning experiences. This is
particularly important in a curriculum where the
learning outcomes of student-centred learning are em-
phasized. Changes in the assessment of postgraduate
trainees and increasing emphasis on peer review of
clinicians will raise the pro®le of these outcomes in
undergraduate education.
Keywords Undergraduate medical education,
*organization, *educational assessment curriculum,
programme evaluation, feedback, Great Britain.
Medical Education 1999;33:276±281
3. presenting the results;
4. evaluating the assessment.
Steps 1 and 4 pose major challenges to medical edu-
cators and curriculum planners. In practice, step 4 is
often omitted, although it is this evaluation phase that
in¯uences the quality and is crucial to the development of
an effective assessment. Increasingly, it will be important
in withstanding legal challenge as high stakes assess-
ments in¯uence career progression at all stages of med-
ical education and training. Brown and her colleagues
have described a 10-point manifesto based on core ed-
ucational principles underlying assessment.1 These are
shown in Table 1 and underpin our argument.
Step 1. Planning and preparation
The most important aspects of this stage of the as-
sessment cycle are devising an assessment policy and
developing a strategy for assessment. First, the overall
purposes of assessment and its role in the curriculum
Evaluating assessment: the missing link? · S L Fowell et al. 277

Table 1 An `Assessment manifesto' taken from Brown et al.1

1 Assessment should be based on an understanding of how students learn. Assessment should play a positive role in the
learning experiences of students.
2 Assessment should accommodate individual differences in students. A diverse range of assessment instruments and
processes should be employed, so as not to disadvantage any particular individual or group of learners. Assessment
processes and instruments should accommodate and encourage creativity and originality shown by students.
3 The purposes of assessment need to be clearly explained. Staff, students and the outside world need to be able to see
why assessment is being used and the rationale for choosing each individual form of assessment in its particular context.
4 Assessment needs to be valid. By this, we mean that assessment methods should be chosen which directly measure that
which it is intended to measure, and not just a re¯ection in a different medium of the knowledge, skills or competences
being assessed.
5 Assessment instruments and processes need to be reliable and consistent. As far as is possible, subjectivity should be
eliminated, and assessment should be carried out in ways where the grades or scores that students are awarded are
independent of the assessor who happens to mark their work. External examiners and moderators should be active
contributors to assessment, rather than observers.
6 All assessment forms should allow students to receive feedback on their learning and their performance. Assessment
should be a developmental activity. There should be no hidden agendas in assessment, and we should be prepared to
justify to students the grades or scores we award them, and help students to work out how to improve. Even when
summative forms of assessment are employed, students should be provided with feedback on their performance, and in-
formation to help them identify where their strengths and weaknesses are.
7 Assessment should provide staff and students with opportunities to re¯ect on their practice and their learning.
Assessment instruments and processes should be the subject of continuous evaluation and adjustment. Monitoring
and adjustment of the quality of assessment should be built in to quality control processes in universities and
professional bodies.
8 Assessment should be an integral component of course design, and not something bolted on afterwards. Teaching
and learning elements of each course should be designed in the full knowledge of the sorts of assessment students will
encounter, and be designed to help them show the outcomes of their learning under favourable conditions
9 The amount of assessment should be appropriate. Students' learning should not be impeded by being driven by an
overload of assessment requirements, nor should the quality of the teaching conducted by staff be impaired by excessive
burdens of assessment tasks.
10 Assessment criteria need to be understandable, explicit and public. Students need to be able to tell what is expected
of them in each form of assessment they encounter. Assessment criteria also need to be understandable to employers,
and others in the outside world

must be made explicit, with time taken to consider why
each individual assessment is proposed. Is the assess-
ment formative or summative? Which aspects of the
course are to be assessed? Which learning objectives are
relevant to the assessment? Are you principally inter-
ested in the course as de®ned by the written curricu-
lum, or by other, wider, learning activities? How will
standards be set? Will pass/fail decisions be based on a
norm-referenced system where the pass mark is ad-
justed to allow a set number of candidates to pass, or
will some method of criterion-referencing, in which the
pass mark is based on predetermined standards, be
employed?2±4
Use of an assessment blueprint
The selection of assessment methods must follow and
not precede the resolution of these issues. The whole
process should start with devising a `blueprint', test-
matrix or table of speci®cations for each assessment, in
which the objectives to be assessed are matched to the
assessments.5 Blueprints vary in complexity, from
an indication of the number of questions taken from
each subject area, module or course, to a more complex
matrix relating content to behavioural objectives. At a
more detailed level, it is also possible to indicate
weightings and question types. Examples of different
blueprints are given by Newble et al. and O'Donnell
et al.6,7 Table 2 shows an outline matrix, based on an
example given by Harden and Gleeson for a clinical
medical examination8 in which the content area is re-
lated to speci®c behavioural objectives.
We recommend the use of a diverse range of assess-
ment methods. All individual methods disadvantage
some students, and, in addition, using a selection of
methods allows `triangulation' with evidence relating to
particular aspects of performance arising from different
sources. The planning phase also includes deciding the
nature and timing of assessments, paying attention to
separating formative and summative assessments, al-
though the same methods may be used for the two
purposes. Importantly the planning phase also includes

Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:276±281

278 Evaluating assessment: the missing link? · S L Fowell et al.
Table 2 Assessment blueprint for a clinical medical examination relating subject area to be tested to behavioural objectives (adapted from
Harden and Gleeson8). Examples of possible assessments are identi®ed in the ®rst row
Behavioural objectives
Subject area Take a history
Weighting 30%
Cardiovascular Take history from
system a patient with angina
Respiratory
system
Central nervous
system
Genito-urinary
system
designing appropriate staff training in devising, imple-
menting and evaluating all aspects of the assessment
programme.
Step 2. Developing and implementing
assessment
Once the assessment policy and strategy are agreed, the
next phase of the cycle follows. This involves preparing
individual assessments, selecting appropriate assess-
ment methods from those identi®ed in the planning
phase and writing items for the assessment. It is im-
portant to refer to the assessment blueprint to ensure
coverage of the curriculum throughout this stage in
order to ensure content validity. This phase should be
strengthened by staff-development to familiarize faculty
with the proposed question formats. `Item writing'
parties are a useful strategy for obtaining questions and
informing the teachers about test construction.9 The
preparation of questions includes the development of
model answers for written responses, and tick lists or
rating forms for observed assessments, together with
the proposed marking schemes. Decisions about how
the standard will be set should be taken during this
stage. If absolute standards are proposed, criteria should
be developed and documented by a group of subject
experts, with explicit statements de®ning the perfor-
mance expected of candidates at different levels of
achievement. In high stakes assessments particular at-
tention should be paid to precision at the pass/fail level.
If the results of assessments are to be combined, or
rules for compensation are proposed, care must be ta-
ken to avoid one component having unintended undue
Conduct a Interpret information Give information to
physical examination and data patients
20% 20% 20%
Examine a patient Interpret chest X-ray Give advice to a patient
with mitral stenosis showing left following a myocardial
ventricular hypertrophy infarction
in¯uence on the overall result. Harmonizing the plan-
ned and the achieved weights of the components is a
technical task, but it is important that the decision
makers understand the need for it, and that the candi-
dates know how scores are to be adjusted and com-
bined, and what pass fail rules will be adopted.10 This
has a profound effect on how candidates will prepare
for the assessment and how they will view the impor-
tance of learning the material it addresses.
Step 3. Presenting the results
During this part of the cycle the assessments are
marked and the results presented. Bias is avoided by
anonymous and double marking of written responses
(where feasible). Where there are too many candidates
for one individual to mark all their responses, one ex-
aminer should mark the responses to one item by all
candidates (rather than all the items by a few candi-
dates). This reduces the problems that can arise when
examiners differ in the rigour with which they apply the
marking scheme.
For assessments such as OSCEs where the number of
examiners is a limiting resource, higher reliability is
achieved by increasing the number of stations with each
station marked by a single examiner, rather than having
fewer stations marked by multiple observers.11
Providing feedback to students
Presentation of results also requires providing effective
feedback to students. This is an integral part of for-
mative assessments and, arguably, similarly important
Ó Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:276±281
Evaluating assessment: the missing link? · S L Fowell et al.
for summative assessments. Students often receive little
feedback from summative assessments other than an
indication of the grade or mark obtained. Meaningful
feedback enables students to identify their strengths
and weaknesses and helps them improve their subse-
quent performance.12,13 Feedback can take many forms
and can be written or oral, and given on an individual
or group basis. Guidelines for ef®cient feedback include
using standardized feedback forms and computer banks
of commonly used statements. Effective feedback
should be immediate or given as soon after the assess-
ment as is feasible.14 Studies have shown that students
value immediate feedback from OSCEs.15
Step 4. Evaluating the assessment
The evaluation of assessment is frequently left out of
the assessment cycle in the general hurly burly of the
academic year. To our minds, this is perhaps the most
important stage, driving the assessment cycle and
prompting further developments. Evaluation can also
feed into curriculum planning by highlighting the
strengths and weaknesses of the curriculum.
First level evaluation
The evaluation step can be divided into 4 levels. The
®rst level relates to the individual questions or items
that make up the assessment. Computer packages for
marking multiple choice questions often provide in-
built item analysis, calculating the dif®culty index
(percentage of respondents who gave the correct re-
sponse) and discrimination index (a comparison of the
numbers of high scoring and low scoring students who
gave the correct response) for each question. Inspection
of the responses and scores in both written tests and
OSCEs gives an indication of the quality of the candi-
dates' and the model answers. Were the students able
to answer each question or perform each task? If not, it
is important to consider why. Have students not un-
derstood the question? Was the question or OSCE
station poorly written or ambiguous? Does the question
match the learning objectives?
Second level evaluation
Second level evaluation involves establishing validity
and calculating reliability of the assessment.
If the assessment has been developed against a
blueprint, content validity (does the assessment mea-
sure what it is intended to measure?) is built in. How-
ever, it is important to consider predictive validity. For
example, if students pass an assessment of communi-
Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:276±281
279
cation skills in year 1, but subsequently fail the com-
munications skills element of an assessment of clinical
competence, this might indicate problems with the va-
lidity of either assessment (or an unfortunate effect of
the course!)
Reliability refers to the extent to which the assessment
provides a consistent measure of whatever it is mea-
suring. Tests of competence which generate numerical
scores should report a measure of internal consistency
(Cronbach alpha) with a level of 0á8 or greater. But the
most important source of error in¯uencing the reliability
of a test stems from the consistently documented ®nding
of `content speci®city'. The performance of a candidate
on one test item is poorly correlated with performance
on another, and is speci®c to content.16 To achieve
generalizable or stable scores many test elements are
required, leading to the need for longer testing time and
the search for ef®cient test methods.
Third level evaluation
The external examiner has a crucial role as part of the
third level of the evaluation step. He or she should be
involved in evaluation of both the assessment process
and the review of assessments. This involves inspection
of the assessments and model answers before the as-
sessment has taken place, and review of a selection of
marked papers. A thorough critique of questions and
expected responses, prior to the assessment, should
help to identify any problems around the validity of the
assessment and provide feedback on whether the as-
sessment is at an appropriate level. External examiners
should also attend non-written assessments
(e.g. OSCEs) to observe proceedings. Obviously, it is
unlikely that it will be feasible or desirable for all ex-
ternal examiners to attend all such assessments; ideally
an external examiner should be present at a sample of
the assessments to enable him or her to observe the
organization and ensure fairness. The presence of the
external examiner provides an opportunity to evaluate
and comment on the face validity of the assessments.
The evaluation phase offers an opportunity to review
the pass/fail standards set for the assessment. This can
occur both at the level of the individual items that
comprise the assessment, and at the level of the as-
sessment as a whole. In each case, this is a useful point
at which to consider whether prespeci®ed standards are
at an appropriate level.
Fourth level evaluation
Levels 1±3 include the more immediate aspects of
evaluation. Level 4 involves taking a `long-term' view of
280 Evaluating assessment: the missing link? · S L Fowell et al.

Figure 2. (a) The relationship between the taught, learned and

planned curriculum.
Figure 1. The assessment cycle.

assessment and maintaining a weather eye for trends. If

it becomes apparent that several cohorts of students are
performing badly in one objective, skill or subject area
this may point to a de®ciency in the curriculum. Once
identi®ed, this may lead to development of new as-
sessments, thus starting another round of the assess-
ment cycle.

Assessing the parts we don't normally assess.

Is assessment congruent with all aspects
of the curriculum?
Besides evaluating the technical aspects of assessment
at the levels described above, it is also important to
consider the wider issue of how assessments relate to
the total learning experience encountered by students.
Inevitably, there is a disparity between the curriculum
`in action' which is delivered to the students, and the
`curriculum experience' as received by the students. In
Fig. 2(a), each solid circle represents one facet of the
curriculum; the planned or written curriculum, the
taught curriculum which is delivered to the students via
a variety of mechanisms and the learned curriculum
which the students experience. Area (a) represents
learning that occurs outside the planned and taught
curricula as a result of peer and social in¯uences and
the climate of learning. This includes favourable in¯u-
ences, for example, adopting appropriate dress for
clinical contact, developing group work and self study
skills and less desirable in¯uences such as adopting
short-term learning strategies and reinforcing social
stereotypes.
Assessment of area (b) represents assessment `by
ambush' and should be avoided. A great deal of
Figure 2. (b) The relationship between assessment and facets of
the curriculum.
importance is placed on the desirable learning out-
comes of student-centred learning which are included
in area (a). These considerations are crucial in
innovative, student-centred courses; in particular
those involving problem-based learning, where stu-
dent identi®ed learning objectives are so important.
Such important, but unidenti®ed issues may include
team working and communication skills, and also
social skills and independent learning skills. These
skills are not usually `examined' and may not be
assessed at any time in the formal curriculum.
However, changes in the way in which postgraduate
trainees are to be assessed and the present emphasis
on peer review of performance of established
clinicians17,18 will ensure that greater attention to the
assessment of these qualities will soon occur in the
undergraduate curriculum.

Ó Blackwell Science Ltd M E D I C A L ED U C A T I ON 1999;33:276±281

Evaluating assessment: the missing link? · S L Fowell et al.
Conclusion
Effective assessment is a continuous cycle of develop-
ment, implementation, presentation and evaluation. As
part of the evaluation phase of the cycle, medical edu-
cators should consider whether the assessment process
is congruent with the students' total learning experi-
ence. This presents a major challenge to identify those
learning outcomes which are not assessed and to de-
velop effective methods for their assessment.
References
1 Brown S, Race P, Smith B. An assessment manifesto. 500 Tips
on Assessment. London: Kogan Page; 1996. p.142±3.
2 Lowry S, Assessment of Students. BMJ 1993;306:51±4.
3 Turnbull JM. What is normative versus criterion-referenced
assessment? Med Teacher 1989;11:145±50.
4 Cusimano MD. Standard setting in medical education. Acad
Med 1996;71:S112±S120.
5 Newble D, Dauphinee D, Macdonald M, Mulholland H,
Dawson B, Page G, Swanson D, Thomson A, van der
Vleuten C. Guidelines for assessing clinical competence.
Teach Learn Med 1994;6:213±9.
6 Newble DI, Hoare J, Elmslie RG. The validity and reliability
of a new examination of the clinical competence of medical
students. Med Educ 1981;15:46±52.
7 O'Donnell MJ, Obenshain SS, Erdmann JB. Background es-
sential to the proper use of results of Step 1 and Step 2 of the
USMLE. Acad Med 1993;68:734±9.
8 Harden RM, Gleeson FA. Assessment of clinical competence
using an objective structured clinical examination (OSCE).
Med Educ 1979;13:41±54.
Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 1999;33:276±281
281
9 Case SM, Swanson DB. Extended matching items: a practical
alternative to free-response questions. Teach Learn Med
1993;1:107±15.
10 Foulkes J. Combining components of assessment. In: Newble
D, Jolly B, Wakeford R, editors. The Certi®cation and
Recerti®cation of Doctors. Issues in the Assessment of Clinical
Competence. Cambridge: Cambridge University Press; 1994.
p.134±50.
11 Newble DI, Swanson DB. Psychometric characteristics of the
objective structured clinical examination. Med Educ
1988;22:325±34.
12 Brown S, Knight P. Purposes of assessment. Assessing Learners
in Higher Education. London: Kogan Page; 1994. p.30±41.
13 Hodder RV, Rivington RN, Calcutt LE, Hart IR. The effec-
tiveness of immediate feedback during the OSCE. Med Educ
1989;23:184±8.
14 Brown S, Knight P. Speedier assessment: Ways of giving
feedback. Assessing Learners in Higher Education. London:
Kogan Page; 1994. p. 112±3.
15 Black NMI, Harden RM. Providing feedback to students on
clinical skills by using the Objective Structured Clinical Ex-
amination. Med Educ 1986;20:48±52.
16 van der Vleuten CPM, Norman GR, de Graaff E. Pitfalls in
the pursuit of objectivity ± issues of reliability. Med Educ
1991;25:110±8.
17 Irvine D. The performance of doctors: I. Professionalism and
self-regulation in a changing world. BMJ 1997;314:1540±2.
18 Irvine D. The performance of doctors. II. Maintaining good
practice, protecting patients from poor performance. BMJ
1997;314:1613±5.
Received 13 January 1999 accepted for publication 19 January 1999