Medical Teacher, Vol. 28, No. 6, 2006, pp.
535–543
Assuring the quality of high-stakes undergraduate
assessments of clinical competence
CHRIS ROBERTS1, DAVID NEWBLE2, BRIAN JOLLY3, MALCOLM REED4 &
KINGSLEY HAMPTON4
1
Office of Teaching and Learning in Medicine, University of Sydney, Australia; 2Department of
Medical Education, Flinders University, Adelaide, Australia; 3Centre for Medical Health Sciences
Education, Monash University, Melbourne, Australia; 4Sheffield Teaching Hospitals Trust and the
University of Sheffield, Sheffield, UK
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
ABSTRACT In the UK, and in many Commonwealth countries, a
university degree is accepted by registration bodies as an indication
of competence to practice as a PRHO or intern. Concerns have
been raised that the quality of university examinations may not
always be sufficient for such high-stakes decision-making.
Assessments of clinical competence are subject to many potential
sources of error. The search for standardization, and high validity
and reliability, demands the identification and reduction of
measurement errors and biases due to poor test design or variation
in test items, judges, patients or examination procedures.
Generalizability and other research studies have identified where
the likely sources of error might arise and have been taken into
For personal use only.
account in the development of published guidelines on interna-
tional best practice, which institutions should strive to follow. The
purpose of this paper is to describe the development of
the integrated final-year assessment of clinical competence at the
University of Sheffield. The aim was to introduce a range of
strategies to ensure the examination met the best practice
guidelines. These included blueprinting the assessment to achieve
a high degree of content validity; lengthening the examination
by adding a written component to the OSCE component to ensure
an adequate level of reliability; providing training and feedback
for examiners and simulated patients; paying attention to item
development; and providing statistical information to assist the
examination committee in standard setting and decision-making.
This evidence-based approach should be readily achievable by all
medical schools.
Background
Over a century ago, concerns about the quality of medical
degrees in North America led to the establishment of
independent national examinations, such as those conducted
by the National Board of Medical Examiners (NBME),
which all students have to pass in order to obtain a license
to practice. In the UK, and many Commonwealth countries,
the award of a degree still certifies graduates as competent to
practice as a Pre-registration House Officer (PRHO), or
intern, having achieved the attributes defined in the General
Medical Council’s (GMC) document Tomorrow’s Doctors
(GMC, 1993, 2003). The accreditation process of the GMC
and the university system of Visiting Examiners are seen as
providing a guarantee of standards (Newble, 2001).
There is increasing disquiet as to whether the quality of
university assessments in the UK is good enough to properly
ISSN 0142–159X print/ISSN 1466–187X online/06/060535–9 ß 2006 Informa UK Ltd.
DOI: 10.1080/01421590600711187
Practice points
. There has been increasing awareness that summative
assessments used for high stakes decision-making
should demonstrate a high degree of validity and
reliability.
. Research has shown that best practice is not always
achieved in university-run clinical competence
examinations.
. A set of best practice guidelines has been developed by
an international group of experts in assessment, which
outlines the principles and steps required to ensure
such procedures are defensible to both internal and
external scrutiny.
. A description of how an institution has developed and
introduced a defensible high-quality assessment in
practice, using its own resources as opposed to a
national institution such as the NBME, should be of
interest to all in medical schools with the responsibility
of certifying the competence of their graduates.
reassure the public that PRHOs will be competent regardless
of their medical school of qualification (Fowell & Bligh,
2001). A recent survey concluded that educationally undesir-
able assessment methods and practices are still used by many
medical schools, which in part appeared to be due to an
apparent lack of knowledge of the technical aspects of
assessment, including the application of robust approaches
to standard setting (Fowell et al., 1998). Consequently, some
medical schools may be making pass/fail decisions on
students’ fitness to practice based on assessments that are
prone to error. This is worrying in the light of evidence that
students who are on the borderline of pass/fail decision-
making may graduate to become doctors whose clinical
performance will give cause for concern (Tamblyn et al.,
1998). Additional emphasis on this issue has come from
a national expectation that the public will be protected
from incompetent doctors. While the main media and
Correspondence: Chris Roberts, Associate Professor in Medical Education,
Office of Teaching and Learning in Medicine, Faculty of Medicine (A27),
Edward Ford Building, University of Sydney, NSW 2006, Australia. Tel: þ61
2 9036 9453; fax: þ61 2 9351 6646; email: chris.roberts@med.usyd.edu.au
535
 C. Roberts et al.
political attention has focused on the performance of doctors
in practice (Bristol Royal Inquiry Report, 2001; Southgate
et al., 2001; Shipman Inquiry, 2005) universities are obliged
to play their part by ensuring they graduate only those
students who have reached the required level of competence.
Dissatisfaction with the present situation in the UK has
led, in some quarters, to calls for the establishment of
a national medical qualifying examination similar to the
National Board of Medical Examiners (NBME) United
States Medical Licensing Examination (Patel, 2001;
Wakeford, 2001; Bligh, 2003). However, the clinical compe-
tence examinations developed by the NBME are complex
and expensive, rely heavily on simulated patient-based (SP)
ratings, and have to be undertaken at national testing centres.
Their quality assurance system (Boulet et al., 2003) depends
on the application of psychometric techniques available only
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
within a few UK research-based centres.
In this paper, we describe an integrated final-year assess-
ment of clinical competence, and the accompanying quality
assurance mechanisms, which have been developed over
the last six years at the University of Sheffield. These ensure
that the examination is valid, reliable and feasible to conduct
in a university setting so that student scores are an accurate
measure of their true ability and reasonably free from
measurement error. We expect the principles we demonstrate
should be of interest to those in medical schools with the
responsibility of certifying the competence of their graduates.
For personal use only.
Principles of examination development
There has been increasing awareness that summative assess-
ments used for high-stakes decision-making should demon-
strate a high degree of validity and reliability. To assist in this
process a set of best practice guidelines has been developed by
an international group of experts in assessment outlining the
principles and steps required to ensure procedures are
defensible to both internal and external scrutiny (Newble
et al., 1994). There are four broad areas for an examination
committee to consider. These are: blueprinting to ensure
content validity; selection of best test formats; applying
strategies to achieve adequate levels of reliability; and
instituting appropriate standard setting and decision-making
procedures.
Blueprinting
Blueprinting is the process whereby an adequate and
representative sample of items to be included in the examina-
tion is determined. In an undergraduate assessment of clinical
competence, the blueprint provides a grid or matrix, which
maps the content of the examination against the learning
outcome objectives of the course (Newble, 2002). It is the
process by which the content validity of the test is established.
Test format
No one single test format is able to assess all aspects of
clinical competence (Wass et al., 2001). The Objective
Structured Clinical Examination (OSCE) provides a format
particularly suitable for assessing many components
of clinical competence, especially clinical, technical and
practical skills, often with a high degree of fidelity
536
(Newble, 2004). The testing of relevant knowledge, including
aspects of diagnosis, investigation and management, can be
more feasibly (and cheaply) tested with written formats.
However, not all aspects of clinical competence can be validly
tested in an examination setting and some need to be part of
summative in-course assessments, for example professional
behaviours, which are often not formally assessed at under-
graduate level.
Reliability
The search for standardization in clinical competence
assessments demands the identification and reduction of
any measurement error or biases due to variation in test
items; examiners; patients; or examination procedures that
might affect the observed scores of individual examinees.
Generalizability studies have been highly influential in
providing guidance for identifying where the largest sources
of measurement error might lie and suggesting ways of
refining assessment procedures in order to enhance relia-
bility. For example, the common finding of lack of correlation
of scores across cases in clinical competence assessments
is often referred to as content specificity (van der Vleuten,
1996). Studies have shown that content specificity is the
major contributor to unreliability, much more so than
unreliability attributable to inconsistencies in marking. This
means that testing competences across a large sample of cases
has to be done before a reliable generalization as to student
performance can be made. This is the reason that OSCE
examinations have to be quite long to be reliable, irrespective
of the effectiveness of structuring marking sheets and
examiner training in reducing error. It also means that
multiple inputs will have to be collected and collated where
in-course assessments are used for summative decision-
making (van der Vleuten, 2000).
Standard setting and decision-making
Standard setting refers to the procedures applied to assess-
ments, which establishes the borderline between those
students who pass or are considered competent, and those
who should fail or be deemed incompetent (Norcini, 2003).
A number of methods have been used to set the standards for
written and clinical examinations, which are usually classified
into relative and absolute. For assessments of competence,
the latter are the most appropriate. All such methods require
the input of a number of credible judges who contribute,
in various ways, to establishing the intended standard to be
applied to the results. The standard is an important, but not
the only, contribution to the approach an examination
committee might use to make the final decision on who
should pass or fail. Whatever the procedures, they should be
carefully documented and made available to students, staff
and to external reviewers.
Development of the final year assessment of
clinical competence
Changes to clinical competence assessments at Sheffield
medical school occurred progressively over a number of years
as part of the development of a comprehensive assessment
strategy. They included modifications to the structure and

format of assessments throughout the course to make them
both integrated and standardized. This allowed the quality
assurance systems to be developed and to be embedded in
the school at both academic and administrative levels.
In particular, academics and students gained increasing
confidence that the new examination procedures would
function as intended and that the resulting decision-making
processes were acceptable to the examination committee,
the external examiners, the university, the students, and the
Postgraduate Dean responsible for PRHO placements and
training.
Blueprinting
The learning outcome objectives for clinical competence are
defined within a core curriculum. These objectives have been
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
related to 95 clinical problems which graduates are likely to
meet as PRHOs (Newble et al., 2005). They are available
to both staff and students in a searchable database accessed
via Minerva, the schools’ networked learning environment
(Roberts et al., 2003). This information forms the content of
the core curriculum pertaining to clinical competence and
the underpinning medical sciences. The aim of the final-year
assessment of clinical competence is to examine an appro-
priate sample of this curricular database. The blueprint for
the final examination has the competences on one axis
(e.g. history-taking, communication skills, practical skills,
For personal use only.
management etc.) and the problem content on the other axis,
with problems allocated as appropriate under body systems,
some therefore being represented in more than one system.
A multi-disciplinary team of medical teachers, including
experts in assessment and psychometrics, has developed
an agreed sampling procedure based on the blueprint
and information from prior examinations. Items are then
requisitioned or accessed from a bank of items used
previously, or developed from workshops as part of the
examiner training system. New items are reviewed by
the team. The proposed written and clinical components
of the examination are then sent for ratification to the
external examiners, usually clinicians who are conversant
with the best-practice assessment principles we endorse.
Test format
An OSCE was first introduced as the clinical component of
the final-year examination in June 1999. The short (five-
minute) station format was adopted, as this is the approach
being used in most medical schools in the UK (Newble,
2002) and by many licensing bodies such as the GMC
(Tombleson et al., 2000) and the Medical Council of Canada
Table 1. Changes in the number of items in the clinical competence examination in the years 2001 to 2004.
2001 2002
No. of items Total marks No. of items Total marks
Clinical 9 200 12 260
Static 3 30 3 30
Written 5 50 5 50
Overall 17 280 20 340
Assuring assessment quality of clinical competence
(Reznick et al., 1996). The initial reliability for this 16-station
examination was 0.64 (Cronbach’s alpha), which was not
considered adequate. However, it was unfeasible to
provide the four or more hours of clinical examination time
requiring at least 40 stations that generalizability studies
predict would be required to give a reliable examination
for high-stakes summative assessment of greater than 0.8
(Newble & Swanson, 1988). As an alternative strategy, in
2003, the OSCE was combined with a written component
consisting of modified essay questions (MEQs). These,
in effect, were almost identical in format to some of the
non-patient-based OSCE stations assessing the ability to
interpret investigational data such as imaging and ECGs. All
stations and written questions were still focused on clinical
tasks determined in advance from the same blueprint. This
now provided a test sample of nearly 40 items.
The examination is taken in two stages, a week apart. The
written component is undertaken first in an examination hall.
Marking of each MEQ and the written OSCE items within
the clinical component of the examination, referred to as
static stations, are marked anonymously by single examiners
using structured answer sheets. The clinical component is
run in a hospital or clinical skills centre setting. There are
four separate OSCE venues each able to conduct four circuits
in the one day. Students rotate through the circuits in groups
of 13–15, depending on the cohort size. While general-
izability studies have suggested only one examiner need be
used for scoring each OSCE station without significantly
affecting examination reliability (van der Vleuten, 1996),
enough examiners had been recruited through our training
programme to double up for added fairness and reassurance
to the students. It adds to collegiality and to faculty
development (e.g. less experienced examiners can be
teamed with more experienced examiners). In the interests
of examination security, students are corralled so there is no
contact between students who have taken the examination
and those about to take it (Swanson et al., 1995).
At clinical stations, a mixture of simulated and
real patients are used. Simulated patients are drawn from a
pool of volunteers and actors, managed by a coordinator.
They are trained on their clinical scenario by clinical skills
staff and senior clinical examiners prior to the examination.
Real patients with well-established physical signs are involved
in at least three of the physical examination stations. One
physical examination station is a 10-minute station, accorded
double marks, allowing time for students to complete a more
extensive task.
The ways in which the refinements to the structure of the
whole examination have been introduced during the period
2001–04 are given in Table 1.
2003 2004
No. of items Total marks No. of items Total marks
10 220 10 220
3 30 3 30
26 260 26 260
39 510 39 510
537
 C. Roberts et al.

Table 2. Typical structured OSCE mark sheet showing marking scheme.

Performed
Performed but not Not performed
competently fully competent or incomptent

Initial approach to the patient 2 1 0

(introduces him/herself,
explains what he/she will be doing)
Periodicity
Previous episodes 1 0.5 0
Single or multiple 1 0.5 0
Associated symptoms
Loin pain 1 0.5 0
Abdominal pain 1 0.5 0
Symptoms of cystitis 1 0.5 0
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14

Drug history 1 0.5 0

Social history
Smoking 1 0.5 0
Occupation 1 0.5 0
Diagnosis
Bladder cancer 1 0.5 0
Kidney cancer 1 0.5 0
Investigations
MSU 0.5 0 0
U & Es 0.5 0 0
Imaging (IVU, USS) 1 0.5 0
Cystoscopy 1 0.5 0
For personal use only.

Overall approach to task 5 4 3 2 1 0

Total (max 20)
Overall rating of station Clear Fail Borderline Clear Pass

Scoring and standard setting
Examiners receive details of the station they will mark several
days prior to the examination. For the written component
and static OSCE stations, questions and the marking sheet
reflect the aim of assessing relevant knowledge and clinical
problem-solving with particular emphasis on diagnosis,
investigations and patient management. For the observed
stations in the clinical component, a briefing on the day
reinforces the rules about the marking system and the
standard-setting procedure. On the examiner marking
sheet, the checklist items reflect the relevant history, physical
examination or practical skills the student should obtain or
perform. There are generally 8–12 checklist items for each
case (see Table 2). Checklist items are weighted to avoid
trivialization. Candidates are rated on each item as: having
not performed or having demonstrated incompetence on the
item (awarded no marks), having performed the item but not
to the required level of competence (awarded half the marks
for the task) or having performed the item competently to
the level of a starting PRHO (awarded the full mark for this
item). Additionally, for both written and observed questions/
stations examiners are asked to provide a global rating,
independently of their checklist scores, as to whether in their
opinion the student had passed, failed or was borderline for
competence on her/his performance of the overall task. This
information is required for the standard-setting procedure.
Whilst various standard-setting methods are used for written
and clinical examinations (Norcini, 2003), the borderline
method was adopted (Smee, 2001; Wilkinson et al., 2001). In
this approach, the borderline score for each question/station
was calculated as the median score of all students identified
as borderline by the examiners. At OSCE stations, where
there were two examiners, their marks were averaged. The
overall borderline score for the whole examination comprises
the sum of the borderline scores for all questions/stations
(Table 3).
Performance of the examination
A summary of the examination results and psychometric data
(Streiner & Norman, 2003) is provided to the examination
board to support decision-making (Tables 3–5). The overall
reliability (internal consistency) of the fully developed
examination in 2003 and 2004 was 0.81 using Cronbach’s

(Table 4). This is above the conventional ‘gold standard’
of 0.8 and confirms that the item sample is now large enough
for high-stakes decision-making purposes. The Standard
Error of Measurement (SEM) of the examination is used
for grading and decision making (Table 5). Various sub-
analyses provide insights into the performance of different
components of the examination (i.e. test items, examiners,
patients or examination procedures), for quality assurance
purposes. We have illustrated the appropriate tests and
interpretations of data with some examples.

538
 Assuring assessment quality of clinical competence

Table 3. Median borderline scores, correlations and marks available for each blueprinted question/station in the examination.

Correlation
to total Max. Median
minus marks Borderline
Station/MEQ ID Competence System Problem item available score

Station1 Physical exam CNS Abnormal Gait 0.30 40 23.50

Station2 (static) Data (X-ray) Resp Wheeze 0.17 10 6.00
Station3 Physical exam GI Abdominal mass 0.26 20 11.75
Station4 History Resp Haemoptysis 0.21 20 12.00
Station6 Patient educ/comms Skin/Misc Rash 0.30 20 13.00
Station7 History Endocrine Weight loss 0.21 20 13.50
Station8 (static) Practical skills Oncology Pain 0.18 10 5.00
Station9 History Eyes/ENT Visual disturbance 0.27 20 12.00
Station10 Physical exam MSS/CNS Laceration 0.32 20 11.00
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14

Station11 Practical skills CVS Collapse 0.31 20 10.50

Station12 Physical exam Oncology Breast lump 0.30 20 13.00
Station14b Patient educ/comms Oncology Dying patient 0.27 20 12.50
Station15 (static) Data Skin/Misc Itch 0.26 10 6.00
Written1 Problem solving/Dx/Manage CVS Claudication 0.47 10 6.00
Written2 Problem solving/Dx/Manage CVS/Endo Palpitations 0.52 10 6.00
Written3 Problem solving/Dx/Manage CVS Heart failure 0.28 10 6.00
Written4 Problem solving/Dx/Manage Resp Haemoptysis 0.31 10 7.00
Written5 Problem solving/Dx/Manage Mental Mania 0.31 10 5.00
Written6 Problem solving/Dx/Manage ENT Hearing difficulty 0.42 10 4.00
Written7 Problem solving/Dx/Manage CNS Coma 0.11 10 7.00
Written8 Problem solving/Dx/Manage MSS Joint pain 0.47 10 5.00
For personal use only.

Written9 Problem solving/Dx/Manage MSS Fracture 0.32 10 5.00

Written10 Problem solving/Dx/Manage GI Diarrhoea 0.41 10 7.00
Written11 Problem solving/Dx/Manage GI Haematemesis 0.52 10 6.00
Written12 Problem solving/Dx/Manage Mental Overdose 0.25 10 5.00
Written13 Problem solving/Dx/Manage Endocrine Weight loss 0.40 10 7.00
Written14 Problem solving/Dx/Manage Oncology Breast lump 0.40 10 6.00
Written15 Problem solving/Dx/Manage Haem Bleeding 0.42 10 5.00
Written16 Problem solving/Dx/Manage Oncology Testicular mass 0.34 10 4.00
Written17 Problem solving/Dx/Manage Eyes Red eye 0.46 10 5.00
Written18 Problem solving/Dx/Manage Endocrine Hairy 0.28 10 5.00
Written19 Problem solving/Dx/Manage CNS Weakness 0.41 10 5.50
Written20 Problem solving/Dx/Manage GI Vomiting 0.42 10 7.00
Static1 Investig/Interpretation CNS Headache 0.43 10 6.00
Static2 Investigation/Interpretation GI Jaundice 0.12 10 5.00
Static3 Investigation/Interpretation Resp SOB 0.17 10 6.00
Static4 Investigation/Interpretation Endocrine Dehydration 0.30 10 5.00
Static5 Investigation/Interpretation CNS Consciousness 0.16 10 6.00
Static6 Investigation/Interpretation ENT Abnormal Hearing 0.21 10 6.00
Overall borderline score 510 297.25
Overall borderline score % 100 58.28

Table 4. Summary of examination data 2001–04.

Mean score Overall borderline Standard Standard error of Pass mark

Number of students (%) score (%) deviation measurement (%) Reliability

2004 195 73.25 58.28 5.50 2.40 60.68 0.81

2003 215 74.4 58.41 5.77 2.52 60.93 0.81
2002 214 73.0 59.81 4.95 2.98 62.79 0.64
2001 195 71.85 58.90 5.86 3.21 62.11 0.70

539
 C. Roberts et al.
Table 5. Summary of provisional grading procedure for 195 students showing bands of 1SEM (2.4%) from the overall
borderline score of 58.28%.
Grade Definition of Band
5 2 SEMs from Top mark
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
4 1 SEM
3 1 SEM
1 SEM Above Borderline Score
2 Borderline
For personal use only.
1 SEM Below Borderline Score
1 1 SEM
Question/station development
The Pearson correlation between a station score and the
average of all the other question/station scores in the
examination provides an indication of how well that ques-
tion/station is functioning (see Table 3). A level above 0.25
is desirable. Those questions/stations returning less are
reviewed as they may benefit from re-writing if they are to
be used again.
Examiner issues
Performance of the examiners is investigated in several ways.
For example, median borderline scores are inspected to see
if for some reason examiners are producing an inappropri-
ately low borderline mark, which might affect the overall
standard. This is most likely to happen in the written
components where a single examiner is used for each
question. This problem is less likely to occur in the clinical
component, because of marks being averaged out across
many examiners.
540
Status Range No Of
Students
Eligible for 80.79–85.59 16
distinction
172
Good pass 63.08–80.78
Pass 60.68–63.07
3
55.89–60.67 4
Fail <55.88
0
Inter-rater reliability is examined across all examiner
pairings for patient-based stations. Overall, there is good
general agreement between examiners. In 2004 Pearson’s
correlations ranged from 0.54 to 0.86. In this examination,
the lowest correlations were at the two patient education/
communication skills stations; items 6 and 14 (see Table 3).
A more detailed inspection of individual examiner pairs across
OSCE sites showed poor correlations in several examiner
pairs on these two stations. In this case, it suggested a need to
pay more attention to examiner training for communication
skills stations to reduce the effect of assessor subjectivity.
In the written component of the examination, a random-
ized sample of four questions is double marked. Inter-rater
agreement ranged from 0.72 to 0.87 showing that examiners
were in good agreement and that our strategy to use single
markers for the written component was defensible. However,
it should be remembered that experienced and trained
examiners were used for this marking task.
Examiners receive feedback on the overall performance
of the examination, as well as appreciation for participating.

80.00
Mean OSCE % score with 95% CI
75.00
70.00
65.00
60.00
B F
Succesive OSCE student groups at one venue
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
Figure 1. Exploring order effects at one OSCE site for four groups of students.
Examination procedures
Within such a complex OSCE examination conducted at
multiple sites with multiple rotations, it is valuable to
investigate possible biases caused by order or location effects.
In most years, it has been possible to show that no such biases
existed. However, in 2004 a significant difference in the
mean scores was detected between two consecutive rotations
For personal use only.
of students at one venue using an analysis of variance
(see Figure 1). This initially suggested a breach of examina-
tion security. However, investigation showed that the
differences between the groups concerned (B and F) were
not evident once the students’ performance had been
adjusted relative to their written component marks. Further
inspection showed that a higher proportion of top-performing
students were in group F and conversely a higher proportion
of less able students were in group B. Because of this
greater attention has been paid by the administration to
randomization of the students into their examination groups.
Patient factors
Finally, qualitative feedback is collected from examiners on
aspects of the marking sheets and performance of the real and
simulated patients. Suggestions are sought for improvements
in the questions/stations, which are considered by the
examination committee before they are put into the bank.
Reporting results and decision-making
The main priority, of course, is to compile the results and
make decisions on who is to pass and who is to fail. Here,
the concern of the medical school, and the GMC, is whether
the students are competent to be provisionally registered
to practice as a PRHO. The university also has regula-
tory requirements necessitating the awarding of grades.
Mechanisms have to be found to satisfy both of these
demands, which can be conflicting. The former requires
a competency-based or criterion-referenced approach while
the latter requires a norm-referenced approach.
Assuring assessment quality of clinical competence
J M
The method Sheffield adopted utilizes the SEM calculated
from the examination performance indicators discussed
previously using the following formula:
p
SEM ¼ S:D:
ð1  rÞ
(where r is the reliability of the test and SD is the standard
deviation (%) of the scores).
If we enter the 2004 data as an example the
p
SEM ¼ 5.50
ð1  0:81Þ ¼ 2:40%. The SEM can now be
used as the equivalent of a confidence interval around the
decision point, which in this examination is the overall
borderline score, determined as described previously.
A system of grade bands has been agreed and is expressed
in a protocol used for decision-making (see Table 5). As this
is a high-stakes examination, the school’s assessment policy
has deemed that a pass will only be awarded to students who
have satisfied the examination board that they are not within
the borderline group (grade 2), as indicated by being at least
one SEM above the overall borderline score. This ensures a
68% confidence interval around the pass/fail decision-making
score. The examination board is actively considering raising
the bar for the borderline group to two SEMs to increase
the confidence interval to 95%. Students more than one
SEM below the overall borderline score are awarded a fail
(grade 1). The majority of students obtain a grade 4. Those
obtaining a mark of two SEMs from the top mark are graded
five and are eligible for a distinction. Students scoring
a grade 3 may benefit from the feedback that they have only
just passed and additional monitoring in the immediate
postgraduate period may be wise. Those scoring grade 2 or
less are invited for a repeat examination (described below).
It is incongruous to use a competence examination to
determine those students who are excellent and deserve a
higher academic grade or a prize. Previously these best
performing students were offered another examination in
the form of a distinction viva. This practice has now been
discontinued on the basis that such an examination is likely
to be less valid and considerably less reliable than the
main examination. However, it is beyond available resources
to create and administer a separate examination of similar
541
 C. Roberts et al.
quality that is able to discriminate reliably amongst such
excellent students.
A summary of the main statistical indicators over the
period 2001–04 is shown in Figure 1. It indicates the
improvement in reliability since the examination was
lengthened in 2003 and generally how stable the mean
scores, borderline scores and SEMs are with the differing
cohorts of students.
Repeat assessments
University regulations demand that all students deemed not
to have passed the examination be given a repeat assessment.
Accordingly, all borderline students and students who have
failed enter the repeat assessment examination, which is
conducted approximately six months later. This has been
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
controversial in that it is usual to provide students with a
much earlier opportunity for redemption. However, students
not shown to be clinically competent cannot reasonably be
expected to improve their performance without a significant
period of additional clinical experience and focused
remediation.
Numbers in the repeat assessment group have become
gratifyingly low (range 2–8). University regulations and
fairness to these students dictates that the repeat assessment
examination should be of the same format and quality as the
main examination. This makes it an expensive exercise for
such a small group, and provides some additional challenges
For personal use only.
when it comes to standard-setting and decision-making.
For efficiency purposes, the examination is blueprinted at
the same time as the main administration with questions
drawn from the pre-existing bank or written afresh. However,
the borderline method of standard-setting does not work for
such small numbers. Generalizability studies (Kramer et al.,
2003) indicate that a minimum of 30 students would need to
sit the exam to get a dependable pass/fail decision using this
approach. The most attractive solution to this dilemma will
arise when the bank of questions/stations, on which a
borderline standard has previously been calculated, is large
enough to construct a valid examination. As of 2004, a hybrid
approach has been adopted in which the historical standard,
i.e. the mean of the main 2003 and 2004 pass marks for the
main examination, is used as the pass mark for the repeat
assessment. This is justified on the basis that the repeat
assessment exam was blueprinted and produced at the same
time as the main examination. As a further check, a modified
Angoff procedure was used to predict the borderline score for
new questions/stations. For previously used items, the
borderline score was used from the item bank database.
Summary and conclusions
Developing a valid and reliable assessment of clinical
competence is not easy to achieve with the resources available
at the university level. Where such assessments are used for
high-stakes decision-making, as in the UK and many other
countries, the institution has a responsibility both to the
students and to the public to ensure that these assessments
meet best practice standards. Research has shown that this is
currently not always the case. The main reason for this may
be that the design and development of such assessments
require a considerable degree of educational and
542
psychometric expertise to ensure that students’ scores reflect
their true ability and that subsequent competence decisions
are reasonably free from error.
The examination we have developed progressively over
several years has followed published criteria for international
best practice, based on evidence derived largely from
generalizability studies, and from many years’ practical
experience in the methods used.
A number of quality-assurance procedures have been
described to demonstrate how the assessment is subject to
continual monitoring for potential sources of error, resulting
in the introduction of several refinements. We endorse
the process of blueprinting the assessment; providing
well-defined training regimes for examiners and simulated
patients; providing regular feedback to those involved
in judging student performances; applying appropriate
standard-setting procedures; attending to item development;
and providing detailed statistical analyses of all scores.
In conclusion, we have demonstrated that a high-quality
assessment of clinical competence can be conducted within
the resources of a university medical school. If the relevant
expertise is not available in-house we would strongly suggest
such support be sought on a consultancy basis and key staff
be supported to attain the appropriate skills. This will reduce
development time and costs, and ensure unnecessary
mistakes are not made.
Notes on contributors
CHRIS ROBERTS is Associate Professor of Medical Education and Director
of the Office of Teaching and Learning in Medicine at the University of
Sydney, Australia.
DAVID NEWBLE is Emeritus Professor of Medical Education at the
University of Sheffield. He returned to Australia in 2004 where he is a
professor of Medical Education at Flinders University and works as a
consultant in Medical Education.
BRIAN JOLLY is Director of the Centre for Medical and Health Sciences
Education, Faculty of Medicine, Nursing & Health Sciences, Monash
University, Australia.
MALCOLM REED is Professor of Surgery in the Department of Surgery in
the Sheffield Teaching Hospitals Trust. He is Chair of the Assessment
Committee responsible for the finals examination.
KINGSLEY HAMPTON is Senior Clinical Lecturer Division of Genomic
Medicine and Consultant Haematologist at the Sheffield Teaching
Hospitals Trust. He is the current Director of Studies responsible for
the finals examination.
References
BLIGH, J. (2003) Nothing is but what is not, Medical Education, 37,
pp. 184–185.
BOULET, J.R., MCKINLEY, D.W., WHELAN, G.P. & HAMBLETON, R.K.
(2003) Quality assurance models for performance-based assessments,
Advances in Health Science Education, 8, pp. 27–47.
BRISTOL ROYAL INQUIRY (2001) Learning from Bristol: The Report of the
Public Inquiry into Children’s Heart Surgery at the Bristol Royal Infirmary
1984–1995 (London, HMSO).
FOWELL, S.L., MAUDSLEY, G., MAGUIRE, P., LEINSTER, S.J. & BLIGH, J.
(1998) Student assessment in undergraduate medical education in
the United Kingdom, Medical Education, 34, pp. 1–49.
FOWELL, S.L. & BLIGH, J. (2001) Assessment of undergraduate medical
education in the UK: time to ditch motherhood and apple pie,
Medical Education, 35, pp. 1006–1007.

GENERAL MEDICAL COUNCIL (1993) Tomorrow’s Doctors: Recommendations
on Undergraduate Medical Education (London, GMC).
GENERAL MEDICAL COUNCIL (2003) Tomorrow’s Doctors: Recommendations
on Undergraduate Medical Education (London, GMC). In: NEWBLE, D.I.,
JOLLY, B. & WAKEFORD, R. (Eds) (1994) The Certification and
Recertification of Doctors: Issues in the Assessment of Clinical Competence
(Cambridge, Cambridge University Press).
KRAMER, A., MUIJTJENS, A., JANSEN, K., DÜSMAN, H., TAN, L. & VAN DER
VLEUTEN, C.P. (2003) Comparison of a rational and an empirical
standard setting procedure for an OSCE, Medical Education, 37,
pp. 132–139.
NEWBLE, D.I. & SWANSON, D.B. (1988) Psychometric characteristics
of the objective structured clinical examination, Medical Education, 22,
pp. 325–334.
NEWBLE, D.I. (2001) (Letter to the editor), Medical Education, 35,
pp. 308–309.
NEWBLE, D.I. (2002) Assessing Clinical Competence at the Undergraduate
Level, Medical Education Booklet, No. 25 (Edinburgh, Association for
Med Teach Downloaded from informahealthcare.com by QUT Queensland University of Tech on 10/31/14
the Study of Medical Education).
NEWBLE, D.I. (2004) Techniques for measuring clinical competence:
objective structured clinical examinations, Medical Education, 35,
pp. 199–203.
NEWBLE, D.I., DAUPHINEE, D., DAWSON-SAUNDERS, B., et al. (1994)
Guidelines for the development of effective and efficient procedures
for the assessment of clinical competence, Teaching and Learning in
Medicine, 6, pp. 213–220.
NEWBLE, D.I., STARK, P., LAWSON, M. & BAX, N.D. (2005) Developing
an outcome-focused core curriculum: the Sheffield approach, Medical
Education (in press).
NORCINI, J.J. (2003) Setting standards on educational tests, Medical
Education, 37, pp. 464–469.
PATEL, K. (2001) USMLE-style exam (letter), Medical Education, 35,
For personal use only.
pp. 306–307.
REZNICK, R.K., BLACKMORE, D., DAUPHINEE, W.D., ROTHMAN, A.I. &
SMEE, S.M. (1996) Large scale high stakes testing with an OSCE:
report from the Medical Council of Canada, Academic Medicine, 71,
pp. S19–S21.
ROBERTS, C., LAWSON, M., NEWBLE, D.I. & SELF, A. (2003) Managing
the learning environment in undergraduate medical education: the
Sheffield Approach, Medical Teacher, 5, pp. 297–301.
Assuring assessment quality of clinical competence
SHIPMAN INQUIRY (2005) Independent inquiry into the issues arising
from the case of Harold Fredrick Shipman. Available at: http://
www.shipmaninquiry.org.uk/ (accessed February 2005).
SMEE, S.M. (2001) Setting standards for objective structured clinical
examination: the borderline group method gains grounds on Angoff,
Medical Education, 35, pp. 1009–1010.
SOUTHGATE, L., CAMPBELL, L., COX, J., FOULKES, J., JOLLY, B.,
MCCRORIE, P. & TOMBLESON, P. (2001) The General Medical
Council’s performance procedures: the development and implemen-
tation of tests of competence with examples from general practice,
Medical Education, 35, pp. 20–28.
STREINER, D.L. & NORMAN, G.R. (2003) Health Measurement Scales: A
Practical Guide to their Development and Use (Oxford, Oxford University
Press).
SWANSON, D.B., NORMAN, G.R. & LINN, R.L. (1995) Performance based
assessment: lessons learnt from the health professions, Educational
Researcher, 24, pp. 5–11.
TAMBLYN, R.M., ABRAHAMOWICZ, M., BRAILOVSKY, C. &
GRAND’MAISON, P. (1998) Association between licensing
examination scores and resource use and quality of care in primary
care practice, Journal of the American Medical Association, 280,
pp. 989–996.
TOMBLESON, P., FOX, R.A. & DACRE, J.A. (2000) Defining the content
for the objective structured clinical examination component of the
Professional and Linguistic Assessment Board examination: develop-
ment of a blueprint, Medical Education, 34, pp. 566–572.
VAN DER VLEUTEN, C.P. (1996) The assessment of professional
competence: developments, research and practical implications,
Advances in Health Science Education, 1, pp. 41–67.
VAN DER VLEUTEN, C.P. (2000) Validity of final examinations in under-
graduate medical training, British Medical Journal, 321, pp. 1217–1219.
WAKEFORD, R. (2001) (Author’s reply), British Medical Journal, 322,
p. 359.
WASS, V., MCGIBBON, D. & VAN DER VLEUTEN, C.P. (2001) Composite
undergraduate clinical examinations: how should the components be
combined to maximize reliability, Medical Education, 35, p. 330.
WILKINSON, T.J., NEWBLE, D.I. & FRAMPTON, C.M. (2001) Standard
setting in an objective structured clinical examination: use of global
ratings of borderline performance to determine the passing score,
Medical Education, 35, pp. 1043–1049.
543