Nephrology Knowledge Based Assessment

(Specialist Examination, MRCPUK(Nephrology))

Jonathan Fox
March 2008

Specialist Examinations: purpose

To ensure that certified specialists have
sufficient knowledge to practise competently and
safely as consultants
To complement workplace-based assessments
To work with specialist societies and SACs
provide a rigorous national assessment to
establish public confidence
To offer a challenge similar to subspeciality
certification exams in North America

Specialist Examinations: delivery

1 diet per year (was to be 2 per year)

Computer-based testing (www.pearsonview.com)
2 papers in each diet
100 best-of-five questions in each paper
Assess core knowledge and application of this
knowledge in a clinical setting
Strict distribution of questions in accordance with
blueprint derived from specialty curriculum
To be taken in ST4 usually and required for CCT

Specialties
First wave

Second wave

Dermatology
Gastroenterology
Geriatric Medicine
Neurology

First diet June 2008

Acute medicine
Cardiology
CPT
Endocrinology/diabetes
ID
Medical oncology
Renal medicine
Respiratory medicine
Rheumatology

First diet November 2008

Timeline: medical specialties

2004-6: JCHMT Pilot (report Oct 2006)
late 2006: MRCP(UK)/Federation of Medical Royal
Colleges proposed partnership with specialist societies
early 2007: Appointment of Examination Board Chairs &
Secretaries
June 2008: First wave examinations
November 2008: Second wave examinations

Timeline: Nephrology
March 2007 Appointment of Chair (J Fox) & Secretary
(J Levy)
March 2007 advertisement for QWG members
(41 responses, 5 not requiring training)
18 July 2007: first training day 24 attended
5 Sept 2007: second training day 11 attended
8-9 Jan 2008: Question Writing Group meeting 26
attended, approx. 333 questions produced
3-4 June 2008: Board meeting
Aug/Sept 2008: Standard Setting Group meeting
November 2008: KBA

Question Writing Groups

2-day meetings twice a year initially

Consultants
Wide geographical spread
Some from MRCP SQG
Attended workshop
30 questions per year per member
Guidance from Secretary/Chair on topics
Stand down if fail to fulfil commitment

A 34-year-old woman was referred for the investigation of bloody diarrhoea. She was opening
her bowels four times daily. On examination, she was well. Her pulse was 64 beats per minute
and her abdomen was soft and non-tender. Colonoscopy revealed an active colitis, limited to
the sigmoid colon. Biopsies confirmed a diagnosis of ulcerative colitis.
Investigations:
haemoglobin
132 g/L (115 - 165)
white cell count
11.5 x 109/L (4 - 11)
platelet count
323 x 109/L (150 - 400)
serum albumin
40 g/L (37 - 49)
serum C-reactive protein
13 mg/L (<10)
What is the most appropriate initial treatment?
A mesalazine enemas
B oral azathioprine
C oral mesalazine
D oral modified-release budesonide
E oral prednisolone
Answer Key: A

The Challenge of Question-Writing

200 questions per year
Curriculum coverage
Each question should not be re-used more often
than once every 3 years
Question bank should contain at least 1000
usable questions
To generate one re-usable question for MRCP(UK)
requires 3-5 questions to be drafted

Question production process

QWG members
Non-medical editors
QWG secretaries
QWG meeting
Final vetting by chairman/secretary
Question Bank
Examination Board
Examination

Standard Setting Group

Board
Roles:
To set papers for each examination
To oversee delivery of examinations
To be responsible for academic matters, misconduct,
complaints, regulatory matters
Composition:
10 members incl. chairman & secretary
2 represent SAC
4 from the Question Writing Group
4 non-writing members
Meets for 2 days a year

Standard Setting Group

Role:
To set pass mark for the exam (modified Angoff method,
Hofstee compromise applied after exam)
To develop assessment strategy
Membership:
6 members incl. Board Chair & Secretary
No-one else should belong to both QWG & Board
Chair should have experience of standard setting
Meets for 2 days a year

Why collaborate with MRCP(UK)?

Common approach for 13 medical specialties
Format used since 2002:
Part 1: 3 diets of 2 papers (200 Qs) per year
Part 2: 3 diets of 3 papers (~270 Qs) per year
Medical experience: SQG, Board, Standard Setting
Statistical & psychometric support
Administration: non-medical editors, organisation of
meetings, etc
IT: question bank & CBT

Reliability (Cronbachs alpha):

MRCP(UK) Part 2 Written Examination
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
2002/1 2002/2 2002/3 2003/1 2003/2 2003/3 2004/1 2004/2 2004/3 2005/1 2005/2 2005/3 2006/1 2006/2

Pass rate
Proposed:
>85% per diet
~98% overall
Desired pass mark?

Challenges
Heavy workload for a relatively small specialty
Small number of candidates (cf 1245 candidates
for MRCP Part 2, 2007/2) will make pass mark
setting, reliability assessment and pre-testing of
questions difficult
Name (MRCPUK(Nephrology))
Cost to candidates/affordability to RA

Links

http://www.jrcptb.org.uk/assessment/Pages/MRCP(UK)KnowledgeB
asedAssessment.aspx

http://www.jrcptb.org.uk/SiteCollectionDocuments/KBA%20Project
%20Final%20Report.pdf

www.pmetb.org.uk/fileadmin/user/QA/Assessment/Assessment_go
od_practice_v0207.pdf

http://www.mrcpuk.org/Pages/Home.aspx