Nephrology Knowledge Based Assessment

(Specialist Examination, MRCPUK(Nephrology))

Jonathan Fox March 2008

Specialist Examinations: purpose

To ensure that certified specialists have sufficient knowledge to practise competently and safely as consultants To complement workplace-based assessments To work with specialist societies and SACs provide a rigorous national assessment to establish public confidence To offer a challenge similar to subspeciality certification exams in North America

Specialist Examinations: delivery

1 diet per year (was to be 2 per year) Computer-based testing (www.pearsonview.com) 2 papers in each diet 100 best-of-five questions in each paper Assess core knowledge and application of this knowledge in a clinical setting Strict distribution of questions in accordance with blueprint derived from specialty curriculum To be taken in ST4 usually and required for CCT

Specialties
First wave Dermatology Gastroenterology Geriatric Medicine Neurology Second wave Acute medicine Cardiology CPT Endocrinology/diabetes ID Medical oncology Renal medicine Respiratory medicine Rheumatology

First diet June 2008

First diet November 2008

Timeline: medical specialties

2004-6: JCHMT Pilot (report Oct 2006)

late 2006: MRCP(UK)/Federation of Medical Royal Colleges proposed partnership with specialist societies
early 2007: Appointment of Examination Board Chairs & Secretaries June 2008: First wave examinations November 2008: Second wave examinations

Timeline: Nephrology
March 2007 Appointment of Chair (J Fox) & Secretary (J Levy) March 2007 advertisement for QWG members (41 responses, 5 not requiring training) 18 July 2007: first training day 24 attended 5 Sept 2007: second training day 11 attended 8-9 Jan 2008: Question Writing Group meeting 26 attended, approx. 333 questions produced 3-4 June 2008: Board meeting Aug/Sept 2008: Standard Setting Group meeting November 2008: KBA

Question Writing Groups

2-day meetings twice a year initially Consultants Wide geographical spread Some from MRCP SQG Attended workshop 30 questions per year per member Guidance from Secretary/Chair on topics Stand down if fail to fulfil commitment

A 34-year-old woman was referred for the investigation of bloody diarrhoea. She was opening her bowels four times daily. On examination, she was well. Her pulse was 64 beats per minute and her abdomen was soft and non-tender. Colonoscopy revealed an active colitis, limited to the sigmoid colon. Biopsies confirmed a diagnosis of ulcerative colitis. Investigations:

haemoglobin white cell count platelet count serum albumin serum C-reactive protein

132 g/L (115 - 165) 11.5 x 109/L (4 - 11) 323 x 109/L (150 - 400) 40 g/L (37 - 49) 13 mg/L (<10)

What is the most appropriate initial treatment? A mesalazine enemas B oral azathioprine C oral mesalazine D oral modified-release budesonide E oral prednisolone Answer Key: A

The Challenge of Question-Writing

200 questions per year Curriculum coverage Each question should not be re-used more often than once every 3 years Question bank should contain at least 1000 usable questions

To generate one re-usable question for MRCP(UK) requires 3-5 questions to be drafted

Question production process

QWG members Non-medical editors QWG secretaries QWG meeting Final vetting by chairman/secretary Question Bank

Examination Board
Examination

Standard Setting Group

Board
Roles: To set papers for each examination To oversee delivery of examinations To be responsible for academic matters, misconduct, complaints, regulatory matters Composition: 10 members incl. chairman & secretary 2 represent SAC 4 from the Question Writing Group 4 non-writing members Meets for 2 days a year

Standard Setting Group

Role: To set pass mark for the exam (modified Angoff method, Hofstee compromise applied after exam) To develop assessment strategy

Membership: 6 members incl. Board Chair & Secretary No-one else should belong to both QWG & Board Chair should have experience of standard setting
Meets for 2 days a year

Why collaborate with MRCP(UK)?

Common approach for 13 medical specialties

Format used since 2002: Part 1: 3 diets of 2 papers (200 Qs) per year Part 2: 3 diets of 3 papers (~270 Qs) per year Medical experience: SQG, Board, Standard Setting
Statistical & psychometric support Administration: non-medical editors, organisation of meetings, etc IT: question bank & CBT

Reliability (Cronbachs alpha): MRCP(UK) Part 2 Written Examination

0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 2002/1 2002/2 2002/3 2003/1 2003/2 2003/3 2004/1 2004/2 2004/3 2005/1 2005/2 2005/3 2006/1 2006/2

Pass rate
Proposed:

>85% per diet ~98% overall

Desired pass mark?

Challenges
Heavy workload for a relatively small specialty
Small number of candidates (cf 1245 candidates for MRCP Part 2, 2007/2) will make pass mark setting, reliability assessment and pre-testing of questions difficult Name (MRCPUK(Nephrology)) Cost to candidates/affordability to RA

Links
http://www.jrcptb.org.uk/assessment/Pages/MRCP(UK)KnowledgeB asedAssessment.aspx http://www.jrcptb.org.uk/SiteCollectionDocuments/KBA%20Project% 20Final%20Report.pdf www.pmetb.org.uk/fileadmin/user/QA/Assessment/Assessment_go od_practice_v0207.pdf http://www.mrcpuk.org/Pages/Home.aspx