Professional Documents
Culture Documents
CONTEXT Ber’s Comprehensive Integrative Puzzle (one-way ANOVA F = 104.00, P < 0.0001). Students of
aims to assess analytical clinical thinking in medical both universities had comparable scores. MATCH 1
students. We developed a paediatric version, the and 2 scores were comparable. Cronbach’s a-values in
MATCH test, in which we added two irrelevant MATCH 1 and 2 were 0.92 and 0.91, respectively, for
options to each question in order to reduce guessing all subjects, and 0.82 and 0.87, respectively, for all
behaviour. We tested its construct validity and studied students.
the development of integrative skills over time.
CONCLUSIONS Analytical clinical thinking develops
METHODS We administered a test (MATCH 1) to over time, independently of the factual content of the
subjects from two universities, both with a 6-year course. This implies that shortened medical training
medical training course. Subjects included 30 stu- programmes could produce less skilled graduates.
dents from university 1 who had completed a paedi-
atric clerkship in Year 4, 23 students from university 2 KEYWORDS multicentre study [publication type]; clinical
who had completed a paediatric clerkship in Year 5, clerkship ⁄ *standards; paediatrics ⁄ *education; clinical compe-
tence ⁄ *standards; humans; students, medical ⁄ *psychology;
13 students from both universities who had com- *thinking.
pleted an advanced paediatric clerkship in Year 6, 28
paediatric residents and 17 paediatricians. We Medical Education 2008: 42: 1037–1043
repeated this procedure using a second test with doi:10.1111/j.1365-2923.2008.03152.x
different domains in a new, comparable group of
subjects (MATCH 2).
INTRODUCTION
RESULTS Mean MATCH 1 scores for the respective
groups were: Year 4 students: 61.2% (standard devi- A number of test formats have been designed to
ation [SD] 1.3); Year 5 students: 71.3% (SD 1.6); Year capture clinical thinking or reasoning skills. One
6 students: 76.2% (SD 1.5); paediatric residents: approach is the use of short-answer items with
88.5% (SD 0.7), and paediatricians: 92.2% (SD 1.1) reasonable psychometric properties.1 The drawback
is that these questions need marking and their
application in large groups of candidates is burden-
1
Department of Paediatrics, Emma Children’s Hospital, Academic some. A compromise between multiple-choice and
Medical Centre, Amsterdam, The Netherlands constructed response questions is found in items with
2
Department of Paediatrics, Wilhelmina Children’s Hospital,
University Medical Centre Utrecht, Utrecht, The Netherlands
a relatively long list of options.2,3 This solution
3
Department of Paediatrics, Zaans Medical Centre, Zaandam, The minimises the effects of cueing. However, construct-
Netherlands ing long lists of options is not easy if it must be done
4
Department of Paediatrics, Flevo Hospital, Almere, The for many items. Another compromise involves
Netherlands
5
Department of Paediatrics, Onze Lieve Vrouwen Gasthuis (OVG), matching type items. A list of options may serve
Amsterdam, The Netherlands several questions in a test by inviting candidates to
6
Centre for Research and Development of Education, University match options with item stems. A well-known format
Medical Centre Utrecht, Utrecht, The Netherlands is the extended matching items test,4,5 which has
Correspondence: Jaap W Groothoff MD, PhD, Emma Children’s been reported to show superior psychometric prop-
Hospital AMC, University of Amsterdam, PO Box 22700, 1100 DE erties.6 The main disadvantage of this approach is
Amsterdam, The Netherlands. Tel: 00 31 20 566 7919; that it requires candidates to sift through a long list of
Fax: 00 31 20 691 7735; E-mail: j.w.groothoff@amc.uva.nl
ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043 1037
J W Groothoff et al
What is already known on this subject The aim of our study was to investigate the construct
validity of the MATCH test. Ber advocated that the CIP
Few tools exist that are easily applied to assess
should be used along the full continuum of medical
analytical clinical thinking in medical students.
training.8 We therefore hypothesised that test scores
In addition, information on the influence of the
length of exposure to medical education on the should correlate with stage of training. If we should
development of integrative skills is lacking. also find a favourable reliability, related to limited
testing time, the MATCH test might well be an
What this study adds efficient tool with widespread application possibilities.
1038 ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043
Analytical thinking assessment takes time
they have different curricula. The medical pro- residents and paediatricians according to stage of
gramme at the University of Utrecht (university 1) training. Cronbach’s a was calculated to establish the
introduces clinical training early in the curriculum. test reliability for both tests. Differences between total
Students undertake their first clerkships in Year 3 and scores, as well as between the corrected separate
take a paediatric clerkship in Year 4, immediately scores for all four blocks in both tests for all groups
after the theoretical paediatric training. During Year were analysed by one-way ANOVA. SPSS Version 12
6, students may choose an advanced clerkship in (SPSS, Inc., Chicago, IL, USA) was used for statistical
paediatrics. analysis.
ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043 1039
J W Groothoff et al
Table 1 Total scores and reliability analysis of all students and doctors
All subjects 119 76.8, 48–99, 13.1 0.92 98 74.0, 40–99, 12.6 0.91
Students 72 68.4, 48–92, 9.5 0.82 81 70.8, 40–95, 11.1 0.87
Doctors 47 89.6, 79–99, 4.6 0.65 17 89.3, 71–99, 6.2 0.79
SD = standard deviation
blocks of questions for all groups of subjects were the doctors (both P < 0.0001). In both tests, mean
significantly different, each time in favour of the scores of Year 4 students of University 1 were
more experienced group (Table 2, Fig. S2).The significantly lower than those of Year 5 students of
corrected mean differences in scores for blocks 1–4 University 2, whereas scores of Year 6 students of
between students and, by contrast, paediatricians both universities were comparable (Table 2). In
and residents, recalculated on a 100-point scale, MATCH 2, mean testing time was 49.7 minutes
were 21.2 (SD 2.2), 25.0 (SD 1.8), 21.8 (SD 1.7) (range 22–90 minutes, SD 14.1) for students and
and 21.8 (SD 2.3), for MATCH 1, respectively, and 34.4 minutes (range 30–40 minutes, SD 3.1) for
17.5 (SD 3.0), 19.6 (SD 2.4), 18.0 (SD 2.2) and 20.9 residents and paediatricians; 83.3% of all students
(SD 2.6) for MATCH 2, each time all in favour of completed the test within 60 minutes.
Table 2 Total scores and separate scores for all four blocks of questions in MATCH 1 and MATCH 2 for all groups of subjects
MATCH 1 MATCH 2
Total
Q1* Q2* Q3* Q4* score Q1* Q2* Q3* Q4* Total score
Year 4 53.9 64.4 65.7 (10.5) 60.7 (14.4) 61.2 (7.2) 66.4 (14.0) 55.3 (16.0) 59.5 (17.0) 67.5 (13.4) 62.2 (11.8)
students (10.4) (12.0)
Year 5 67.0 73.9 74.2 (10.8) 70.1 (13.3) 71.3 (7.7) 80.1 (9.4) 67.9 (11.0) 71.5 (14.6) 70.4 (13.4) 72.4 (8.0)
studentsà (11.1) (12.8)
Year 6 73.3 77.7 73.3 (10.3) 78.3 (15.7) 75.7 (5.3) 83.6 (11.7) 74.4 (13.0) 79.6 (9.9) 84.4 (12.0) 80.5 (9.6)
§
students (8.2) (9.7)
Residents– 82.4 94.7 90.3 (5.9) 86.6 (8.8) 88.5 (3.8) 93.6 (2.0) 83.7 (7.6) 86.9 (5.5) 91.5 (10.2) 88.9 (6.1)
(9.4) (5.6)
Paediatricians– 84.9 96.0 94.8 (5.1) 93.2 (6.7) 92.2 (4.7) 99.0 (2.0) 90.0 (9.5) 88.0 (9.8) 94.0 (4.0) 92.3 (5.2)
(9.1) (4.7)
F-value one-way 41.0 49.1 44.3 27.4 104.5 19.0 16.4 11.0 13.6 27.6
ANOVA**
* Scores on questions 1–4 multiplied by 4; Year 4)6 students, paediatric residents and paediatricians
Medical students at the end of the Year 4 paediatric clerkship at UMC Utrecht
à
Medical students at the end of the Year 5 paediatric clerkship at the University of Amsterdam
§
Medical students at the end of the Year 6 advanced paediatric clerkship at both universities
–
Paediatric residents and paediatricians at both universities
** All P < 0.0001
1040 ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043
Analytical thinking assessment takes time
ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043 1041
J W Groothoff et al
with which to assess clinical reasoning skills in In: Scherpbier AJJA, van der Vleuten CPM, Rethans JJ,
medical students. Scores on the MATCH test appear van der Steeg AFW, eds. Advances in Medical Education.
to depend upon length of clinical experience, rather Dordrecht: Kluwer Academic Publishers 1997;84–6.
than on theoretical examinations in the field. This 8 Ber R. The CIP (Comprehensive Integrative Puzzle)
assessment method. Med Teach 2003;25 (2):171–6.
hypothesis should be investigated in greater depth in
9 Wilson RB, Case SM. Extended matching questions: an
the future.
alternative to multiple-choice or free-response
questions. J Vet Med Educ 1993;20 http://scholar.
lib.vt.edu/ejournals/JVME/V20-3/wilson.html.
Contributors: JWG was the primary investigator of the study, [Accessed 10 November 2007.]
initiated and participated in the construction of the tests, 10 van der Vleuten CP, Schuwirth LW. Assessing profes-
and wrote the paper. JF, GAMT, WBV and DKB participated sional competence: from methods to programmes.
in the construction of the tests and the conducting of the Med Educ 2005;39 (3):309–17.
study. OThJtC acted as senior educational supervisor of the 11 Elstein AS, Shulman LS, Sprafka SA. Medical Problem
project. All authors contributed to the revision of the Solving: an Analysis of Clinical Reasoning. Cambridge,
manuscript and approved the final version for publication. MA: Harvard University Press 1978;278–9.
Acknowledgements: we thank all the individuals who 12 Custers EJ, Boshuizen HP, Schmidt HG. The influence
participated in the study. of medical expertise, case typicality, and illness script
Funding: none. component on case processing and disease probability
Conflicts of interest: none. estimates. Mem Cognit 1996;24 (3):384–99.
Ethical approval: the Ethical Committee of the Academic 13 Charlin B, Roy L, Brailovsky C, Goulet F, van der Vle-
Medical Centre has declared that no approval by their uten C. The Script Concordance test: a tool to assess
committee was needed to conduct this study. the reflective clinician. Teach Learn Med 2000;12
(4):189–95.
REFERENCES
SUPPORTING INFORMATION
1 Rademakers J, ten Cate TJ, Bär PR. Progress testing
with short-answer questions. Med Teach 2005;27 Additional supporting information may be found in the
(7):578–82. online version of this article.
2 Schuwirth LWT, van der Vleuten CPM, Stoffers HEJH,
Peperkamp AGW. Computerised long-menu questions Figure S1. Mean scores of students versus paediatricians
as an alternative to open-ended questions in and residents of MATCH 1 (1a) and MATCH 2 (1b).
computerised assessment. Med Educ 1996;30:50–5.
3 Rotthoff T, Baehring T, Dicken HD, Fahron U, Richter Figure S2. Scores of MATCH 1 (2a) and of MATCH 2 (2b)
B, Fischer MR, Scherbaum WA. Comparison between of students according time exposure to medical training in
long-menu and open-ended questions in computerised comparison with residents and pediatricians. 4th year ¼ -
medical assessments. A randomised controlled trial. clerkship students University 1 (UU); 5th year ¼ clerkship
BMC Med Educ 2006;6:50. students university 2 (UvA); 6th year ¼ advanced clerkship
4 Case SM, Swanson DB. Extended-matching items: a students both universities; residents ¼ residents all centres;
practical alternative to free-response questions. Teach pediatricians ¼ paediatricians all centres (see text).
Learn Med 1993;5 (2):107–15.
5 Case SM, Swanson DB. Constructing Written Test Ques- Please note: Wiley-Blackwell are not responsible for the
tions for the Basic and Clinical Sciences . National Board of content or functionality of any supporting materials
Medical Examiners 2002. http://www.nbme.org. supplied by the authors. Any queries (other than missing
[Accessed 20 October 2007.] material) should be directed to the corresponding author
6 Fenderson BA, Damjanov I, Robeson MR, Veloski JJ, for the article.
Rubin E. The virtues of extended matching and
uncued tests as alternatives to multiple-choice
questions. Hum Pathol 1997;28:526–32. Received 16 November 2007; editorial comments to authors
7 February 2008; accepted for publication 17 March 2008
7 Ber R. Design of an integrative course and assessment
method: the CIP (Comprehensive Integrative Puzzle).
1042 ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043
Analytical thinking assessment takes time
APPENDIX
Cystitis
Chronic renal failure
Nephrotic syndrome
Acute glomerulonephritis
Vesical ureteral reflux + pyelonephritis
1 History
a A 10-year-old boy has a headache for 1 day and brown-coloured urine; 2 weeks previously he had a sore throat
b A 15-year-old girl has severe colic pain and red-coloured urine
c A 15-year-old girl has pain in the belly and dysuria; she has never experienced these complaints before
d A 6-year-old girl has swollen eyes and a swollen belly; the previous week she had rhinitis and a sore throat
e A 5-year-old girl feels very ill, with back pain and dysuria; she also ‘feels warm’
f A 6-month-old boy with failure to thrive is vomiting frequently and is polyuric
g A 16-year-old girl has progressive polyuria and thirst of 3 weeks’ duration
2 Physical examination
a Strikingly pale child, slight tachycardia, normal blood pressure, height < ) 2 standard deviation
b No abnormalities
c Oedema in the eyelids, blood pressure 180 ⁄ 105, enlarged lymph nodes in the neck
d Dystrophic child, rapid pulse, thirsty
e Oedema in the legs, ascites and eyelids, blood pressure 90 ⁄ 60
f Temperature 38.0 C, lower belly pain
g Temperature 39.5 C, pain in the right flank
3 Laboratory tests
a Cloudy urine, leukocyturia, normal serum creatinine
b Urine protein stick strongly positive, normal serum creatinine
c Urine protein stick positive, sediment: a few leukocytes and erytrocytes, very low Hb, very high serum creatinine
d Very low serum potassium, low serum phosphate, high fractional Na-excretion
e Haematuria, normal Hb, high serum creatinine
f High fractional phosphate excretion, high fractional salt excretion, glucosuria
g Low fractional phosphate excretion, glucosuria
4 Imaging results
a Ultrasound belly and later voiding cystogram and Di Mercapto Succinic Acid (DMSA) scan
b Ultrasound: anatomy, kidneys?
c Computed tomography, kidneys
d X-thorax: overhydration?
e Not indicated; if necessary X-thorax to establish hydration situation
f Not indicated
g Magnetic resonance imaging, kidneys
5 Therapy
a Broad-spectrum antibiotics
b Salt and fluid restriction, anti-hypertensive drugs
c Calorie-enriched feeding, extra water and salt supply, 1-HO vitamin D
d Corticosteroids, salt restriction
e Co-trimoxazole, amoxicillin of nitrofurantoin
f Hydration, salt, phosphate, magnesium and other electrolyte supply
g Corticosteroids and cyclophosphamide i.v.
ª Blackwell Publishing Ltd 2008. MEDICAL EDUCATION 2008; 42: 1037–1043 1043