You are on page 1of 10


Defining and Assessing

Professional Competence
Ronald M. Epstein, MD Context Current assessment formats for physicians and trainees reliably test core knowl-
Edward M. Hundert, MD edge and basic skills. However, they may underemphasize some important domains
of professional medical practice, including interpersonal skills, lifelong learning, pro-

EDICAL SCHOOLS, POST- fessionalism, and integration of core knowledge into clinical practice.
graduate training pro- Objectives To propose a definition of professional competence, to review current
grams, and licensing bod- means for assessing it, and to suggest new approaches to assessment.
ies conduct assessments Data Sources We searched the MEDLINE database from 1966 to 2001 and refer-
to certify the competence of future prac- ence lists of relevant articles for English-language studies of reliability or validity of
titioners, discriminate among candi- measures of competence of physicians, medical students, and residents.
dates for advanced training, provide Study Selection We excluded articles of a purely descriptive nature, duplicate re-
motivation and direction for learning, ports, reviews, and opinions and position statements, which yielded 195 relevant ci-
and judge the adequacy of training tations.
programs. Standards for professional Data Extraction Data were abstracted by 1 of us (R.M.E.). Quality criteria for in-
competence delineate key technical, clusion were broad, given the heterogeneity of interventions, complexity of outcome
cognitive, and emotional aspects of measures, and paucity of randomized or longitudinal study designs.
practice, including those that may not Data Synthesis We generated an inclusive definition of competence: the habitual
be measurable.1,2 However, there is no and judicious use of communication, knowledge, technical skills, clinical reasoning, emo-
agreed-upon definition of competence tions, values, and reflection in daily practice for the benefit of the individual and the
that encompasses all important domains community being served. Aside from protecting the public and limiting access to ad-
of professional medical practice. In vanced training, assessments should foster habits of learning and self-reflection and
response, the Accreditation Council for drive institutional change. Subjective, multiple-choice, and standardized patient as-
Graduate Medical Education defined 6 sessments, although reliable, underemphasize important domains of professional com-
petence: integration of knowledge and skills, context of care, information manage-
areas of competence and some means
ment, teamwork, health systems, and patient-physician relationships. Few assessments
of assessing them3: patient care (includ- observe trainees in real-life situations, incorporate the perspectives of peers and pa-
ing clinical reasoning), medical knowl- tients, or use measures that predict clinical outcomes.
edge, practice-based learning and
Conclusions In addition to assessments of basic skills, new formats that assess clini-
improvement (including information cal reasoning, expert judgment, management of ambiguity, professionalism, time man-
management), interpersonal and agement, learning strategies, and teamwork promise a multidimensional assessment
communication skills, professional- while maintaining adequate reliability and validity. Institutional support, reflection, and
ism, and systems-based practice (in- mentoring must accompany the development of assessment programs.
cluding health economics and team- JAMA. 2002;287:226-235
In this article, we will advance a defi- DEFINING PROFESSIONAL tific knowledge, and moral develop-
nition of professional competence of COMPETENCE ment. It includes a cognitive function
physicians and trainees that expands on Building on prior definitions,1-3 we pro- acquiring and using knowledge to solve
these 6 areas, perform an evidence- pose that professional competence is the
based critique of current methods of as- habitual and judicious use of communi- Author Affiliations: Departments of Family Medi-
cine (Dr Epstein), Psychiatry (Drs Epstein and Hun-
sessing these areas of competence, and cation, knowledge, technical skills, clini- dert), and Medical Humanities (Dr Hundert), Univer-
propose new means for assessing resi- cal reasoning, emotions, values, and re- sity of Rochester School of Medicine and Dentistry,
Rochester, NY.
dents and medical students. flection in daily practice for the benefit Corresponding Author and Reprints: Ronald M. Ep-
of the individual and community being stein, MD, University of Rochester School of Medi-
cine and Dentistry, 885 South Ave, Rochester, NY
served. Competence builds on a foun- 14620 (e-mail: ronald_epstein@urmc.rochester
For editorial comment see p 243.
dation of basic clinical skills, scien- .edu).

226 JAMA, January 9, 2002Vol 287, No. 2 (Reprinted) 2002 American Medical Association. All rights reserved.

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


real-life problems; an integrative func-

tionusing biomedical and psycho- Box 1. Dimensions of Professional Competence
social data in clinical reasoning; a re- Cognitive
lational functioncommunicating
Core knowledge
effectively with patients and col- Basic communication skills
leagues; and an affective/moral func- Information management
tionthe willingness, patience, and Applying knowledge to real-world situations
emotional awareness to use these skills Using tacit knowledge and personal experience
judiciously and humanely (BOX 1). Abstract problem-solving
Competence depends on habits of Self-directed acquisition of new knowledge
mind, including attentiveness, critical Recognizing gaps in knowledge
curiosity, self-awareness, and pres- Generating questions
ence. Professional competence is de- Using resources (eg, published evidence, colleagues)
Learning from experience
velopmental, impermanent, and con-
text-dependent. Technical
Physical examination skills
Acquisition and Use of Knowledge Surgical/procedural skills
Evidence-based medicine is an ex- Integrative
plicit means for generating an impor- Incorporating scientific, clinical, and humanistic judgment
tant answerable question, interpreting Using clinical reasoning strategies appropriately (hypothetico-deductive,
new knowledge, and judging how to ap- pattern-recognition, elaborated knowledge)
ply that knowledge in a clinical set- Linking basic and clinical knowledge across disciplines
ting.4 But Polanyi5 argues that compe- Managing uncertainty
tence is defined by tacit rather than Context
explicit knowledge. Tacit knowledge is Clinical setting
that which we know but normally do Use of time
not explain easily, including the in-
formed use of heuristics (rules of
Communication skills
thumb), intuition, and pattern recog- Handling conflict
nition. The assessment of evidence- Teamwork
based medicine skills is difficult be- Teaching others (eg, patients, students, and colleagues)
cause many of the heuristics used by
novices are replaced by shortcuts in the
Tolerance of ambiguity and anxiety
hands of experts,6 as are other clinical
Emotional intelligence
skills.7 Respect for patients
Personal knowledge is usable knowl- Responsiveness to patients and society
edge gained through experience.8 Cli- Caring
nicians use personal knowledge when
Habits of Mind
they observe a patients demeanor (such
Observations of ones own thinking, emotions, and techniques
as a facial expression) and arrive at a
provisional diagnosis (such as Parkin- Critical curiosity
son disease) before eliciting the spe- Recognition of and response to cognitive and emotional biases
cific information to confirm it. Be- Willingness to acknowledge and correct errors
cause experience does not necessarily
lead to learning and competence,9 cog-
nitive and emotional self-awareness is
necessary to help physicians question, them in isolation.11 For example, the possesses the integrative ability to think,
seek new information, and adjust for student who can elicit historical data feel, and act like a physician. 6,12-15
their own biases. and physical findings, who can suture Schon16 argues that professional com-
well, who knows the anatomy of the petence is more than factual knowl-
Integrative Aspects of Care gallbladder and the bile ducts, and who edge and the ability to solve problems
Professional competence is more than can draw the biosynthetic pathway of with clear-cut solutions: it is defined by
a demonstration of isolated competen- bilirubin may not accurately diagnose the ability to manage ambiguous prob-
cies10; when we see the whole, we see and manage a patient with symptom- lems, tolerate uncertainty, and make de-
its parts differently than when we see atic gallstones. A competent clinician cisions with limited information.
2002 American Medical Association. All rights reserved. (Reprinted) JAMA, January 9, 2002Vol 287, No. 2 227

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


Competence depends on using ex- Habits of Mind toms and receives episodic care at an
pert scientific, clinical, and humanis- Competence depends on habits of mind inner-city clinic.
tic judgment to engage in clinical that allow the practitioner to be atten-
reasoning.14,15,17,18 Although expert cli- tive, curious, self-aware, and willing to Development
nicians often use pattern recognition for recognize and correct errors.43 Many Competence is developmental. There
routine problems19 and hypothetico- physicians would consider these hab- is debate about which aspects of com-
deductive reasoning for complex prob- its of mind characteristic of good prac- petence should be acquired at each stage
lems outside their areas of expertise, ex- tice, but they are especially difficult to of training. For example, early clinical
pert clinical reasoning usually involves objectify. A competent physician, for experiences and problem-based learn-
working interpretations12 that are elabo- example, should be able to judge his or ing formats encourage clinical reason-
rated into branching networks of con- her level of anxiety when facing an am- ing skills formerly relegated to the fi-
cepts.20-22 These networks help profes- biguous clinical presentation and be nal years of medical school. But students
sionals initiate a process of problem aware of how the anxiety of uncer- tend to use the same cognitive strat-
solving from minimal information and tainty may be influencing his or her egy for solving all problems, whereas
use subsequent information to refine clinical judgment. Errors in medicine, experts draw on several strategies,6
their understanding of the problem. according to this view, may result from which raises the question of whether as-
Reflection allows practitioners to overcertainty that ones impressions are sessment of practicing physicians
examine their own clinical reasoning beyond doubt.41,43,44 should be qualitatively different from
strategies. the assessment of a student. Determin-
Context ing how and at what level of training
Building Therapeutic Relationships Competence is context-dependent. the patient-physician relationship
The quality of the patient-physician re- Competence is a statement of relation- should be assessed is also difficult. For
lationship affects health and the recov- ship between an ability (in the per- example, participatory decision mak-
ery from illness,23,24 costs,25 and out- son), a task (in the world),45 and the ing correlates with clinical out-
comes of chronic diseases26-29 by altering ecology of the health systems and clini- comes,25,29 but it is unclear when train-
patients understanding of their ill- cal contexts in which those tasks oc- ees should be assessed on this skill.
nesses and reducing patient anxiety.26 cur.46,47 This view stands in contrast to Although a third-year resident might be
Key measurable patient-centered 28 an abstract set of attributes that the phy- expected to counsel a fearful diabetic
(or relationship-centered)30,31 behav- sician possessesknowledge, skills, patient about the need to start insulin,
iors include responding to patients and attitudesthat are assumed to a third-year student might be ex-
emotions and participatory decision serve the physician well in all the situ- pected only to elicit the patients pref-
making.29 ations that he or she encounters. For erences, emotions, and expectations.
Medical errors are often due to the example, rather than assessing a stu- Changes in medical practice and the
failure of health systems rather than dents competence in diagnosing and context of care invite redefinitions of
individual deficiencies.32-34 Thus, the treating heart disease (a disease- competence; for example, the use of
assessment of teamwork and institu- specific domain) by dividing it into electronic communication media48 and
tional self-assessment might effec- competencies (physical examination, changes in patient expectations.49,50
tively complement individual assess- interpretation of electrocardiogram, and
ments. pharmacology of -blockers), our view CURRENT MEANS
is that competence is defined by the in- OF ASSESSMENT
Affective and Moral Dimensions teraction of the task (the concrete pro- Assessment must take into account
Moral and affective domains of prac- cess of diagnosing and treating Mrs what is assessed, how it is assessed, and
tice may be evaluated more accurately Brown, a 52-year-old business execu- the assessments usefulness in foster-
by patients and peers than by licens- tive who is now in the emergency de- ing future learning. In discussing va-
ing bodies or superiors. 35 Only re- partment because of new-onset chest lidity of measures of competence in an
cently have validated measures cap- pain), the clinicians abilities (elicit- era when reliable assessments of core
tured some of the intangibles in ing information, forming a therapeu- knowledge, abstract problem solving,
medicine, such as trust36,37 and profes- tic relationship, performing diagnos- and basic clinical skills have been de-
sionalism.38,39 Recent neurobiological tic maneuvers, and making judgments veloped,45,51-56 we must now establish
research indicates that the emotions are about treatment), and the health sys- that they encompass the qualities that
central to all judgment and decision tem (good insurance and ready access define a good physician: the cogni-
making,13 further emphasizing the im- to care). Caring for Mrs Brown re- tive, technical, integrative, contex-
portance of assessing emotional intel- quires different skills than caring for Ms tual, relational, reflective, affective, and
ligence and self-awareness in clinical Hall, a 52-year-old uninsured home- moral aspects of competence. We dis-
practice.1,40-42 less woman who has similar symp- tinguish between expert opinion, in-
228 JAMA, January 9, 2002Vol 287, No. 2 (Reprinted) 2002 American Medical Association. All rights reserved.

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


termediate outcomes, and the few stud- level involves the ability to solve prob- petence of physicians, medical stu-
ies that show associations between lems and describe procedures. The dents, and residents. An initial search
results of assessments and actual clini- shows how level usually involves hu- using the following Medical Subject
cal performance.57-60 man (standardized patient), mechani- Headings of the National Library of
We consider how the process of cal, or computer simulations that in- Medicine yielded 2266 references: edu-
assessment might foster future learn- volve demonstration of skills in a cational measurement, patient simula-
ing. Too often, practitioners select edu- controlled setting. The does level re- tion, clinical competence OR profes-
cational programs that are unlikely to fers to observations of real practice. For sional competence AND reproducibility
influence clinical practice.61 Good as- each of these levels, the student can of results, validity OR research, OR the
sessment is a form of learning and should demonstrate the ability to imitate or text word reliability. This set was re-
provide guidance and support to ad- replicate a protocol, apply principles in duced by including any of 20 text words
dress learning needs. Finally, we ad- a familiar situation, adapt principles to describing assessment techniques; we
dress concerns that the medical profes- new situations, and associate new used words such as OSCE, oral exami-
sion still lacks adequate accountability knowledge with previously learned nation, peer assessment, triple jump, es-
to the public62 and has not done enough principles.65 say, portfolio, and CEX (clinical evalu-
to reduce medical errors.32,63 ation exercise), yielding 430 references.
Within each domain of assessment, METHODS Articles of a purely descriptive nature,
there are 4 levels at which a trainee Using the MEDLINE database for 1966 reviews that offered no new data, and
might be assessed (F IGURE ). 64 The to 2001, we searched for articles that opinions and position statements were
knows level refers to the recall of facts, studied the reliability or validity of mea- excluded, yielding 101 English-
principles, and theories. The knows how sures of clinical or professional com- language references. We surveyed the

Figure. A Framework for Assessment


Knows Shows New Chronic Acute

Knows Does Emergency Preventive
how how problem illness hospital

Self-assessment and reflection

Information gathering from patients and families

Relationship-building and professionalism

Sharing information, behavior change, and patient involvement

Physical examination

Patient procedural skills (suturing, drawing blood)

Interpretation of diagnostic tests (electrocardiogram,

mental status, imaging)

Diagnostic reasoning: Psychosocial issues

Diagnostic reasoning: Biomedical issues

Diagnostic reasoning: Diagnostic uncertainty

Clinical judgment: Planning further diagnostic workup

Clinical judgment: Generating therapeutic plan

Accessing, interpreting, and applying the medical literature

Presenting data to colleagues (referral letter, chart note)

Basic mechanisms (anatomy, immunology, microbiology)


Pathophysiology of disease (dermatology, renal, gastrointestinal)

Social science (epidemiology, psychology, culture/diversity)

Special topics (spirituality, ethics, economics)

The grid is filled out according to the type of assessment conducted, ie, standardized patient or simulation, video, postencounter probe, essay, or computer exercises.
Each category can be combined with a number designating a category such as the name of a patient, a type of computer exercise, or a team exercise.

2002 American Medical Association. All rights reserved. (Reprinted) JAMA, January 9, 2002Vol 287, No. 2 229

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


first 200 of the 2165 references ex- subsequent faculty76 and peer77 rat- correlate with multiple-choice exami-
cluded and found none that met our ings. Many have questioned the valid- nations and academic grades,90,100,117
search criteria. Quality criteria for in- ity of multiple-choice examinations, suggesting that these tools measure dif-
clusion were broad, given the small though.78-81 For example, compared ferent skills. Clinicians may behave dif-
number of controlled trials of assess- with Florida family physicians who are ferently in examination settings than in
ment interventions and the complex- not board-certified, those who are have real practice,106,118 and short OSCE sta-
ity of outcome measures. Because we nearly twice the risk of being sued.82 tions can risk fragmentation and trivi-
knew that MEDLINE search strategies Standardized test scores have been in- alization of isolated elements of what
would not capture all relevant studies, versely correlated with empathy, re- should be a coherent whole.119 The
we searched reference lists in the 101 sponsibility, and tolerance.83 Also, be- OSCE also has low test reliability for
articles, other review articles, and books cause of lack of expertise and resources, measuring clinical ethics.120
and did additional literature searches few medical school examinations can There are few validated strategies to
using the key authors of recent re- claim to achieve the high psychomet- assess actual clinical practice, or Mill-
views; we gathered 94 additional rel- ric standards of the licensing boards. ers does level. Subjective evaluation by
evant references. Of the 195 refer- The Objective Structured Clinical Ex- residents and attending physicians is the
ences, 124 presented new data on amination (OSCE) is a timed multista- major form of assessment during resi-
assessment of physicians. tion examination often using standard- dency and the clinical clerkships and
ized patients (SPs) to simulate clinical often includes the tacit elements of pro-
Summary of Studies scenarios. The roles are portrayed ac- fessional competence otherwise over-
The 3 most commonly used assess- curately56,84 and simulations are con- looked by objective assessment instru-
ment methods are subjective assess- vincing; the detection rate of unan- ments. Faculty ratings of humanism
ments by supervising clinicians, mul- nounced SPs in community practice is predicted patient satisfaction in one
tiple-choice examinations to evaluate less than 10%.57,59,85-89 Communica- study.121 However, evaluators often do
factual knowledge and abstract prob- tion, physical examination, counsel- not observe trainees directly. They
lem solving,66 and standardized pa- ing, and technical skills can be rated re- often have different standards122,123 and
tient assessments of physical examina- liably if there is a sufficiently large are subject to halo effects124 and racial
tion and technical and communication number of SP cases67,90-100 and if crite- and sex bias.125,126 Because of interpa-
skills.67-69 Although curricular designs ria for competence are based on evi- tient variability and low interrater re-
increasingly integrate core knowledge dence.101 Although few cases are needed liability, each trainee must be subject
and clinical skills, most assessment to assess straightforward skills, up to to multiple assessments for patterns to
methods evaluate these domains in iso- 27 cases may be necessary to assess in- emerge. Standardized rating forms for
lation. Few assessments use measures terpersonal skills reliably in high- direct observation of trainees127-132 and
such as participatory decision mak- stakes examinations.102,103 Although SPs structured oral examination formats
ing70 that predict clinical outcomes in ratings usually correlate with those of have been developed in response to this
real practice. Few reliably assess clini- real patients,104 differences have been criticism.133,134
cal reasoning, systems-based care, tech- noted.105-107 The Royal College of General Prac-
nology, and the patient-physician re- Defining pass/fail criteria for OSCEs titioners, dissatisfied with the capabil-
lationship. 3,69 The literature makes has been complex.54,108-111 There is de- ity of the OSCE to evaluate compe-
important distinctions between crite- bate about who should rate student per- tence for the final professional licensing
ria for licensing examinations and pro- formance in an OSCE.112 Ratings by the examination, developed a format in
gram-specific assessments with mixed SP are generally accurate52 but may be which candidates for certification pre-
formative and summative goals. hampered by memory failure, whereas sent several best-case videotapes of their
Evaluation of factual knowledge and external raters, either physicians or performance in real clinical settings to
problem-solving skills by using mul- other SPs, may be less attuned to affec- a trained examiner who uses specified
tiple-choice questions offers excellent tive aspects of the interview and sig- criteria for evaluation.135 Although the
reliability71-75 and assesses some as- nificantly increase the cost of the ex- face validity of such a measure is high
pects of context and clinical reason- amination. and the format is well accepted by phy-
ing. Scores on Canadian licensing ex- Checklist scores completed by phy- sicians,136 the number of cases that
aminations, which include standardized sician-examiners in some studies im- should be presented to achieve ad-
patient assessment and multiple- prove with expertise of the examin- equate reliability is unclear.137-139
choice tests, correlated positively with ees113 and with the reputation of the Profiling by managed-care data-
subsequent appropriate prescribing, training program.90,114 But global rat- bases is increasingly used as an evalu-
mammographic screening, and refer- ing scales of interpersonal skills may be ation measure of clinical competence.
rals,58 and multiple-choice certifica- more valid than behavioral check- However, data abstraction is com-
tion examination scores correlated with lists.7,115,116 The OSCE scores may not plex140 and defining competence in
230 JAMA, January 9, 2002Vol 287, No. 2 (Reprinted) 2002 American Medical Association. All rights reserved.

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


terms of cost and value is difficult. The

underlying assumptions driving such Box 2. Some Purposes of Assessment
evaluation systems may not be ex- For the Trainee
plicit. For example, cost analyses may
Provide useful feedback about individual strengths and weaknesses that guides
favor physicians caring for more highly future learning
educated patients.141 Foster habits of self-reflection and self-remediation
Peer ratings are accurate and reli- Promote access to advanced training
able measures of physician perfor-
For the Curriculum
mance.77,142 Peers may be in the best po-
Respond to lack of demonstrated competence (denial of promotion, mandated
sition to evaluate professionalism;
people often act differently when not Certify achievement of curricular goals
under direct scrutiny.143 Anonymous Foster course or curricular change
medical student peer assessments of Create curricular coherence
professionalism have raised aware- Cross-validate other forms of assessment in the curriculum
ness of professional behavior, fostered Establish standards of competence for trainees at different levels
further reflection, helped students iden- For the Institution
tify specific mutable behaviors, and Guide a process of institutional self-reflection and remediation
been well accepted by students.35 Stu- Discriminate among candidates for further training or promotion
dents should be assessed by at least 8 Express institutional values by determining what is assessed and how assessment
of their classmates. The composite re- is conducted
sults should be edited to protect the Develop shared educational values among a diverse community of educators
confidentiality of the raters. Promote faculty development
Self-assessments have been used with Provide data for educational research
some success in standardized patient For the Public
exercises144 and in programs that offer Certify competence of graduates
explicit training in the use of self-
assessment instruments. 145 Among
trainees who did not have such train-
ing, however, self-assessment was nei- tinctions between these goals often are Whereas performance is directly mea-
ther valid nor accurate. Rather, it was blurred in practice. For example, for- surable, competence is an inferred qual-
more closely linked to the trainees psy- mative feedback is intended to foster in- ity. 148 Performance on a multiple-
chological sense of self-efficacy and self- dividual reflection and remediation146 choice test may exceed competence, as
confidence than to appropriate crite- but may be perceived as having evalu- in the case of a trainee with a photo-
ria, even among bright and motivated ative consequences. Summative evalu- graphic memory but poor clinical judg-
individuals. ation is a powerful means for driving ment. Conversely, competence may ex-
curricular content and what students ceed test performance, as in the case of
COMMENT learn. Assessment provides informa- a trainee with severe test anxiety. Cor-
Aside from the need to protect the pub- tion to allow institutions to choose relation with National Board scores and
lic by denying graduation to those few among candidates for advanced train- feedback on graduates performance can
trainees who are not expected to over- ing. The public expects greater self- be useful in validating some assess-
come their deficiencies, the outcomes of monitoring, communication, and team- ment instruments but should be done
assessment should foster learning, in- work from health care practitioners.147 with caution. For example, efficiency is
spire confidence in the learner, en- The decline of public trust in medicine highly valued in residents but less so in
hance the learners ability to self- may reflect a growing concern that phy- medical students.
monitor, and drive institutional self- sicians are not achieving these goals.36
assessment and curricular change. Given Assessment is also a statement of in- Future Directions
the difficulty in validating tests of basic stitutional values. Devoting valuable Medical schools in Canada, the United
skills, it is not surprising that there is curricular time to peer assessment of Kingdom, Australia, Spain, the Neth-
scant literature on the assessment of professionalism, for example, can pro- erlands, and the United States have
learning, professionalism, teamwork, and mote those values that are assessed made commitments to developing in-
systems-based care or on the ability of while encouraging curricular coher- novative assessments of professional
assessment programs to drive curricu- ence and faculty development, espe- competence, some of which we de-
lar change or reduce medical errors. cially if there are corresponding ef- scribe. These assessments are increas-
Assessment serves personal, institu- forts at the institution toward self- ingly multimodal and tailored to the
tional, and societal goals (BOX 2). Dis- assessment and change. goals and context in which they will be
2002 American Medical Association. All rights reserved. (Reprinted) JAMA, January 9, 2002Vol 287, No. 2 231

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


that use patient vignettes followed by and then asked to judge how her diag-
Box 3. Innovations in Assessing questions that require clinical judg- nostic hypotheses or therapeutic ac-
Professional Competence ment. These measures reflect stu- tions would change. Failure to in-
Multimethod assessment dents capacity to organize and link in- clude a key diagnostic possibility or the
Clinical reasoning in situations that formation; also, they predict clinical overestimation or underestimation of
involve clinical uncertainty reasoning ability 2 years later.153 Com- probability are criteria for evaluation.
Standardized patient exercises linked bining formats appears to have added The goal of the exercise is to demon-
to postencounter probes of value with no loss in reliability.150,154 strate emotional intelligence40 and self-
pathophysiology and clinical Ongoing educational outcomes re- awareness in the context of conflict and
search will show whether composite ambiguity. Similar observations might
Exercises to assess use of the
medical literature
formats help students learn how to learn be made with trainees video portfo-
Long-station standardized patient more effectively, develop habits of mind lios of real clinical encounters.
exercises that characterize exemplary prac-
Simulated continuity tice,43 and provide a more multidimen- Well-functioning health systems are
Teamwork exercises sional picture of the examinee than the characterized by continuity, partner-
Unannounced standardized individual unlinked elements. Two ex- ship between physicians and patients,
patients in clinical settings amples of comprehensive assessment teamwork between health care practi-
Assessments by patients formats follow. tioners, and communication between
Peer assessment of professionalism health care settings.156,157 The use of time
Portfolios of videotapes Genetics, Evidence-Based Medi- in a continuity relationship can be
Mentored self-assessment
cine, Screening, and Communica- assessed with a series of SP or real-
Remediation based on a learning plan
tion. A student is instructed to per- patient exercises. To assess partner-
form a literature search about genetic ship, patient assessment, currently used
screening test for Alzheimer disease in to assess physicians in practice,158 is being
used. Large-scale licensure examina- anticipation of an SP encounter later that tested for students and residents.159,160
tions must use computer-gradable for- day. Assessment instruments include a These efforts are guided by data show-
mats, but comprehensive examina- structured evaluation of the search strat- ing that patients ratings of communi-
tions using structured direct egy and a communication rating scale, cation and satisfaction correlate well with
observation,107 OSCE stations, real pa- completed by an SP, that assesses the biomedical outcomes,24,29 emotional dis-
tient cases,107 case-based questions,79 clarity of the students presentation and tress,161 health care use,25 and malprac-
peer assessments, and essay-type ques- the students ability to involve the pa- tice litigation.162 Patient ratings also have
tions149 are reliable as well. Propo- tient in the decision-making process. the potential to validate other measures
nents of the new formats argue that they Next, the student completes an essay of competence.163 Several institutions
provide more useful feedback and are about the ethics of genetic screening and assess teamwork by using peer assess-
more efficient at the medical school or the genetics of Alzheimer disease. This ments. Others use sophisticated man-
residency level (Box 1 and BOX 3) than exercise assesses the students commu- nequins to simulate acute cardiovascu-
traditional formats.81,150 They target core nication skills, clinical reasoning, abil- lar physiological derangements found in
knowledge and clinical skills in differ- ity to acquire and use new knowledge, intensive care settings164-169; trainees are
ent contexts and at different levels of and contextualized use of knowledge of graded on teamwork as well as indi-
assessment. Because of their complex- genetics, health economics, and medi- vidual problem solving, and statistical
ity, a matrix (Figure) can be useful to cal ethics. adjustments can account for team com-
display the domains assessed. Cognitive and Affective Chal- position. Communication between
Comprehensive assessments link lenges of Clinical Uncertainty. A rat- health settings could be assessed at the
content across several formats. Post- ing scale is used to assess a resident on student level, for example, by grading
encounter probes immediately after SP her ability to agree on a plan of action of their written referral letters.170
exercises using oral, essay, or multiple- with an SP who portrays an outpatient Although it could be argued that li-
choice questions test pathophysiology demanding a computed tomographic censing boards do not have the man-
and clinical reasoning in context.151,152 scan for headaches without neurologi- date to remediate examinees who per-
Triple-jump exercises152consisting of cal signs. In a postencounter exercise, form poorly or modify educational
a case presentation, an independent lit- the resident creates a rank-order dif- curricula, medical schools and resi-
erature search, and then an oral or writ- ferential diagnosis and then answers a dency programs do. Tests that demon-
ten postencounter examinationtest series of script concordance153,155 ques- strate students strengths or weak-
the use and application of the medical tions in which the examinee is pre- nesses may not provide the student with
literature. Validated measures of reflec- sented hypothetical additional data (for the opportunity to reflect on actual be-
tive thinking153 have been developed example, numbness in the left hand) haviors and patterns of thought that
232 JAMA, January 9, 2002Vol 287, No. 2 (Reprinted) 2002 American Medical Association. All rights reserved.

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


should be changed. To foster reflec- 3. ACGME Outcome Project. Accreditation Council nication and health outcomes: a review. CMAJ. 1995;
for Graduate Medical Education Web site. Available 152:1423-1433.
tion and action, some institutions re- at: 2000. Accessed October 28. Stewart M, Brown JB, Boon H, Galajda J, Me-
quire a learning plan in which train- 1, 2001. redith L, Sangster M. Evidence on patient-doctor com-
4. Sackett DL. Evidence-Based Medicine: How to Prac- munication. Cancer Prev Control. 1999;3:25-30.
ees chart their learning needs, the tice and Teach EBM. New York, NY: Churchill Living- 29. Kaplan SH, Greenfield S, Ware JE Jr. Assessing the
means of achieving them, expected time stone; 1997. effects of physician-patient interactions on the out-
of completion, and means of verifica- 5. Polanyi M. Personal Knowledge: Towards a Post- comes of chronic disease [published correction ap-
Critical Philosophy. Chicago, Ill: University of Chi- pears in Med Care. 1989;27:679]. Med Care. 1989;
tion146,171 as a required outcome of an cago Press; 1974. 27:S110-S127.
assessment. 6. Benner P. From Novice to Expert. Menlo Park, Calif: 30. Tresolini C, and the Pew-Fetzer Task Force. Health
Addison-Wesley; 1984. Professions Education and Relationship-Centered Care:
A strong mentoring system should 7. Hodges B, Regehr G, McNaughton N, Tiberius R, Report of the Pew-Fetzer Task Force on Advancing
accompany any comprehensive assess- Hanson M. OSCE checklists do not capture increas- Psychosocial Health Education. San Francisco, Calif:
ing levels of expertise. Acad Med. 1999;74:1129- Pew Health Professions Commission; 1994.
ment program. An inadequate system 1134. 31. Candib LM. Medicine and the Family: A Femi-
for feedback, mentoring, and remedia- 8. Dewey J. Experience and Nature. 1932 ed. New nist Perspective. New York, NY: Basic Books; 1995.
tion will subvert even the most well- York, NY: Dover; 1958. 32. Ternov S. The human side of medical mistakes.
9. Langer EJ. The Power of Mindful Learning. Read- In: Spath PL, ed. Error Reduction in Healthcare: A Sys-
conceived and validated examination. ing, Mass: Perseus Books; 1997. tems Approach to Improving Patient Safety. San Fran-
Curricular change also can be guided 10. Eraut M. Learning professional processes: public cisco, Calif: Jossey Bass; 1999:97-138.
knowledge and personal experience. In: Eraut M, ed. 33. Chopra V, Bovill JG, Spierdijk J, Koornneef F. Re-
by results of assessments but requires Developing Professional Knowledge and Compe- ported significant observations during anaesthesia: a
a parallel process of institutional re- tence. London, England: Falmer Press; 1994:100- prospective analysis over an 18-month period. Br J An-
flection, feedback, and remediation. 122. aesth. 1992;68:13-17.
11. Polanyi M. The logic of tacit inference. In: Grene 34. Cooper JB, Newbower RS, Long CD, McPeek B.
These new assessment formats are M, ed. Knowing and Being: Essays. Chicago, Ill: Uni- Preventable anesthesia mishaps: a study of human fac-
feasible, and several institutions have versity of Chicago; 1969:123-158. tors. Anesthesiology. 1978;49:399-406.
12. Gale J, Marsden P. Clinical problem solving: the 35. Asch E, Saltzberg D, Kaiser S. Reinforcement of
invested significant time and re- beginning of the process. Med Educ. 1982;16:22- self-directed learning and the development of pro-
sources to develop them. The promise 26. fessional attitudes through peer- and self-
13. Damasio AR. Descartes Error: Emotion, Reason, assessment. Acad Med. 1998;73:575.
that a more comprehensive assess- and the Human Brain. New York, NY: GP Putnams 36. Safran DG, Montgomery JE, Chang H, Murphy
ment of professional competence might Sons; 1994. J, Rogers WH. Switching doctors: predictors of vol-
improve practice, change medical edu- 14. Mandin H, Jones A, Woloschuk W, Harasym P. untary disenrollment from a primary physicians prac-
Helping students learn to think like experts when solv- tice. J Fam Pract. 2001;50:130-136.
cation, and reduce medical errors ing clinical problems. Acad Med. 1997;72:173-179. 37. Safran DG, Taira DA, Rogers WH, Kosinski M,
should be studied in controlled trials. 15. Friedman MH, Connell KJ, Olthoff AJ, Sinacore Ware JE, Tarlov AR. Linking primary care perfor-
JM, Bordage G. Medical student errors in making a mance to outcomes of care. J Fam Pract. 1998;47:
The publics trust in the medical pro- diagnosis. Acad Med. 1998;73(suppl):S19-S21. 213-220.
fession and the ability of medical prac- 16. Schon DA. The Reflective Practitioner. New York, 38. American Board of Internal Medicine. Project Pro-
titioners to learn from mistakes de- NY: Basic Books; 1983. fessionalism. Philadelphia, Pa: American Board of In-
17. Feinstein AR. Clinical Judgment revisited: the ternal Medicine; 1998.
pends on valid and reliable means of distraction of quantitative models. Ann Intern Med. 39. Peabody F. The care of the patient. JAMA. 1927;
assessment. Medical educators, profes- 1994;120:799-805. 88:877-882.
18. Downie RS, Macnaughton J, Randall R. Clinical 40. Goleman D. Emotional Intelligence. New York,
sional societies, and licensing boards Judgement: Evidence in Practice. Oxford, England: Ox- NY: Bantam Books; 1995.
should view professional competence ford University Press; 2000. 41. Novack DH, Suchman AL, Clark W, Epstein RM,
19. Dunn MM, Woolliscroft JO. Assessment of a pat- Najberg E, Kaplan C. Calibrating the physician: per-
more comprehensively to improve the tern-recognition examination in a clinical clerkship. sonal awareness and effective patient care. JAMA.
process of assessment. Acad Med. 1994;69:683-684. 1997;278:502-509.
20. Schmidt HG, Norman GR, Boshuizen HP. A cog- 42. Ramsey PG, Wenrich M, Carline JD, Inui TS, Lar-
Author Contributions: Study concept and design, criti- nitive perspective on medical expertise: theory and im- son EB, LoGerfo JP. Use of peer ratings to evaluate
cal revision of the manuscript for important intellec- plication [published correction appears in Acad Med. physician performance. JAMA. 1993;269:1655-
tual content, and administrative, technical, or mate- 1992;67:287]. Acad Med. 1990;65:611-621. 1660.
rial support: Epstein, Hundert. 21. Bordage G, Zacks R. The structure of medical 43. Epstein RM. Mindful practice. JAMA. 1999;282:
Acquisition of data, analysis and interpretation of data, knowledge in the memories of medical students and 833-839.
and drafting of the manuscript: Epstein. general practitioners: categories and prototypes. Med 44. Novack DH, Epstein RM, Paulsen RH. Toward cre-
Acknowledgment: We would like to express thanks Educ. 1984;18:406-416. ating physician-healers: fostering medical students self-
to Francesc Borrell-Carrio, MD, Daniel Federman, MD, 22. Bordage G. Elaborated knowledge: a key to suc- awareness, personal growth, and well-being. Acad
Brian Hodges, MD, Daniel Klass, MD, Larry Mauksch, cessful diagnostic thinking. Acad Med. 1994;69:883- Med. 1999;74:516-520.
CSW, Timothy Quill, MD, Andres Sciolla, MD, and 885. 45. Klass D. Reevaluation of clinical competency. Am
Kevin Volkan, PhD, for their critical review of the manu- 23. Starfield B, Wray C, Hess K, Gross R, Birk PS, J Phys Med Rehabil. 2000;79:481-486.
script. Also we would like to acknowledge Anthony DLugoff BC. The influence of patient-practitioner 46. LaDuca A, Taylor DD, Hill IK. The design of a new
LaDuca, PhD, and Albert Oriol-Bosch, MD, for their agreement on outcome of care. Am J Public Health. physician licensure examination. Eval Health Prof.
contributions to our formulation of professional com- 1981;71:127-131. 1984;7:115-140.
petence. 24. The Headache Study Group of the University of 47. Kane M. Model-based practice analysis and test
Western Ontario. Predictors of outcome in headache specifications. Appl Meas Educ. 1997;10:18.
patients presenting to family physiciansa one year 48. Dugdale DC, Epstein R, Pantilat SZ. Time and the
prospective study. Headache. 1986;26:285-294. patient-physician relationship. J Gen Intern Med. 1999;
1. American Board of Internal Medicine. Guide to 25. Stewart M, Brown JB, Donner A, et al. The im- 14(suppl 1):S34-S40.
Evaluation of Residents in Internal Medicine. Phila- pact of patient-centered care on outcomes. J Fam Pract. 49. Emanuel EJ, Dubler NN. Preserving the physician-
delphia, Pa: American Board of Internal Medicine; 2000;49:796-804. patient relationship in the era of managed care. JAMA.
1999. 26. Di Blasi Z, Harkness E, Ernst E, Georgiou A, Klei- 1995;273:323-335.
2. Norman GR. Defining competence: a method- jnen J. Influence of context effects on health out- 50. Kravitz RL, Cope DW, Bhrany V, Leake B. Inter-
ological review. In: Neufeld VR, Norman GR, eds. As- comes: a systematic review. Lancet. 2001;357:757- nal medicine patients expectations for care during of-
sessing Clinical Competence. New York, NY: Springer; 762. fice visits. J Gen Intern Med. 1994;9:75-81.
1985:15-35. 27. Stewart M. Effective physician-patient commu- 51. LaDuca A. Validation of professional licensure ex-

2002 American Medical Association. All rights reserved. (Reprinted) JAMA, January 9, 2002Vol 287, No. 2 233

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


aminations: professions theory, test design, and con- Reliability, validity and efficiency of multiple choice tion and clinical examination. Med Teacher. 1992;14:
struct validity. Eval Health Prof. 1994;17:178-197. question and patient management problem item for- 179-183.
52. De Champlain AF, Margolis MJ, King A, Klass DJ. mats in assessment of clinical competence. Med Educ. 94. MacRae HM, Cohen R, Regehr G, Reznick R, Burn-
Standardized patients accuracy in recording exam- 1985;19:238-247. stein M. A new assessment tool: the patient assess-
inees behaviors using checklists. Acad Med. 1997; 74. Case SM, Ripkey DR, Swanson DB. The relation- ment and management examination. Surgery. 1997;
72(suppl 1):S85-S87. ship between clinical science performance in 20 medi- 122:335-343.
53. Kopelow ML, Schnabl GK, Hassard TH, et al. As- cal schools and performance on Step 2 of the USMLE 95. Matsell DG, Wolfish NM, Hsu E. Reliability and
sessing practicing physicians in two settings using stan- licensing examination: 1994-95 Validity Study Group validity of the objective structured clinical examina-
dardized patients. Acad Med. 1992;67(suppl):S19- for USMLE Step 1 and 2 Pass/Fail Standards. Acad tion in paediatrics. Med Educ. 1991;25:293-299.
S21. Med. 1996;71(suppl):S31-S33. 96. Vu NV, Barrows HS, Marcy ML, Verhulst SJ, Col-
54. Margolis MJ, De Champlain AF, Klass DJ. Set- 75. Case SM, Swanson DB. Validity of NBME Part I liver JA, Travis T. Six years of comprehensive, clinical,
ting examination-level standards for a performance- and Part II scores for selection of residents in ortho- performance-based assessment using standardized pa-
based assessment of physicians clinical skills. Acad paedic surgery, dermatology, and preventive medi- tients at the Southern Illinois University School of Medi-
Med. 1998;73(suppl):S114-S116. cine. Acad Med. 1993;68(suppl):S51-S56. cine. Acad Med. 1992;67:42-50.
55. Ross LP, Clauser BE, Margolis MJ, Orr NA, Klass 76. Downing SM. The assessment of clinical compe- 97. Vu NV, Distlehorst LH, Verhulst SJ, Colliver JA.
DJ. An expert-judgment approach to setting stan- tence on the emergency medicine specialty certifica- Clinical performance-based test sensitivity and speci-
dards for a standardized-patient examination. Acad tion examination: the validity of clinically relevant mul- ficity in predicting first-year residency performance.
Med. 1996;71(suppl):S4-S6. tiple-choice items. Ann Emerg Med. 1980;9:554- Acad Med. 1993;68(suppl):S41-S45.
56. Tamblyn RM, Klass DJ, Schnabl GK, Kopelow ML. 556. 98. Newble DI, Swanson DB. Psychometric charac-
The accuracy of standardized patient presentation. Med 77. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Lar- teristics of the objective structured clinical examina-
Educ. 1991;25:100-109. son EB, LoGerfo JP. Use of peer ratings to evaluate tion. Med Educ. 1988;22:325-334.
57. McLeod PJ, Tamblyn RM, Gayton D, et al. Use physician performance. JAMA. 1993;269:1655- 99. Petrusa ER, Blackwell TA, Ainsworth MA. Reli-
of standardized patients to assess between-physician 1660. ability and validity of an objective structured clinical
variations in resource utilization. JAMA. 1997;278: 78. Holsgrove GJ. Guide to postgraduate exams: mul- examination for assessing the clinical performance of
1164-1168. tiple-choice questions. Br J Hosp Med. 1992;48:757- residents. Arch Intern Med. 1990;150:573-577.
58. Tamblyn R, Abrahamowicz M, Brailovsky C, et al. 761. 100. Gomez JM, Prieto L, Pujol R, et al. Clinical skills
Association between licensing examination scores and 79. Schuwirth LW, Verheggen MM, van der Vleu- assessment with standardized patients. Med Educ.
resource use and quality of care in primary care prac- ten C, Boshuizen HP, Dinant GJ. Do short cases elicit 1997;31:94-98.
tice. JAMA. 1998;280:989-996. different thinking processes than factual knowledge 101. Gorter S, Rethans JJ, Scherpbier A, et al. Devel-
59. Tamblyn RM. Use of standardized patients in the questions do? Med Educ. 2001;35:348-356. oping case-specific checklists for standardized-patient-
assessment of medical practice. CMAJ. 1998;158: 80. Veloski JJ, Rabinowitz HK, Robeson MR, Young based assessments in internal medicine: a review of
205-207. PR. Patients dont present with five choices: an alter- the literature. Acad Med. 2000;75:1130-1137.
60. Abrahamowicz M, Tamblyn RM, Ramsay JO, Klass native to multiple-choice tests in assessing physi- 102. Swartz MH, Colliver JA, Bardes CL, Charon R,
DK, Kopelow ML. Detecting and correcting for rater- cians competence. Acad Med. 1999;74:539-546. Fried ED, Moroff S. Validating the standardized-
induced differences in standardized patient tests of clini- 81. van der Vleuten C. Validity of final examinations patient assessment administered to medical students
cal competence. Acad Med. 1990;65(suppl):S25- in undergraduate medical training. BMJ. 2000;321: in the New York City Consortium. Acad Med. 1997;
S26. 1217-1219. 72:619-626.
61. Davis D. Does CME work? an analysis of the effect 82. Ely JW, Dawson JD, Young PR, et al. Malprac- 103. Carline JD, Paauw DS, Thiede KW, Ramsey PG.
of educational activities on physician performance or tice claims against family physicians: are the best doc- Factors affecting the reliability of ratings of students
health care outcomes. Int J Psychiatry Med. 1998;28: tors sued more? J Fam Pract. 1999;48:23-30. clinical skills in a medicine clerkship. J Gen Intern Med.
21-39. 83. Tutton PJ. Psychometric test results associated with 1992;7:506-510.
62. Kassirer JP. Pseudoaccountability. Ann Intern Med. high achievement in basic science components of a 104. Sanson-Fisher RW, Poole AD. Simulated pa-
2001;134:587-590. medical curriculum. Acad Med. 1996;71:181-186. tients and the assessment of medical students inter-
63. Kohn LT, Corrigan JM, Donaldson MS. To Err Is 84. Tamblyn RM, Klass DK, Schanbl GK, Kopelow ML. personal skills. Med Educ. 1980;14:249-253.
Human: Building a Safer Health System. Washing- Factors associated with the accuracy of standardized 105. Tamblyn R, Abrahamowicz M, Schnarch B, et
ton, DC: National Academy Press; 2000. patient presentation. Acad Med. 1990;65(suppl):S55- al. Can standardized patients predict real-patient sat-
64. Miller GE. The assessment of clinical skills/ S56. isfaction with the doctor-patient relationship? Teach-
competence/performance. Acad Med. 1990;65(suppl): 85. Tamblyn R, Berkson L, Dauphinee WD, et al. Un- ing Learning Med. 1994;6:36-44.
S63-S67. necessary prescribing of NSAIDs and the manage- 106. Pieters HM, Touw-Otten FW, De Melker RA.
65. Broudy HS, Smith BO, Burnett J. Democracy and ment of NSAID-related gastropathy in medical prac- Simulated patients in assessing consultation skills of
Excellence in American Secondary Education. Chi- tice. Ann Intern Med. 1997;127:429-438. trainees in general practice vocational training: a va-
cago, Ill: Rand McNally; 1964. 86. Carney PA, Allen JD, Eliassen MS, Owen M, Bad- lidity study. Med Educ. 1994;28:226-233.
66. Newble D, Dawson B, Dauphinee D. Guidelines ger LW. Recognizing and managing depression in pri- 107. Wass V, Jones R, van der Vleuten C. Standard-
for assessing clinical competence. Teaching Learning mary care: a standardized patient study. J Fam Pract. ized or real patients to test clinical competence? the
Med. 1994;6:213-220. 1999;48:965-972. long case revisited. Med Educ. 2001;35:321-325.
67. Colliver JA, Swartz MH. Assessing clinical perfor- 87. Carney PA, Dietrich AJ, Freeman DH Jr, Mott LA. 108. Cusimano MD, Rothman A, Keystone J. Defin-
mance with standardized patients. JAMA. 1997;278: The periodic health examination provided to asymp- ing standards of competent performance on an OSCE.
790-791. tomatic older women: an assessment using standard- Acad Med. 1998;73(suppl):S112-S113.
68. Barrows HS. An overview of the uses of stan- ized patients. Ann Intern Med. 1993;119:129-135. 109. Kaufman DM, Mann KV, Muijtjens AM, van der
dardized patients for teaching and evaluating clinical 88. Carney PA, Dietrich AJ, Freeman DH Jr, Mott LA. Vleuten C. A comparison of standard-setting proce-
skills: AAMC. Acad Med. 1993;68:443-451. A standardized-patient assessment of a continuing dures for an OSCE in undergraduate medical educa-
69. Kassebaum DG, Eaglen RH. Shortcomings in the medical education program to improve physicians can- tion. Acad Med. 2000;75:267-271.
evaluation of students clinical skills and behaviors in cer-control clinical skills. Acad Med. 1995;70:52-58. 110. Travis TA, Colliver JA, Robbs RS, et al. Validity of
medical school. Acad Med. 1999;74:842-849. 89. Tamblyn RM, Klass DJ, Schnabl GK, Kopelow ML. simple approach to scoring and standard setting for stan-
70. Greenfield S, Kaplan S, Ware JE Jr. Expanding pa- The accuracy of standardized patient presentation. Med dardized-patient cases in an examination of clinical com-
tient involvement in care: effects on patient out- Educ. 1991;25:100-109. petence. Acad Med. 1996;71(suppl):S84-S86.
comes. Ann Intern Med. 1985;102:520-528. 90. van der Vleuten C, Sawnson DB. Assessment of 111. Norcini JJ, Stillman PL, Sutnick AI, et al. Scoring
71. Case SM, Becker DF, Swanson DB. Relationship clinical skills with standardized patients: state of the and standard setting with standardized patients. Eval
between scores on NBME basic science subject tests art. Teaching Learning Med. 1990;2:58-76. Health Prof. 1993;16:322-332.
and the first administration of the newly designed 91. GrandMaison P, Lescop J, Rainsberry P, Brai- 112. Martin JA, Reznick RK, Rothman A, Tamblyn RM,
NBME Part I examination. Acad Med. 1992;67(suppl): lovsky CA. Large-scale use of an objective, struc- Regehr G. Who should rate candidates in an objec-
S13-S15. tured clinical examination for licensing family physi- tive structured clinical examination? Acad Med. 1996;
72. Norcini JJ, Swanson DB, Grosso LJ, Shea JA, Web- cians. CMAJ. 1992;146:1735-1740. 71:170-175.
ster GD. A comparison of knowledge, synthesis, and 92. Joorabchi B. Objective structured clinical exami- 113. Blane CE, Calhoun JG. Objectively evaluating stu-
clinical judgment: multiple-choice questions in the as- nation in a pediatric residency program. AJDC. 1991; dent case presentations. Invest Radiol. 1985;20:121-
sessment of physician competence. Eval Health Prof. 145:757-762. 123.
1984;7:485-499. 93. Latif A. An examination of the examinations: the 114. Stillman P, Swanson D, Regan MB, et al. As-
73. Norcini JJ, Swanson DB, Grosso LJ, Webster GD. reliability of the objective structured clinical examina- sessment of clinical skills of residents utilizing stan-

234 JAMA, January 9, 2002Vol 287, No. 2 (Reprinted) 2002 American Medical Association. All rights reserved.

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015


dardized patients: a follow-up study and recommen- 134. Stillman RM, Lane KM, Beeth S, Jaffe BM. Evalu- riculum at the University of Hawaii. Acad Med. 1993;
dations for application. Ann Intern Med. 1991;114: ation of the student: improving validity of the oral ex- 68:366-372.
393-401. amination. Surgery. 1983;93:439-442. 153. Brailovsky C, Charlin B, Beausoleil S, Cote S, van
115. Cohen R, Rothman AI, Poldre P, Ross J. Valid- 135. Lockie C. The Examination for Membership in der Vleuten C. Measurement of clinical reflective ca-
ity and generalizability of global ratings in an objec- the Royal College of General Practitioners. London, pacity early in training as a predictor of clinical rea-
tive structured clinical examination. Acad Med. 1991; England: Royal College of General Practitioners; 1990. soning performance at the end of residency: an ex-
66:545-548. Occasional Paper 46. perimental study on the script concordance test. Med
116. Swartz MH, Colliver JA, Bardes CL, Charon R, Fried 136. Ram P, van der Vleuten C, Rethans JJ, Grol R, Educ. 2001;35:430-436.
ED, Moroff S. Global ratings of videotaped perfor- Aretz K. Assessment of practicing family physicians: 154. Verhoeven BH, Hamers JG, Scherpbier AJ,
mance versus global ratings of actions recorded on check- comparison of observation in a multiple-station ex- Hoogenboom RJ, van der Vleuten C. The effect on
lists: a criterion for performance assessment with stan- amination using standardized patients with observa- reliability of adding a separate written assessment com-
dardized patients. Acad Med. 1999;74:1028-1032. tion of consultations in daily practice. Acad Med. 1999; ponent to an objective structured clinical examina-
117. Mumford E, Schlesinger H, Cuerdon T, Scully J. 74:62-69. tion. Med Educ. 2000;34:525-529.
Ratings of videotaped simulated patient interviews and 137. Ram P, van der Vleuten C, Rethans JJ, Schouten 155. Charlin B, Roy L, Brailovsky C, Goulet F, van der
four other methods of evaluating a psychiatry clerk- B, Hobma S, Grol R. Assessment in general practice: Vleuten C. The Script Concordance test: a tool to as-
ship. Am J Psychiatry. 1987;144:316-322. the predictive value of written-knowledge tests and sess the reflective clinician. Teaching Learning Med.
118. Tamblyn RM, Abrahamowicz M, Berkson L, et a multiple-station examination for actual medical per- 2000;12:189-195.
al. First-visit bias in the measurement of clinical com- formance in daily practice. Med Educ. 1999;33:197- 156. Risser DT, Simon R, Rice MM, Salisbury ML. A
petence with standardized patients. Acad Med. 1992; 203. structured teamwork system to reduce clinical errors.
67(suppl):S22-S24. 138. Ram P, Grol R, Rethans JJ, Schouten B, van der In: Spath PL, ed. Error Reduction in Health Care: A
119. Ferrell BG, Thompson BL. Standardized pa- Vleuten C, Kester A. Assessment of general practi- Systems Approach to Improving Patient Safety. New
tients: a long-station clinical examination format. Med tioners by video observation of communicative and York, NY: John Wiley & Sons; 1999:235-277.
Educ. 1993;27:376-381. medical performance in daily practice: issues of va- 157. Suchman AL, Botelho RJ, Hinton-Walker P. Part-
120. Singer PA, Robb A, Cohen R, Norman G, Turn- lidity, reliability and feasibility. Med Educ. 1999;33: nerships in Healthcare: Transforming Relational Pro-
bull J. Performance-based assessment of clinical eth- 447-454. cess. Rochester, NY: University of Rochester Press;
ics using an objective structured clinical examination. 139. Tate P, Foulkes J, Neighbour R, Campion P, Field 1998.
Acad Med. 1996;71:495-498. S. Assessing physicians interpersonal skills via video- 158. American Board of Internal Medicine. Patient and
121. McLeod PJ, Tamblyn R, Benaroya S, Snell L. Fac- taped encounters: a new approach for the Royal Col- Physician Peer Assessment and the ABIM Program of
ulty ratings of resident humanism predict patient sat- lege of General Practitioners Membership examina- Continuous Professional Development. Philadel-
isfaction ratings in ambulatory medical clinics. J Gen tion. J Health Commun. 1999;4:143-152. phia, Pa: American Board of Internal Medicine; 2001.
Intern Med. 1994;9:321-326. 140. Luck J, Peabody JW, Dresselhaus TR, Lee M, 159. Dauphinee WD. Assessing clinical perfor-
122. Irby D. Teaching and learning in ambulatory care Glassman P. How well does chart abstraction mea- mance. Where do we stand and what might we ex-
settings: a thematic review of the literature. Acad Med. sure quality? a prospective comparison of standard- pect? JAMA. 1995;274:741-743.
1995;70:898-931. ized patients with the medical record. Am J Med. 2000; 160. Violato C, Marini A, Toews J, Lockyer J, Fidler
123. Kalet A, Earp JA, Kowlowitz V. How well do fac- 108:642-649. H. Feasibility and psychometric properties of using
ulty evaluate the interviewing skills of medical stu- 141. Fiscella K, Franks P. Influence of patient educa- peers, consulting physicians, co-workers, and pa-
dents? J Gen Intern Med. 1992;7:499-505. tion on profiles of physician practices. Ann Intern Med. tients to assess physicians. Acad Med. 1997;72
124. Rowland-Morin PA, Burchard KW, Garb JL, Coe 1999;131:745-751. (suppl 1):S82-S84.
NP. Influence of effective communication by surgery 142. Tornow W, London M. Maximizing the Value 161. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA,
students on their oral examination scores. Acad Med. of 360-Degree Feedback: A Process for Successful In- Roca RP. Improving physicians interviewing skills and
1991;66:169-171. dividual and Organizational Development. San Fran- reducing patients emotional distressa randomized
125. Wang-Cheng RM, Fulkerson PK, Barnas GP, cisco, Calif: Jossey-Bass Inc; 1998. clinical trial. Arch Intern Med. 1995;155:1877-1884.
Lawrence SL. Effect of student and preceptor gender 143. Hundert EM, Hafferty F, Christakis D. Charac- 162. Beckman HB, Markakis KM, Suchman AL, Frankel
on clinical grades in an ambulatory care clerkship. Acad teristics of the informal curriculum and trainees ethi- RM. The doctor-patient relationship and malprac-
Med. 1995;70:324-326. cal choices. Acad Med. 1996;71:624-642. tice: lessons from plaintiff depositions. Arch Intern Med.
126. Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pan- 144. Kaiser S, Bauer JJ. Checklist self-evaluation in a 1994;154:1365-1370.
garo LN, Weaver MJ. How accurate are faculty evalu- standardized patient exercise. Am J Surg. 1995;169: 163. Tamblyn R, Benaroya S, Snell L, McLeod P,
ations of clinical competence? J Gen Intern Med. 1989; 418-420. Schnarch B, Abrahamowicz M. The feasibility and value
4:202-208. 145. Gordon MJ. A review of the validity and accu- of using patient satisfaction ratings to evaluate inter-
127. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The racy of self-assessments in health professions train- nal medicine residents. J Gen Intern Med. 1994;9:
mini-CEX (clinical evaluation exercise): a preliminary ing. Acad Med. 1991;66:762-769. 146-152.
investigation. Ann Intern Med. 1995;123:795-799. 146. Westburg J, Jason H. Collaborative Clinical Edu- 164. Helmreich RL. On error management: lessons
128. Dunnington GL, Wright K, Hoffman K. A pilot cation: The Foundation of Effective Patient Care. New from aviation. BMJ. 2000;320:781-785.
experience with competency-based clinical skills as- York, NY: Springer Publishing Co; 1993. 165. Helmreich RL, Davies JM. Anaesthetic simula-
sessment in a surgical clerkship. Am J Surg. 1994;167: 147. Institute of Medicine. Crossing the Quality tion and lessons to be learned from aviation. Can J
604-606. Chasm: A New Health System for the 21st Century. Anaesth. 1997;44:907-912.
129. Dunnington G, Reisner L, Witzke D, Fulginiti J. Washington, DC: National Academy Press; 2001. 166. Schaefer HG, Helmreich RL. The importance of
Structured single-observer methods of evaluation for 148. Gonzi A. The Development of Competence- human factors in the operating room. Anesthesiol-
the assessment of ward performance on the surgical Based Assessment Strategies for the Professions. Can- ogy. 1994;80:479.
clerkship. Am J Surg. 1990;159:423-426. berra: Australian Government Publishing Service; 1993. 167. Davies JM, Helmreich RL. Virtual reality in medi-
130. Keynan A, Friedman M, Benbassat J. Reliability National Office of Everseas Skills Recognition Re- cal training. CMAJ. 1997;157:1352-1353.
of global rating scales in the assessment of clinical com- search Paper # 8. 168. Helmreich RL, Musson DM. Surgery as team en-
petence of medical students. Med Educ. 1987;21: 149. Wass V, McGibbon D, van der Vleuten C. Com- deavour. Can J Anaesth. 2000;47:391-392.
477-481. posite undergraduate clinical examinations: how should 169. Sexton JB, Thomas EJ, Helmreich RL. Error, stress,
131. Kroboth FJ, Hanusa BH, Parker S, et al. The inter- the components be combined to maximize reliabil- and teamwork in medicine and aviation: cross sec-
rater reliability and internal consistency of a clinical evalu- ity? Med Educ. 2001;35:326-330. tional surveys. BMJ. 2000;320:745-749.
ation exercise. J Gen Intern Med. 1992;7:174-179. 150. van der Vleuten C CP, Norman GR, De Graaff 170. Epstein RM. Communication between primary
132. McKinley RK, Fraser RC, van der Vleuten C, Hast- E. Pitfalls in the pursuit of objectivity: issues of reli- care physicians and consultants. In: Suchman AL, Bo-
ings AM. Formative assessment of the consultation per- ability. Med Educ. 1991;25:110-118. telho RJ, Hinton-Walker P, eds. Partnerships in Health
formance of medical students in the setting of gen- 151. Hodges B. Assessment Across Basic Science and Care: Transforming Relational Process. Rochester, NY:
eral practice using a modified version of the Leicester Clinical Skills: Using OSCEs With Postencounter Ba- University of Rochester Press; 1998:171-183.
Assessment Package. Med Educ. 2000;34:573-579. sic Science Probes. 2001. Manual located at the Uni- 171. Frankford DM, Konrad TR. Responsive medical
133. Yang JC, Laube DW. Improvement of reliabil- versity of Toronto, Toronto, Ontario. professionalism: integrating education, practice, and
ity of an oral examination by a structured evaluation 152. Smith RM. The triple-jump examination as an community in a market-driven era. Acad Med. 1998;
instrument. J Med Educ. 1983;58:864-872. assessment tool in the problem-based medical cur- 73:138-145.

2002 American Medical Association. All rights reserved. (Reprinted) JAMA, January 9, 2002Vol 287, No. 2 235

Downloaded From: by a University Of Connecticut Health Center User on 05/14/2015