USMLE Step 2 Clinical Skills

Richard E. Hawkins, MD
National Board of Medical Examiners
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Presentation Structure
Overview of USMLE Step 2 CS

Measures to ensure fairness
and consistency
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Overview of USMLE
Step 2 CS
Purpose and Delivery Model
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE: Purpose
Three step examination system primarily
designed to support the medical
licensing process
Jointly sponsored by
Federation of State Medical Boards
(FSMB)
National Board of Medical Examiners
(NBME)
Step 2 CS: collaboration with ECFMG
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE: Purpose
Overall purpose
Assess physician's ability to apply
knowledge, concepts, and principles, and
to demonstrate fundamental patient-
centered skills that constitute the basis
of safe and effective patient care
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE: Purpose
Overall purpose
Assess physician's ability to apply
knowledge, concepts, and principles, and
to demonstrate fundamental patient-
centered skills that constitute the basis
of safe and effective patient care
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE: Purpose
Patient-centered Skills
History taking / Physical examination
Communication and Interpersonal Skills
Medical Record Documentation

Important for safe and effective patient care
History/PE Diagnosis and Management
Communication Patient Health Outcomes
Medical Record Errors and Patient Safety
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE Step 2 CS
Enhancement to USMLE Step 2
USMLE Step 2 components:
Clinical knowledge (CK)
Clinical skills (CS)
Standard for Step 2 appropriate for
entry into postgraduate training
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE Step 2 CS: Logistics
Regional delivery model
Optimal combination of convenience,
cost-efficiency and standardization
Five regional test centers across US
Projected examine volume 30,000+ / year
Individual center capacity
3 examinations / day (33 examinees); up
to 7 days/week
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Requirements of a High Stakes
Performance-based
Examination
Ensuring Fairness and
Consistency
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Threats to Validity and Reliability
Content and/or tasks not relevant or realistic
Individual test forms vary in content
coverage
Scoring methods not appropriate for skills
tested
Inconsistency in SP portrayal and scoring
Between cases, across sites, over time
Level of difficulty of cases / exams
inconsistent
Standard setting approach appropriate
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE Step 2 CS: Content
Step 2 CS Blueprint
Defines content categories
In meeting blueprint specifications,
each test form provides:
Adequate sampling of content
domain
Comparable content between test
forms
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
USMLE Step 2 CS: Content
Blueprint Content Categories:
Common and important medical problems
/ patient presentations
Acuity
Age
Gender
Race / ethnicity
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Case Content

Cardiovascular
Respiratory
Gastrointestinal
Musculoskeletal
Constitutional
Neurological
Psychiatric
Genitourinary
Womens health
Unclassified / multi-system

Test Form
Patient age

Age less than 18
Age 18 44
Age 45 64
Age 65 +

Patient Gender

Male
Female
Case Acuity

Acute
Subacute / Chronic

Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Step 2 CS: Structure & Tasks
12 patient encounters
15 min. for encounter / 10 min. for patient note
Each encounter:
Elicit pertinent history,
Perform appropriate physical examination,
Communicate effectively
Document:
Findings from the history and physical
Diagnostic impression / Further work-up

Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Step 2 CS: Score Components
Integrated Clinical Encounter (ICE)
Data gathering: history and physical exam
Patient note
Communication / Interpersonal Skills (CIS)
Gathering information; sharing
information; manner & rapport
Spoken English Proficiency (SEP)
Listener effort, examinee pronunciation /
word choice
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Scoring Approach: ICE

Data gathering (Hx / PE)
Dichotomous checklists completed by SPs
+
Patient note
Physician raters using holistic methods
=
Clinical Process + Clinical Outcome Measure
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Case and Checklist Development
Iterative Process
Involves test committees
Focus on clinical presentation
Checklists limited
Encounters with SPs
Appropriateness of content and
difficulty
Stepwise progression through
pilot and calibration stages
Validation / refinement via
review of examinee
performance data

Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Patient Note
Holistic scoring
Rater training
General policies
Case specific:
Consensus development on key
features
Calibration phase
Quality Assurance
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Derivation of Communication /
Interpersonal Skills (CIS) Scale
Initial instrument validated ECFMG
scale

Reviewed against national
consensus recommendations and
commonly used scales
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
CIS Subscales
Data gathering skills
Open-ended questions, transitional statements,
not interrupting the patient
Information sharing skills
Responsiveness to patient questions/concerns,
provision of counseling when appropriate,
avoidance of jargon
Personal manner and rapport
Expression of interest in the impact of the illness,
concern for patient comfort and modesty
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Step 2 CS: Scoring
Reporting is Pass or Fail only
Examinees must pass all three
subcomponents
Feedback to examinees
Performance report overall and
subcomponent outcomes
Failing examinees only graphical
profiles
Intended to show relative strengths
and weaknesses


Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona

USMLE Step 2 Clinical Skills

P/F Outcome

Total Test FAIL
-----------------------------------------------------------------------------------------
Examination Subcomponents

Integrated Clinical Encounter
Pass

Communication/Interpersonal Skills
Fail

Spoken English Proficiency
Pass
Sample Performance Report
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Sample Performance Profile
Integrated Clinical Encounter
Data Gathering
Patient Note

Communication /
Interpersonal Skiills

Spoken English Proficiency

Lower
Performance
Borderline
Performance
Higher
Performance
XXXXXXXXXXXXX
XXXXXXXXXXXXXXXX
XXXXXXXXXXXXXX
XXXXXXXXXXXXX
XXXXXXXXXXXXXXXX
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Threats to Validity and Reliability
Content and/or tasks not relevant or realistic
Individual test forms vary in content
coverage
Scoring methods not appropriate for skills
tested
Inconsistency in SP portrayal and scoring
Between cases, across sites, over time
Level of difficulty of cases / exams
inconsistent
Standard setting approach appropriate
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Ensuring Fairness and Consistency
Rigorous SP (and SP trainer) training
Meticulous attention to quality assurance
Application of equating procedures
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
SP and SP Trainer Training
Standardized Patients
Rigorous training
generic and case
specific
Electronic delivery
(Ecase) of case
materials
Sign off process for
SPs Criteria : #
portrayals, tests
SP Trainers
Adherence to training
protocols
Training academy for
SP trainers
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Quality Assurance
Begins with the SP sign-off process

Monitoring procedures and analyses:
Qualitative (portrayal and scoring
accuracy):
Live and video review of SP
performances
Quantitative
Score-based analyses
Case level and item level comparisons

Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Equating Procedures
Within site
SP-case combination
Between site
Central video review
Data gathering
Communication and Interpersonal Skills
Patient note rater case combination
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Standard Setting
USMLE Standard Setting System
Committee assigned responsibility to
establish and monitor standards
Decision-making process
Survey of constituent opinion
Standard-setting exercises by
independent groups
Examinee performance data and score
reliability
Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Conclusions
Numerous logistical and psychometric
challenges were identified and responses
described
In order to ensure exam fairness and
consistency:
Sound test and case development practices
Intensive SP and SP trainer training
Rigorous quality assurance
Well-considered equating and standard setting
procedures
Supported by ongoing research on reliability
and validity

Presented at the 2005 CLEAR Annual Conference
September 15-17 Phoenix, Arizona
Speaker Contact Information
Richard E. Hawkins, MD
National Board of Medical Examiners
3750 Market Street
Philadelphia, PA 19104
215-590-9204 / Fax 215-590-9440
Email: rhawkins@nbme.org
Websites:
http://www.usmle.org
http://www.nbme.org