Comparison: Traditional vs.

Outcome Project Evaluative

Processes
Craig McClure, MD
Educational Outcomes Service Group
University of Arizona
December 2004

Current Problem
Increasing public concerns with quality
and safety.
Variable patterns of care that are not
based on medical science.
Poor quality of interpersonal service.
Public encounters difficulty in assessing
physician competence (initial and
continuing ) and judging quality.

The ACGME Mission

To improve the quality of health
care in the United States by
ensuring and improving the
quality of graduate medical
educational experiences for
physicians in training.

Problem Plus Mission

ACGME responded to the challenge by
changing focus to:
How well do we learn what is being taught

How well do we practice what we learn?

A new way of thinking

How to change the educational
and accreditation system from

Structure &
process

Competency

Program Goal

OLD: goal was for NEW: the Program

the Program to
Director must
comply with the
determine if
written RRC
residents achieve
Requirements
the learning
objectives set by
the Program.

Six Domains

Medical Knowledge
Patient Care
Professionalism
Communication and Interpersonal
Practice Based Learning and
Improvement
Systems Based Practice

Purpose of Assessment
1. Assess residents' attainment of
competency-based objectives
2. Facilitate continuous improvement
of the educational experience
3. Facilitate continuous improvement
of resident performance
4. Facilitate continuous improvement
of residency program performance

Whatever we measure we tend to

improve.
David C. Leach, M.D.
Executive Director
ACGME
September 12, 2002

Characteristics of good
assessment
Measures actual performance
Identifies areas for improvement
Satisfies reasonable request for
accountability
Is practical
Is done over time to discern growth

Types of Evaluation
Formative
Improve performance

Summative
Note achievement

Both types of evaluation can be used to

evaluate either an individual or a program.

Characteristics of good assessment

Systematic
Dependable
Comprehensive
Congruent
Practical

Characteristics of good assessment

(continued)

Makes professional practice more

transparent
Deconstructs the role of physician
Clarifies levels of expertise by
distinguishing functional levels

Characteristics of good assessment

(continued)

Measures actual performance

Identifies areas for improvement, i.e.,
self, others
Satisfies reasonable requests for
accountability

Traditional Evaluation
1.Global
2.End of rotation
3.Subjective
1.Anchored to norms seen by attending
(therefore variable)
2.I like/didnt like the resident

4.Focused on rotation goals (not

movement toward competency)

Outcome Based Evaluation

1. Formative, focused on specific
competencies required for a physician
2. Measure the full scope of professional
characteristics from very specific
procedures to skills involving a synthesis
of component abilities
3. Specific evaluative techniques chosen to
match the skill being assessed

Assessment Tools
(The Toolbox)
360 Evaluation Instrument
Chart Stimulated Recall Oral Exam
(CSR)
Checklist Evaluation of Live or
Recorded Performance
Objective Structured Clinical Exam
(OSCE)
Procedure, Operative or Case Logs

The Toolbox (continued)

Patient Surveys
Portfolios
Record Review
Simulations and Models
Standardized Oral Exams
Standardized Patients (SP)
Written Exams (MCQ)

Evaluation Method

OLD: global
checklist format

NEW: Type of
evaluation chosen
specifically to
measure the
chosen skill drawn
from the 6 domains

Frequency of Evaluation
OLD: once per
rotation

NEW: multiple
intervals
assessing
component
behaviors as well
as the integrated
practice of
medicine.

Timing of Assessment
OLD: End of
rotation

NEW: Timing
chosen to facilitate
evaluation of a
specific
competency

Anchors for Evaluation

OLD: Most
frequently the
preceptor
evaluated the
resident against
the norm of
previous
residents in that
experience

NEW: Criteria
defining
competence are
utilized as the
standard against
which resident
performance is
measured

Target of Evaluation

OLD: at best
tended to
address the
residents
success at the
goals for the
rotation

NEW:Criteria for
evaluation
describe the
qualities of the
competent
physician, so are
more wide
ranging or more
specific

Number of Evaluators
OLD: typically one
per rotation

NEW: multiple,
both physician and
non-physician
evaluators

Other Outcome Characteristics

Authentic
More Individualized
Reflection and Self-knowledge
Critical

Authentic
Justification for elements included in
the curriculum is that competence as
a practicing physician requires that
skill, knowledge or attitude
Evaluation is of the actual skill,
knowledge or attitude used by
practicing physicians

More Individualized
A principle of a criteria-driven physician
curriculum is that everyone can become
competent with sufficient exposure
Residents obtain skills at different rates
with requirements for disparate learning
experiences
An optimal outcome-driven system would
have an intake assessment followed by an
individualized program of study

Reflection and Self-knowledge

Critical
Criteria for competence are provided
to the learner
Impetus for improvement arises from
desire to narrow the gap between
criteria and performance
Accurate self-assessment is
essential to the resident gauging
personal performance

In Summary
Traditional method:
Not systematic
Subjective & Normative based
Global evaluations @ rotation end

Outcomes-based:
Systemic and comprehensive
Based on criteria defining competence
Multiple measures and intervals