Medical Teacher

ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: http://www.tandfonline.com/loi/imte20

Competency milestones for medical students:

Design, implementation, and analysis at one
medical school

Kimberly D. Lomis, Regina G. Russell, Mario A. Davidson, Amy E. Fleming,

Cathleen C. Pettepher, William B. Cutrer, Geoffrey M. Fleming & Bonnie M.
Miller

To cite this article: Kimberly D. Lomis, Regina G. Russell, Mario A. Davidson, Amy E. Fleming,
Cathleen C. Pettepher, William B. Cutrer, Geoffrey M. Fleming & Bonnie M. Miller (2017)
Competency milestones for medical students: Design, implementation, and analysis at one medical
school, Medical Teacher, 39:5, 494-504, DOI: 10.1080/0142159X.2017.1299924

To link to this article: http://dx.doi.org/10.1080/0142159X.2017.1299924

View supplementary material

Published online: 10 Mar 2017.

Submit your article to this journal

Article views: 501

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=imte20

Download by: [New York University] Date: 08 June 2017, At: 06:40
MEDICAL TEACHER, 2017
VOL. 39, NO. 5, 494–504
http://dx.doi.org/10.1080/0142159X.2017.1299924

Competency milestones for medical students: Design, implementation, and

analysis at one medical school
Kimberly D. Lomisa, Regina G. Russella, Mario A. Davidsonb, Amy E. Flemingc, Cathleen C. Pettephera,
William B. Cutrerd, Geoffrey M. Flemingd and Bonnie M. Millere
a
Office of Undergraduate Medical Education, Vanderbilt University School of Medicine, Nashville, TN, USA; bDepartment of Biostatistics,
Vanderbilt University School of Medicine, Nashville, TN, USA; cOffice of Medical Student Affairs, Vanderbilt University School of Medicine,
Nashville, TN, USA; dDivision of Pediatric Critical Care, Vanderbilt University Medical Center, Nashville, TN, USA; eOffice of Health
Sciences Education, Vanderbilt University School of Medicine, Nashville, TN, USA

ABSTRACT
Competency-based assessment seeks to align measures of performance directly with desired learning outcomes based upon
the needs of patients and the healthcare system. Recognizing that assessment methods profoundly influence student motiv-
ation and effort, it is critical to measure all desired aspects of performance throughout an individual’s medical training. The
Accreditation Council for Graduate Medical Education (ACGME) defined domains of competency for residency; the subse-
quent Milestones Project seeks to describe each learner’s progress toward competence within each domain. Because the
various clinical disciplines defined unique competencies and milestones within each domain, it is difficult for undergraduate
medical education to adopt existing GME milestones language. This paper outlines the process undertaken by one medical
school to design, implement and improve competency milestones for medical students. A team of assessment experts devel-
oped milestones for a set of focus competencies; these have now been monitored in medical students over two years.
A unique digital dashboard enables individual, aggregate and longitudinal views of student progress by domain. Validation
and continuous quality improvement cycles are based upon expert review, user feedback, and analysis of variation between
students and between assessors. Experience to date indicates that milestone-based assessment has significant potential to
guide the development of medical students.

Introduction
Competency-based medical education seeks to directly link
the outcomes of a learning program with the needs of
patients (Frank et al. 2010a). Competencies are trainable
attributes of an individual that must be developed in order
to successfully perform professional duties, and these are
often grouped within domains encompassing similar skills.
This manuscript draws upon the common taxonomy of
competencies recommended by Englander et al. (2013).
Milestones describe the typical developmental pathway for
a given competency (Nasca et al. 2012). Milestones are
expressed using behavioral anchors that describe escalating
levels of performance from novice (“entry”) to aspirational
(Dreyfus 2004) (see Figure 1). These milestones enable the
learner to recognize current status and to understand what
behaviors are necessary to achieve higher levels of per-
formance (Schumacher et al. 2013; Swing et al. 2013).
Milestones also provide guidance to assessors, generating a
shared mental model across disciplines regarding the
knowledge, skills and behaviors expected of learners.
Competency-based assessment does not seek to ensure
only minimal competence, but rather promotes each
learner’s trajectory toward excellence across multiple
domains of performance (Carraccio & Englander 2013;
Frank et al. 2010b). It is advantageous to introduce stu-
dents to the competency domains and milestones early in
training and to emphasize the need for continual
Practice points

An assessment program should be designed to
promote all desired learning outcomes.

Milestones serve as a standardized articulation of
expectations, providing clarity for both assessors
and learners, and are useful to describe learner
development across competency domains.

A portfolio approach involving professional judg-
ment in the review of performance evidence miti-
gates against the multiple confounders of a
complex system.

To advance students’ future learning efforts,
meaningful assessment evidence from multiple
settings should be integrated and targeted learn-
ing opportunities should be provided.

Extensive collaboration and training, as well as
continual critical appraisal of outcomes, is neces-
sary to successfully implement a competency-
based assessment system.
development throughout one’s medical career. The incorp-
oration of more active and team-based learning activities
in the pre-clerkship curriculum enables demonstration
of domains of competency that were previously under-
emphasized at early stages of training. As a result, the

CONTACT Kimberly D. Lomis kim.lomis@vanderbilt.edu Office of Undergraduate Medical Education, Vanderbilt University School of Medicine, 201
Light Hall, Nashville TN 37232, USA
Supplemental data for this article can be accessed here.
ß 2017 Informa UK Limited, trading as Taylor & Francis Group
 MEDICAL TEACHER 495

Figure 1. Example of milestone rating form and display.

competency domains are now relevant throughout all of Design

undergraduate medical education and align with expecta-
Beginning in 2012, a committee of faculty members with
tions in graduate medical education and beyond (Anderson
experience teaching learners of all levels began drafting
2012).
milestones specific to medical students. To focus our initial
To foster learner development, the assessment process
implementation, the team defined core behaviors that
must inform future learning. The framework of milestones
should be continually re-assessed throughout a student’s
is best utilized to describe competencies that develop over
training. This selection was based upon faculty judgment of
time and must be demonstrated repeatedly throughout
relative importance; priorities represented in multiple exist-
one’s career in increasingly complex settings (Krupat &
ing assessment forms; consideration of areas in which stu-
Pelletier 2016). Traditional normative assessments are lim-
dents historically struggle; and areas amenable to
ited in this regard; performance ratings relative to one’s
measurement starting in the first year. A modified Delphi
peers provide little guidance about specific behaviors that
technique (Clayton 1997) was employed to create consen-
are desired or areas for improvement. Milestones are criter-
sus around priority competencies.
ion-referenced, offering a common set of performance
Work groups for each domain of competency were pro-
benchmarks applicable across multiple courses and set-
vided background materials, including internal grading
tings. This enhances the identification of trends, providing
forms from multiple courses and graduate medical educa-
stronger evidence to prompt the student to take action.
tion (GME) milestones published at that time (medicine,
Our competency milestones were designed to continually
pediatrics, and emergency medicine). Entry-level GME
populate each learner’s digital portfolio (Chertoff et al.
expectations informed undergraduate medical education
2016). This performance evidence supports a cyclical
(UME) graduation targets (Swing 2007).
informed self-assessment process in which learners, with
The following guidelines were provided to milestone
coaches, identify relative strengths and weaknesses and
writing teams:
create appropriate learning plans.

Create behavioral anchors that describe readily observ-

Methods
Design and implementation involved several steps. An
existing assessment committee was tasked with drafting
and improving milestones language. Initial implementation
focused on two cohorts of assessors who were trained
extensively, were given protected time for student assess-
ment, and shared experiences at monthly faculty develop-
ment meetings. An assessment sub-committee was charged
with the continual review of outcomes from the milestones
system using both quantitative and qualitative approaches.
able actions (Regehr et al. 2012)

Avoid normative language: “better than average” or “as
expected for level”

Avoid vague language: “good” or “fairly accurate”

Create five (5) anchors for all competencies

The first level should reflect expected behaviors at entry
to medical school; fifth is aspirational (high-performing
intern at 3 months)

Lower-level anchors should not simply describe the
absence of behaviors associated with higher levels, but
rather should describe typical early behaviors as part of
496 K. D. LOMIS ET AL.
a normal developmental progression (Holmboe &
Batalden 2015).
Anticipating the potential need for promotions commit-
tees to render definitive decisions regarding unacceptable
performance trends in some students, a separate process
later defined undesirable behaviors as distinct from devel-
opmentally expected, lower-level behaviors.
Several iterative rounds of editing were completed.
Work groups critiqued each other’s language and chal-
lenged assumptions about typical developmental steps.
Stakeholder input from course directors, potential assessors,
and students was solicited at multiple points.
Early pilots were executed to enhance construct align-
ment (Crossley et al. 2011). Based upon feedback, the
assessment committee made changes to the wording of
some anchors. An option to indicate an intermediate level
of performance between anchors was added, indicating
that the student intermittently displayed the higher level.
The initial layout of the milestone form for each compe-
tency mimicked the charts used in GME, which include mul-
tiple behaviors within each level of performance (Swing
et al. 2013). Assessors found that some of the grouped ele-
ments develop independently; therefore, each competency
was separated into more discrete units.
Having completed these refinements, the assessment
committee resolved to avoid editing the milestones lan-
guage for two years after implementation. Accumulated
outcomes data would provide insight into the performance
of the milestones for each competency, which would guide
subsequent revisions.
Implementation
The competency milestones were implemented with the
onset of a new curriculum in 2013. Implementation was ini-
tially limited to two faculty assessor groups with existing
responsibility for direct observation of students. These fac-
ulty cohorts had protected time for student interactions
and attended standing monthly faculty development meet-
ings. One cohort was based in the pre-clerkship phase as
facilitators in case-based learning sessions (SGFs) and spent
6 hours per week in a classroom setting with rotating
groups of 8 first year students. The other assessor cohort,
master clinical teachers (MCTs), was based in the clerkship
phase to provide direct observation in the workplace
regarding skills in history, physical exam, and clinical rea-
soning. Each student spent at least 12 hours interacting
with various MCTs over the clerkship year.
Students were introduced to the new assessment pro-
cess in a pre-matriculation online course, and received add-
itional training at orientation, in town halls, and during
practice sessions. First year students were trained to use
the milestones for peer evaluation, which served to solidify
the student awareness of competency expectations.
Each course or program was asked to select only those
competencies most applicable to the specific setting. No
course was expected to utilize all of the established mile-
stones. Course directors were not, however, permitted to
change the milestone anchor language; when a given com-
petency was being evaluated in different settings, the
standardized performance descriptors were maintained.
A customized assessment form was generated for each
course, including metadata regarding contextual informa-
tion such as setting, assessor, and level of difficulty.
Assessments were captured in a digital learning portfolio
designed to facilitate the collation and visualization of data
(Spickard et al. 2016).
A developmental framework of assessment is intended
to direct future learning; therefore, the process must have
meaningful consequences for the student. The milestones
serve as one source of performance evidence supporting a
cyclical, faculty-facilitated self-assessment process. At pre-
scribed intervals, the student reviews evidence with a port-
folio coach to identify relative strengths and weaknesses
and develop appropriate personalized learning plans.
Coaches support students in setting and monitoring goals
to improve across competency domains.
The student promotions process was redesigned to
include the review of competency milestone evidence in
addition to course grades. All domains are tracked from the
beginning of a student’s medical education. By using the
same metrics across courses, trends in performance can be
identified more readily. Given the context-specificity of
many skills (Linn et al. 1991), we did not envision that each
learner would make linear progress in each competency;
rather we expected fluctuating performance given various
settings and complexities, with general improvement over
time (see Figure 2). Multiple milestone data points can pro-
vide a signal about areas of excellence and areas for
improvement, but no single rating event is considered high
stakes for a student. There was deliberate avoidance of cal-
culating average scores. Indeed, numeric reference to mile-
stone levels within the digital platform was discouraged,
with the intent to focus on the words in each behavioral
anchor. It was deemed essential that the student and coach
interpret the relative merits and import of each data point.
Trends identified after such critical review did trigger action
by the promotion committee; most commonly this involved
a targeted learning plan directed at the competency
domain of concern.
Analysis
The milestones are intended to measure medical student
performance in core competency domains and to provide
developmentally-based feedback. It is critical that the mile-
stones represent the domains of interest and that the
anchors accurately describe the developmental trajectory of
these learners. The assessment tool should be “fit for
purpose,” valid for the intended application (van der
Vleuten & Schuwirth 2005). Since individual performance
does not occur in a vacuum, there is inherent complexity in
assessing competency development. Ongoing variations in
the environment, context, and level of difficulty will result
in fluctuations of performance ratings for an individual; this
creates challenges for validation of the milestones.
Quality improvement and validation of the milestones
has been an iterative process involving evaluation of con-
tent, the impact of rater variation, and qualitative evidence
of feasibility and user satisfaction (Hicks et al. 2010a, 2010b;
Englander et al. 2012). Messick’s (1989) approach to validity
criteria for educational testing provided guidance to these
efforts. He conceptualized validity as a holistic summary of

Figure 2. One student’s performance evidence for multiple competencies in one domain displayed in the digital portfolio.
the evidence for using a particular test as well as the inter-
pretations of test scores relative to possible consequences.
Messick reminds us that validity claims must engage the
entire assessment process, not only the item structure.
In the “post-psychometric era” (Hodges 2013), medical
education assessment experts are much more aware of the
“conceptual and practical challenges” (Whitehead et al.
2015) in the measurement of competencies and validation
of performance rating systems in unstandardized, real-
world contexts (Boursicot et al. 2011). Approaches that rec-
ognize the importance (and statistically confounding
nature) of the “patient-by-doctor-by-context interactions”
(van der Vleuten 2014) are critical to understanding the
development of complex competencies such as communi-
cation, professionalism, and systems-based practice:
The new understanding that we are currently documenting is
that assessors are not passive, perfectly calibrated measurement
instruments, but active agents constructing judgments using
information from situational factors and personal experiences.
From this perspective, different assessors are not expected to
make similar judgments, and this may actually be desirable!”
(pg. 235)
Downing and Haladyna (2004) use a medical education
context to highlight the two most significant threats to
construct validity outlined by Messick: construct under-rep-
resentation and construct irrelevant variance. Rater vari-
ation is often categorized as “construct irrelevant variance.”
However, the authors note statistical difficulties in differen-
tiating rater error from rater variation that is appropriate to
particular situations and contexts. Additionally, new
approaches to assessing competencies through big data
and learning analytics (Pecaric et al. 2017) and qualitative
methods from the social and behavioral sciences
(Whitehead et al. 2015) hold promise for improving our
MEDICAL TEACHER 497
understanding of the best approaches to ensure high-qual-
ity measurements of the intended constructs and appropri-
ate, meaningful interpretation of performance data.
Incorporating insights from traditional psychometric and
post-psychometric views, our data collection and analysis
follows a holistic approach. We have incorporated both
quantitative and qualitative methods to investigate the
impact of milestone ratings in the complex system of med-
ical education (Baartman et al. 2007). Quantitative analysis
of student performance on the assessments as well as
informal focus groups and surveys were used to assess the
impact and function of the milestones language.
Descriptive statistics were calculated, and graphic visual-
izations were created to consider the variability in the
aggregate distribution of milestone ratings for each compe-
tency in relation to the medians. These data were consid-
ered longitudinally, and from the perspective of individual
students as well as individual assessors. During this explora-
tory analysis of the milestones, statistical inference and
analysis was not conducted. Variability was the main con-
sideration in investigating rater discrimination and was indi-
cated by reviewing the interquartile range. Atypical
patterns in the distributions of milestone levels assigned,
such as minimal variability or extreme outliers, were
explored in discussions with raters. Stakeholder input for
milestone improvement was driven by quantitative evi-
dence and remains critical for the continued development
and effective use of the new milestone language.
Results
Medical students enrolled from 2013 to 2015 were
assessed using the milestone framework. First year students
underwent approximately 36 milestone-based primary
498 K. D. LOMIS ET AL.
Figure 3. Distribution of all students in one phase of training on one competency.
Figure 4. Student performances as compared to predicted benchmark for one competency.
assessments from peers and faculty involving 3–18 compe-
tencies depending on context. In the clerkship phase (year
2), students received approximately 14 milestone-based
assessments from faculty, with 13–20 competencies repre-
sented. In 2015, the assessment committee conducted an
intensive review of the milestone results at the level of
individual students, individual assessors, and aggregate
results within and across levels of training. Qualitative feed-
back was collected from all users (students, faculty, and
administrators).
Validity evidence
Several trends in the data affirm the validity of the
milestones.
Aggregate data regarding the performance of learners at
the same level of training for each competency
demonstrate a mixture in milestone levels assigned (see
Figure 3). This suggests that the milestones elucidate differ-
ences in development among students.
The committee that drafted the milestones predicted
levels of performance that students at various stages of
training would attain. For some competencies, early stu-
dents reached higher levels of performance than predicted
(first year students were assessed in a controlled classroom
setting for which they had significant time to prepare).
Aggregate data did confirm, however, that performances
below predicted benchmarks were low as compared to the
peer group (see Figure 4).
Cross-sectional comparisons of students at different
stages of training showed an anticipated shift to higher
milestone achievements as learners advance temporally
(see Figure 5). This is suggestive that the milestone anchors
do describe developmental progression.

Figure 5. Cross-sectional view of two different classes on one competency shows generally higher ratings for the more senior class.
Figure 6. Example of a faculty member’s ratings of a single student in an observed patient encounter regarding specific aspects of the history and physical
exam.
Review of the performance ratings for individual stu-
dents across competencies revealed variations; a student
does not receive similar performance ratings in all compe-
tencies (see Figure 6). This indicates that milestone-based
feedback is discriminating for discrete aspects of
performance.
Rater variation
It is possible that the aggregate milestones distributions
could be attributable to rater effect rather than true differ-
ences in student performance; therefore, variation in rater
results was investigated.
Faculty assessors
Box plots and bar graphs were created for each compe-
tency to show the ratings used by all assessors within an
assessor cohort (see Figure 7). Upon reviewing this for
MEDICAL TEACHER 499
multiple competencies, very few assessors were identified
as consistently grading low or high across all competencies.
Box plots demonstrate that most raters applied a mix of
milestones levels to describe the performance of a group
of students in a given competency. Additionally, each
assessor used multiple milestone levels to describe each
individual student’s performance, indicating that the mile-
stone language is discriminating for discrete aspects of
performance.
Peer assessors
Student ratings of peer performance also show a diversity
of milestone levels when describing a given student’s per-
formance. This indicates that the milestone language is dis-
criminating for discrete aspects of performance.
In the pre-clerkship small groups, both faculty and peers
rate students based upon interactions in the case-based
learning sessions. The aggregate distribution of milestone
500 K. D. LOMIS ET AL.

Ratings by Master Clinical Teacher for Interpersonal and

Communication Skills: Rapport with Patients and Families
Aspirational
Milestone Level

Entry

Undesirable

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Master Clinical Teacher ID

Figure 7. Example of comparative assessor milestone ratings data provided to Master Clinical Teachers as part of faculty development.

Figure 8. Distribution of class milestones ratings by faculty and by peers for one competency.

levels from peer and faculty evaluations is generally similar,
indicating a shared interpretation of the milestone lan-
guage (see Figure 8). For certain competencies, some differ-
ences between faculty and peer patterns occur; review of
associated qualitative comments indicates that the students
have unique insights into these aspects of performance
based upon interaction outside of class.
Each student is evaluated by two peers at each review
point. Generally, milestone levels align between the two
raters, or differences are clarified via comments. In one situ-
ation of a known personal conflict between two peers,
these students limited the use of low performance ratings
to the specific competency of concern and rated each
other more highly in other aspects of performance.
User feedback
Rising second and third-year medical students who
received milestone-based performance feedback were sur-
veyed in September 2015 regarding the assessment system
and the portfolio review process. Responses on a five-point
scale to questions regarding user understanding of mile-
stones and their influence on learner development trended
in the range of neutral to agree (see Table 1). It is apparent
from comments that some students had a positive experi-
ence using milestone evidence in the portfolio process,
while others struggled to make sense of the feedback and
use it for improvement. In addition to potential limitations
of the milestone language and lack of familiarity with this
novel criterion-based approach, this variation in meaning-
fulness may have been influenced by characteristics of the
student, the portfolio coach, and their interaction.
Formal focus groups were conducted with students,
portfolio coaches, and promotions committee members.
Results were similarly mixed across participants: some
found the milestone feedback important to student devel-
opment, others doubted its effectiveness and/or found it
overly cumbersome. Students expressed concerns about
the quantification of milestone-based feedback and
whether “scores” were a valid measure of different perform-
ance dimensions.
 MEDICAL TEACHER 501

Table 1. Sample of survey results from students using milestones.

Feedback from Medical Students About Competency Milestones
1-strongly disagree, 2-disagree, 3-neutral, 4-agree, S-strongly agree
183 survey respondents
% SD þ D % Neutral % A þ SA Median
I understand the milestones language. 21% 26% 53% 4
Feedback using milestones helps me track my development. 23% 37% 40% 3
Selected Student Comments:

I think it is good to be cognizant of your strengths and weaknesses, and the portfolio process has helped create an environment in which you strive
to improve as well as to foster humility. 1 think it is wonderful to have a portfolio coach- mine has helped me get in contact with mentors, has good
study tips, is a good point of contact for questions, and 1 think genuinely cares about my success.

The portfolio process has really allowed me to see how my peers and faculty in nontraditional roles (i.e. not directly providing grades) view my per-
formance. This has been helpful in changing the way 1 interact with groups and growing as a student.

The milestones and ratings seem very arbitrary and there is not enough variation in the middle ratings. The low scores are very low and almost
unrealistic, it seems like there should be more middle ground.

The portfolio milestone system has many advantages and a lot of potential. However, it currently struggles from evaluators having “preconceived”
notions of where a student is supposed to fall on the number scale at a current point in his or her education such that growth can be demonstrated.

Standing faculty development meetings were used to
collect qualitative feedback from faculty assessor groups.
Feasibility concerns included the total length of the assess-
ment forms and the time needed for completion. This issue
improved as assessors became more familiar with the mile-
stone language, and as the digital interface was refined.
Specific milestone anchor language was also a subject of
discussion among assessors.
Faculty members noted that the quality of peer feed-
back improved compared to prior models at our institution.
The milestone anchors prompted more specific comments
regarding both strengths and weaknesses. Some small
group facilitators anecdotally noted that students utilized
language from milestone anchors when providing verbal
feedback to peers during group debriefings.
For some competencies, patterns in aggregate outcomes
data differed from what was expected (see Figure 9). This
triggered additional qualitative analysis in informal focus
groups. Stakeholders were presented the quantitative
results and were provided the milestone language, then
asked to consider potential contributing factors including:
limitations in the anchor language, limitations of the assess-
ment setting, and possible true differences in developmen-
tal patterns for some competencies. These discussions
determined the appropriate improvement response. Based
upon evidence, a revised milestone menu was endorsed in
the fall of 2015 (see Supplemental materials).
Discussion
Reviewing one medical school’s approach to the design
and implementation of competency milestones, and per-
forming analysis of two years of resultant data, is instruct-
ive regarding the effort needed to ensure a practical, useful
and valid structure. Preliminary findings indicate that it is
feasible to create standardized milestone anchors to be
applied across settings. Generally, the language of the mile-
stone anchors is meaningful to students and to faculty
members. Aggregate and individual student data regarding
achievement of these milestones affirm that they distin-
guish relative strengths and weaknesses within the individ-
ual and among students.
The critical analysis of milestone performance over two
years’ experience has generated enough confidence to
include this evidence in formal reviews of student progress.
In the past, a specific performance issue may have been
overlooked because there were insufficient data within the
context of any given course to prompt remediation. The
ability to collate measures across settings allows identifica-
tion of trends in performance indicating a need for
improvement. Performance challenges in professionalism,
interpersonal communication skills, and systems-based
practice have been identified for some students in the pre-
clerkship phase, enabling coaching and improvement
before these students progressed into clerkships.
Additionally, a subset of clerkship students were identified
who benefited from targeted coaching in diagnostic skills.
To ensure the equitable use of milestone evidence, the
portfolio review process is designed to mitigate against
major action over a single data point. It is challenging to
determine whether variations in milestones data are due to
real differences in student performance, are primarily
related to changing contexts, or are artifacts of rater vari-
ation. However, this mimics the reality of clinical practice;
students must demonstrate competency across all domains
in many different settings and to various supervisors, peers
and patients. Students are instructed to view all feedback
as valuable information emanating from various perspec-
tives; but through the coaching and review mechanism
they can argue that a particular rating is not indicative of
their overall performance.
Aggregate milestone outcome data also provide insights
into the performance of the curriculum. Results for a class
of students indicate whether desired competency outcomes
are generally being attained. This is useful to drive curricu-
lar improvements and to monitor potential impacts of cur-
ricular changes. As an example, our recent curricular
revision included a shift in the timing of core clerkships
from year three to year two. Aggregate student perform-
ance in history and physical examination at the end of the
clerkship phase for the first transition class was comparable
to historical benchmarks, indicating that the acceleration of
the core clinical experience was successful for those
measures.
Future directions
Our initial implementation of the milestone framework was
deliberately limited to controlled cohorts of assessors with
protected time who contributed actively to the
502 K. D. LOMIS ET AL.
Figure 9. Example of comparison of aggregate milestone rating distributions.
development and improvement of the milestones. The util-
ity of the milestones for clinical teams assessing students
within the context of active care delivery remains to be
determined. Challenges regarding training of assessors and
issues of feasibility given time pressures in the clinical
workplace must be addressed. We have partnered with a
clinical department with a strong track record of providing
meaningful student feedback in order to pilot the imple-
mentation of the milestones language with frontline clinical
faculty and resident assessors.
Through our analysis, we are increasingly aware of the
importance of the process we established around the util-
ization of the milestones as evidence of performance. The
structured, faculty-facilitated self-assessment cycles promote
thoughtful interpretation of the milestone results for each
learner; the learner and coach weigh the value of each
data point. This helps to control for error in the system,
and optimizes the application of this evidence into an
actionable learning plan.
Ongoing improvements will strengthen the meaningful-
ness of the milestones. We plan to focus on how these
data are used to direct future learning. Qualitative results
indicate there is still work to be done to make the mile-
stone framework most effective for student development.
Additional training for students has been instituted. Many
personalized learning goals can be met within the context
of existing experiences; some may require special learning
opportunities. We have had anecdotal success with tar-
geted remediation plans. High performers must be offered
the ability to “level up,” aligning advanced activities with
their competency progression. The high degree of flexibility
and individualization in our revised curriculum puts such
accommodations within reach. After ensuring the quality of
longitudinal milestone data, we envision the potential for
advancement through our educational program based
upon competency rather than time.
The initial milestone process at our institution has been
successful, but will require critical oversight and ongoing
refinement. Extensive, continual training for faculty and stu-
dents remains critical. Communication with stakeholders to
address the practicalities of implementation has promoted
sustainability. Critical, data-driven appraisal of our outcomes
will help avoid unintended consequences. Confounders,
such as context and inter-rater variation, must be monitored
and considered in any analysis or application of the mile-
stone data. Most importantly, our team actively acknowl-
edges and addresses limitations in the model, resulting in a
transparent process of continual improvement.

Conclusions
Based on the experience at one medical school, the imple-
mentation of competency milestones for medical students
is feasible and useful; it is also resource intensive and
requires ongoing training, validation and improvement. A
committed group of thoughtful educators was essential to
development of the program, and collaboration in an envir-
onment of mutual respect generated a shared sense of
ownership. Protected time for assessors and coaches has
contributed to the quality of performance evidence and its
application to future learning. Pilots before full implementa-
tion, and critical analysis of outcome evidence, have
informed revisions. Significant resources were dedicated to
technological solutions for data capture and display that
enhance the feasibility and effectiveness of this framework.
Ongoing quality assurance and agility of thinking enables
alignment between educational theory and the challenges
of real world implementation. Our results demonstrate that
a competency-based framework can provide actionable
information for learners, thus enabling achievement of
desired educational outcomes.
Acknowledgements
The authors acknowledge Stan Hamstra, PhD for his helpful review
and feedback on a draft of this paper.
The content reflects the views the authors and does not necessarily
represent the views of AMA or other participants in this Initiative.
Disclosure statement
The authors for this article have no interests to report.
Funding
This publication was prepared with financial support from the
American Medical Association (AMA) as part of the Accelerating
Change in Medical Education Initiative.
Notes on contributors
Kimberly D. Lomis, MD is associate dean for undergraduate medical
education, professor of surgery, and professor of medical education
and administration at Vanderbilt University School of Medicine. Her
scholarly interests focus around curricular innovation, competency-
based assessment, and change management.
Regina G. Russell, MEd, MA is the director of learning system out-
comes for the MD program at Vanderbilt University School of
Medicine. Her higher education experience includes student services
administration, sociology and social work education, learner assess-
ment and program evaluation. Current work includes measurement of
medical student outcomes and the impact of organizational environ-
ments on learning.
Mario Davidson, PhD, MA (Mathematics Education), MS (Statistics),
MAS (Mathematics) is a biostatistician for Vanderbilt University Medical
Center. In his career, he has worked in medical and statistical educa-
tion as an instructor and researcher. He is currently the course director
for the Statistical Collaborator course presented in the Department of
Biostatistics.
Amy E. Fleming, MD, MHPE is the associate dean for medical student
affairs at the Vanderbilt University School of Medicine and associate
professor of pediatrics, division of hospital medicine at Monroe Carell
Jr Children's Hospital at Vanderbilt. Her research interests span
MEDICAL TEACHER 503
learning communities in medical education, portfolios, learning goals,
and mentoring.
William B. Cutrer, MD, MEd is the assistant dean of undergraduate
medical education, and associate professor of Pediatrics, Critical Care
Medicine at Vanderbilt University School of Medicine. He has published
and presented widely on how students learn in the workplace and
how to help them more effectively.
Geoffrey M. Fleming, MD FAAP is a Pediatric Intensive Care Unit phys-
ician with a history of education pursuits including Program Director
for an ACGME accredited fellowship training program and Vice
President for Continuous Professional Development in an ACCME
accredited institution. He has been involved extensively in education
administration leadership at both the local and national level through
workshop development, course administration, educational design
committees and advocacy.
Cathleen C. Pettepher, PhD, is a Professor of Cancer Biology and
Medical Education and Administration and Assistant Dean of Medical
Student Assessment at Vanderbilt University School of Medicine. Her
scholarly interests have focused on designing and implementing cur-
ricular innovations and the development, implementation and evalua-
tions of value of peer assessments in professional growth and identify
formation.
Bonnie Miller, MD, MHHC is senior associate dean for health science
education at Vanderbilt University School of Medicine and Executive
Vice-President for Educational Affairs at Vanderbilt University Medical
Center, Professor of Medical Education and Administration and
Professor of Clinical Surgery. Her scholarly interests include curriculum
innovation and the factors that influence the professional formation of
physicians.
Glossary
Competency-based assessment (CBA)
Orients the assessment of learner performance to graduate out-
come abilities. It is organized around competencies derived
from an analysis of societal and patient needs. Ideally, a major
portion of the assessments should be performed in the context
of the clinical workplace and should be criterion-referenced.
The assessment process is designed to facilitate the develop-
mental progression of competence.
Adapted from:
1. The role of assessment in competency-based medical educa-
tion. Eric S. Holmboe, Jonathan Sherbino, Donlin M. Long,
Susan R. Swing, Jason R. Frank, and for the International
CBME Collaborators. Medical Teacher Vol. 32, Iss. 8,2010
2. Toward a definition of competency-based education in medi-
cine: a systematic review of published definitions. Jason R.
Frank, Rani Mungroo, Yasmine Ahmad, Mimi Wang, Stefanie
De Rossi, and Tanya Horsley. Medical Teacher Vol. 32, Iss.
8,2010
Milestones
Are discrete points or stages in a developmental progression.
Milestones frameworks in medical education mirror the Dreyfus
model of learner development from novice to expert. Well-
articulated milestones for a specific competency provides a
shared observational approach for assessors and a clear frame
of reference for learners.
Adapted from:
1. Frameworks for learner assessment in medicine: AMEE Guide
No. 78. Louis Pangaro and Olle ten Cate. Medical Teacher
Vol. 35, Iss. 6, 2013.
2. Accreditation Council for Graduate Medical Education
Milestones implementation.
504 K. D. LOMIS ET AL.
References
Anderson MB. 2012. Aligning education and assessment: improving
medical education through assessment. Med Sci Educ. 22:267–269.
Baartman LKJ, Bastiaens TJ, Kirschner PA, van der Vleuten CPM. 2007.
Evaluating assessment quality in competence-based education: a
qualitative comparison of two frameworks. Educ Res Rev. 2:114–129.
Boursicot K, Etheridge L, Setna Z, Sturrock A, Ker J, Smee S,
Sambandam E. 2011. Performance in assessment: consensus state-
ment and recommendations from the Ottawa conference. Med
Teach. 33:370–383.
Carraccio CL, Englander R. 2013. From Flexner to competencies: reflec-
tions on a decade and the journey ahead. Acad Med. 88:1067–1073.
Chertoff J, Wright A, Novak M, Fantone J, Fleming A, Ahmed T, Green
MM, Kalet A, Linsenmeyer M, Jacobs J, et al. 2016. Status of port-
folios in undergraduate medical education in the LCME accredited
US medical school. Med Teach. 38:886–896.
Clayton MJ. 1997. Delphi: a technique to harness expert opinion for crit-
ical decision-making tasks in education. Educ Psychol. 17:373–386.
Crossley J, Johnson G, Booth J, Wade W. 2011. Good questions, good
answers: construct alignment improves the performance of work-
place-based assessment scales. Med Educ. 45:560–569.
Downing SM, Haladyna TM. 2004. Validity threats: overcoming interfer-
ence with proposed interpretations of assessment data. Med Educ.
38:327–333.
Dreyfus SE. 2004. The five-stage model of adult skill acquisition. Bull
Sci Technol Soc. 24:177–181.
Englander R, Burke AE, Guralnick S, Benson B, Hicks PJ, Ludwig S,
Schumacher D, Johnson L, Carraccio C. 2012. The pediatrics mile-
stones: a continuous quality improvement project is launched-now
the hard work begins! Acad Pediatr. 12:471–474.
Englander R, Cameron T, Ballard AJ, Dodge J, Bull J, Aschenbrener CA.
2013. Toward a common taxonomy of competency domains for the
health professions and competencies for physicians. Acad Med.
88:1088–1094.
Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. 2010a.
Toward a definition of competency-based education in medicine: a
systematic review of published definitions. Med Teach. 32:631–637.
Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR,
Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010b.
Competency-based medical education: theory to practice. Med
Teach. 32:638–645.
Hicks PJ, Englander R, Schumacher DJ, Burke A, Benson BJ, Guralnick S,
Ludwig S, Carraccio C. 2010a. Pediatrics milestone project: next
steps toward meaningful outcomes assessment. J Grad Med Educ.
2:577–584.
Hicks PJ, Schumacher DJ, Benson BJ, Burke AE, Englander R,
Guralnick S, Ludwig S, Carraccio C. 2010b. The pediatrics
milestones: conceptual framework, guiding principles, and approach
to development. J Grad Med Educ. 2:410–418.
Hodges B. 2013. Assessment in the post-psychometric era: learning to
love the subjective and collective. Med Teach. 35:564–568.
Holmboe ES, Batalden P. 2015. Achieving the desired transformation:
thoughts on next steps for outcomes-based medical education.
Acad Med. 90:1215–1223.
Krupat E, Pelletier SR. 2016. The development of medical student com-
petence: tracking its trajectory over time. Med Sci Educ. 26:61–67.
Linn R, Baker E, Dunbar S. 1991. Complex, performance-based assess-
ment: expectations and validation criteria. Educ Res. 20:15–21.
Messick S. 1989. Validity. In: Educational measurement. 3rd ed. New
York (NY): American Council on Education: Macmillian Publishing
Company.
Nasca TJ, Philibert I, Brigham T, Flynn TC. 2012. The next GME
accreditation system-rationale and benefits. N Engl J Med.
366:1051–1056.
Pecaric M, Boutis K, Beckstead J, Pusic M. 2017. A big data and
learning analytics approach to process-level feedback in cognitive
simulations. Acad Med. 92:175–184.
Regehr G, Ginsburg S, Herold J, Hatala R, Eva K, Oulanova O. 2012.
Using “standardized narratives” to explore new ways to represent
faculty opinions of resident performance. Acad Med. 87:419–427.
Schumacher DJ, Lewis KO, Burke AE, Smith ML, Schumacher JB, Pitman
MA, Ludwig S, Hicks PJ, Guralnick S, Englander R, et al. 2013. The
pediatrics milestones: initial evidence for their use as learning road
maps for residents. Acad Pediatr. 13:40–47.
Spickard A 3rd, Ahmed T, Lomis K, Johnson K, Miller B. 2016. Changing
medical school IT to support medical education transformation.
Teach Learn Med. 28:80–87.
Swing SR. 2007. The ACGME outcome project: retrospective and pro-
spective. Med Teach. 29:648–654.
Swing SR, Beeson MS, Carraccio C, Coburn M, Iobst W, Selden NR,
Stern PJ, Vydareny K. 2013. Educational milestone development in
the first 7 specialties to enter the next accreditation system. J Grad
Med Educ. 5:98–106.
van der Vleuten CP. 2014. When I say … context specificity. Med
Educ. 48:234–235.
van der Vleuten CP, Schuwirth LW. 2005. Assessing professional com-
petence: from methods to programmes. Med Educ. 39:309–317.
Whitehead CR, Kuper A, Hodges B, Ellaway R. 2015. Conceptual and
practical challenges in the assessment of physician competencies.
Med Teach. 37:245–251.