Otolaryngology–Head and Neck Surgery (2010) 143, 327-330
COMMENTARY
Improving the letter of recommendation
Jeremy D. Prager, MD, Charles M. Myer III, MD, and Myles L. Pensak, MD,
Cincinnati, OH
No sponsorships or competing interests have been disclosed for
this article.
ABSTRACT
The selection of applicants to otolaryngology training programs is
a challenging task. Applicants and their evaluators rely on objec-
tive and subjective data to facilitate the selection process. Unfor-
tunately, data are often less helpful than either side assumes,
suffering from poor validity and reliability in predicting future
performance. The traditional resume-based letter of recommenda-
tion bears some responsibility in this. It is often a lengthy reiter-
ation of already available objective data and contains nonstand-
ardized, superlative evaluations of personal attributes. As a result,
many letters are similar, describing “excellent” candidates who
have done well on previous examinations and clerkships. Research
has indicated improved reliability and satisfaction as well as de-
creased time expenditure using standardized letters of recommen-
dation. These letters demonstrate how basic, easy-to-implement
improvements can create letters that provide accurate information,
separate applicants, and improve the selection process. Consider-
ation should be given to adopting these improvements at the
program director and/or educational committee level.
© 2010 American Academy of Otolaryngology–Head and Neck
Surgery Foundation. All rights reserved.
W hat is the letter of recommendation (LOR)? We all write
them. We all receive them. We all solicit them. But what
are they supposed to communicate and in what fashion? How
much importance do we impart to these letters and what is the
evidence for doing so? These are difficult questions to answer.
Now compound this situation with the competitive and imperfect
process of selecting residents and hiring partners. Would an im-
proved process for letters of recommendation decrease the chance
of an incorrect hiring or match? Are we willing to continue betting
that it would not? The following commentary includes a collection
of our thoughts on the current LOR process and how and why it
might be reformed. Table 1 summarizes some of the more salient
points.
The Problems with the Letter of
Recommendation
While the curriculum vitae provides a factual view of an
individual’s accomplishments, it typically does so in a ster-
Received March 4, 2010; revised March 9, 2010; accepted March 15, 2010.
0194-5998/$36.00 © 2010 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved.
doi:10.1016/j.otohns.2010.03.017
ile fashion, without allowing the reader to know the appli-
cant. As a result, those in a position of recruitment often
receive an LOR as a means of personalizing a candidate,
understanding that their own practice requirements and eco-
nomic constraints may prohibit a personal interview with
every potential candidate. However, with few exceptions, a
uniform set of expectations or format regarding these letters
does not exist. A traditional LOR is subject to interpretation
(or misinterpretation) and may eliminate an exceptional
candidate for consideration for potential employment either
as a resident or practicing physician.
What is expected from the LOR? In most medical fields,
this is not defined. The writer is without guidance in his or
her task. For this reason, a portion of the LOR often reit-
erates academic information already available in other for-
mats. In addition, there is no agreement on what “excellent”
means (What other adjectives are used? How many appli-
cants are excellent? Out of what denominator?). Also, letter
writers are selected by the applicant, creating an inherent
selection bias. As a result, nearly all applicants are highly
rated with little to truly separate them. This creates frustra-
tion for those who must read the LOR with the intent of
ranking candidates.
The reliability and validity of the LOR when used as a
selection tool are dubious. Few things about a person have
been effectively and repeatedly shown to predict how he or
she will act or what he or she will do in the future. The
traditional LOR is not one of them. Although a few studies
have demonstrated a correlation between the LOR and mea-
sures of resident performance, many studies have reached
the opposite conclusion.1 In addition, the interrater reliabil-
ity when evaluating the LOR leaves something to be de-
sired.2,3
A Potential Solution
Those who read and write these letters should agree on a
few things from the outset. Should we get rid of the LOR if
it cannot be improved? Well, maybe, but this is not a
realistic solution. Despite its limitations, many consider
these letters essential.1 LOR length must be curtailed. No
physician has the time to create or read an excessively long
LOR. A candidate for whom such a letter has been written
328 Otolaryngology–Head and Neck Surgery, Vol 143, No 3, September 2010
Table 1
“Do’s” and “don’ts” of the letter of recommendation
Do’s
Agree as a field on a standard structure for the LOR
Include background information to describe the
writer’s level of experience
Describe and use a standardized list of adjectives
Give concrete examples of applicant’s traits
(demonstrate writer’s depth of understanding of
the applicant)
Consider self-directed reading or training session to
improve the writing of the LOR
LOR, letter of recommendation.
may be penalized by the recipient’s reluctance to read the
entire letter. Valuable information may be missed as the
reader scans the LOR without absorbing hidden details.
Likewise, the writer should not feel obligated to produce a
short story to describe and characterize a candidate. It is not
unreasonable to expect an appropriate LOR to be contained
within a single page. If writers and readers alike can agree
on this limit, a letter can become an asset for a worthy
candidate instead of a detriment where salient points are
buried within an excessively long letter.
What should be contained within the LOR? For this, we
need to reach an agreement. A large survey of internists
demonstrated that letters need to show depth of understand-
ing of the applicant, a numeric comparison of the applicant
to his or her peer group, and a summary statement.1 Stan-
dardization of the LOR and the inclusion of specific exam-
ples of how an individual’s behavior or attitude compares
with a reference group are useful.4
The field of emergency medicine has adapted to the
problem of LOR content through standardization. A council
of their residency directors created a form LOR for writers.5
The letter is two pages in length and is broken into four
sections: background information on the applicant and letter
writer (including the relative number of students who re-
ceived the same grade the previous academic year to pro-
vide the reader with a denominator), personal characteristics
such as commitment and work ethic in comparison with
other emergency medicine applicants (authors are limited to
specific descriptive terms), a summary statement that asks
how highly the applicant would rank at the writer’s institution,
and an open narrative section limited to 150 to 200 words.
The standardized LOR was implemented in 1995. Pre-
liminary evaluation of the standardized LOR was favorable
(easier and faster to read and write, improved ability to
differentiate between applicants).6 Compared with the usual
narrative LOR, the standardized LOR also demonstrated
improved interrater reliability.7 We await further studies of
the validity and reliability of this standardized letter.
Don’ts
Reiterate academic data available elsewhere
Write a too lengthy LOR (aim for one page)
Omit grade and rank distribution and a numeric
comparison of the applicant with his/her peer group
Forget that you may decline to write a requested LOR
Be unfamiliar with legal issues surrounding a negative LOR
Our Proposal
We propose to develop a framework to assist surgical sub-
specialists in both the academic and private arenas in the
creation of an LOR that can be interpreted in an unambig-
uous manner (Table 2). The initial paragraph should be brief
and direct and should cover the writer’s background and
how the applicant is known to him or her. It should contain
an evaluation of a candidate’s personal traits, with an em-
phasis on actual events that demonstrate receptive and ex-
pressive language skills, reasoning ability, development of
interpersonal relationships with individuals of varied socio-
economic and cognitive abilities, personality, ethical behav-
ior, work ethic, and surgical skills.
The body of the LOR should include a commentary
regarding how this candidate ranks in comparison with his
or her contemporaries or predecessors in training. In doing
so, a frame of reference is created for the reader as he or she
attempts to distill subjective information into a concrete and
finite decision regarding employment. The candidate’s
grade on service rotation should be placed in the context of
the grade distribution for all rotating medical students. The
candidate’s commitment to otolaryngology, work ethic, as-
sumption of responsibility, ability to create a differential
and treatment plan, surgical skills, and communicative skills
(among other qualities) may all be ranked within a category
of achievement in comparison with other applicants (i.e.,
outstanding [top 10%], excellent [top one third], good [mid-
dle one third], average [lower one third]). Finally, a global
assessment of the candidate’s rank in relation to other can-
didates may be offered, along with the distribution of such
rankings over a specified time period. This will provide
further objective data to the letter recipient.
Last, the letter should conclude with a definitive statement
regarding the candidate’s suitability for employment. Without
embellishment or exaggeration, four simple categories should be
used: do not recommend; recommend with reservation; recom-
mend without reservation; recommend highly without reservation.
By creating this rating system, the writer is forced to make a
definitive statement regarding an applicant’s candidacy, thus ful-
Prager et al Improving the letter of recommendation 329

Table 2
Proposed framework of the letter of recommendation

Portion of letter Contents Example

Introductory paragraph
Body
Summary statement
on recommendation
Writer’s background and experience
Brief background on applicant and how he/she
is known to the writer
Concrete examples of personal traits
(language, reasoning, compassion, ethics,
surgical skills)
Comparison of applicant to contemporaries
and predecessors in training using
standardized language and denominators
Description of grading system for clinical
rotations: honors, high pass, pass, low pass,
and fail
Description of ranking system for categories of
achievement: outstanding ⫽ top 10%,
excellent ⫽ top one third, good ⫽ middle
one third, average ⫽ lower one third
Global assessment of the candidate’s rank in
relation to other candidates, along with the
distribution of such rankings over a specified
time period
Brief narrative section (optional)
Four categories of recommendation: do not
recommend, recommend with reservation,
recommend without reservation, recommend
highly without reservation
Invitation to call the writer
I have been an otolaryngologist for 15
years and program director for five.
The applicant is a third-year medical
student who rotated on my service
for two months.
In addition I am the applicant’s
advisor.
In my presence, the applicant
demonstrated compassion in
relating a poor diagnosis to the
patient and family.
This applicant received an honors on
our rotation. Of the 20 students
who did this rotation last year, 20%
received honors.
I would consider the applicant’s
commitment to otolaryngology and
work ethic to be outstanding (top
10%). His/her surgical skills are
good (middle one third)
The applicant’s clinical performance
has been excellent (top one third).
Over the past five years, I have
written 14 other letters of which
one candidate was rated
outstanding, three were excellent,
five were good, and five were
average.
I recommend this candidate highly
without reservation.
Please call me for other questions.

filling the expectations of the applicant, writer, and reader alike.
The letter should end with an invitation to call the writer should
there be any ambiguity.
Additional Considerations
Few physicians are counseled or trained in how to write the
LOR. A brief review of the literature will provide a reason-
able background from which the writer may work. Aca-
demic departments may consider a brief training session for
physicians who are often asked to write an LOR.
It is perfectly acceptable to decline an invitation to write
an LOR. Defamation requires proof of false and derogatory
statements, either written or spoken, which causes harm to
an individual. Physicians who include negative comments
in an LOR should review their comments to make sure they
are accurate and factual.4 Lawsuits have occurred. Recom-
mendations for writers of the LOR in light of this legal
situation are available.4 In today’s litigious society, a writer
may be concerned about how expressive and open to be in
an LOR. By following the format we propose, the writer
prepares a concise yet informative letter in a manner that
should engender relatively little controversy.
Conclusion
We have attempted to acknowledge the importance of the
LOR for both applicants as well as employers. The well-
written LOR should be concise and consist of sections
detailing both objective and subjective data while providing
a clear-cut statement of a candidate’s suitability for employ-
ment. A format for the LOR may be adopted to simplify,
unify, and improve the evaluation process of surgical sub-
specialty candidates. Respecting this model will protect and
assist applicants, writers, and readers of the LOR as the
330 Otolaryngology–Head and Neck Surgery, Vol 143, No 3, September 2010
letter becomes a consistent, repeatable means of transmit-
ting accurate information. Leadership in otolaryngology
training should consider adopting these changes to the LOR
as a means of improving the selection of residents.
Author Information
From the Department of Pediatric Otolaryngology–Head and Neck Sur-
gery, Cincinnati Children’s Hospital Medical Center (Drs. Prager and
Myer), Cincinnati, OH; and Department of Otolaryngology, University of
Cincinnati College of Medicine (Dr. Pensak), Cincinnati, OH.
Corresponding author: Jeremy D. Prager, MD, MLC 2018, 3333 Burnet
Ave., Cincinnati, OH 45229-3039.
E-mail address: jeremy_prager@yahoo.com.
Author Contributions
Jeremy D. Prager, author, critical revision, concept, and design;
Charles M. Myer III, author, critical revision, final approval, concept, and
design; Myles L. Pensak, critical revision, final approval, concept, and
design.
Disclosures
Competing interests: None.
Sponsorships: None.
References
1. DeZee KJ, Thomas MR, Mintz M, et al. Letters of recommendation:
rating, writing, and reading by clerkship directors of internal medicine.
Teach Learn Med 2009;21:153– 8.
2. Dirschl DR, Adams GL. Reliability in evaluating letters of recommen-
dation. Acad Med 2000;75:1029.
3. O’Halloran CM, Altmaier EM, Smith WL, et al. Evaluation of resident
applicants by letters of recommendation: a comparison of traditional
and behavior-based formats. Invest Radiol 1993;28:274 –7.
4. Wright SM, Ziegelstein RC. Writing more informative letters of refer-
ence. J Gen Intern Med 2004;19:588 –93.
5. Council of Emergency Medicine Residency Directors. Standard Letter
of Recommendation. Available at: www.cordem.org/slor.htm. Accessed
February 20, 2010.
6. Keim SM, Rein JA, Chisholm C, et al. A standardized letter of recom-
mendation for residency application. Acad Emerg Med 1999;6:1141– 6.
7. Girzadas DV Jr., Harwood RC, Dearie J, et al. A comparison of
standardized and narrative letters of recommendation. Acad Emerg Med
1998;5:1101– 4.