The Role of Experience on Judgments of Dysphonia

*Tanya L. Eadie, *Mara Kapsner, Juli Rosenzweig, Patricia Waugh, Allen Hillel, and Albert Merati,
*yzSeattle, Washington
Summary: Objectives. The objectives of the study were (1) to determine differences in judgments of overall sever-
ity (OS) and vocal effort (VE) of dysphonic speech when judgments were made by experienced and inexperienced
listeners, and when self-rated by individuals with dysphonia; and (2) to determine relationships between auditory-
perceptual judgments of voice and voice handicap.
Study Design. Prospective and exploratory.
Methods. Twenty speakers with dysphonia and four normal controls provided speech recordings. Participants judged
their own speech samples for OS and VE and completed the Voice Handicap Index (VHI). Twenty-four inexperienced
and 10 experienced listeners evaluated the same speech samples for OS and VE using 100-mm visual analog scales.
Results. No differences were found for judgments of OS and VE across the groups. However, relationships between
judgments made by experienced and inexperienced listeners were strong, whereas those between individuals with
dysphonia and other listeners were weak to moderate. All listeners judgments of voice were moderate predictors of
VHI scores, with patient-perceived VE and clinician-rated OS being relatively strongest.
Conclusions. Although there is no systematic effect of listener experience on judgments of dysphonia, individuals
with dysphonia appear to self-rate their voices using different perceptual strategies than other listeners. Auditory-
perceptual measures are only moderately related to voice handicap scores, indicating that they are complementary
measures of voice.
Key Words: Perceptual voice judgmentsVoice handicapListener experience.
INTRODUCTION
Auditory-perceptual evaluation is a highly valued clinical tool
used in voice diagnosis, assessment, and treatment.
1
During
voice evaluations, different types of listeners are asked to
make voice quality judgments. First, individuals with dyspho-
nia often are asked to judge the overall severity (OS) of their
voices and perceived vocal effort (VE) using formal
2
or infor-
mal rating tools.
3
Subsequently, the otolaryngologist and
speech-language pathologist assess and measure treatment out-
comes in voice patients using auditory-perceptual methods.
Finally, research studies often include judgments from listeners
who have limited exposure to voice disorders (ie, inexperienced
listeners).
1
Judgments from inexperienced listeners are used to
gauge the impact of voice disorders on communication part-
ners. Although auditory-perceptual evaluations are performed
by different types of listeners, the effect of experience on judg-
ing dysphonia remains unclear.
Listener experience and perceived dysphonia
Understanding how experience affects perceptual judgments is
critical to clinical decision making, because information
derived from different listeners affects the measurement of out-
comes and treatment decisions. However, results are inconsis-
tent when listeners with different experience levels are
compared. For example, a number of studies have examined
differences between how experienced and inexperienced lis-
teners judge the voices of individuals with dysphonia. Some
studies have found that experience with patients with voice dis-
orders increases a clinicians sensitivity to the severity of dys-
phonia (ie, experienced listeners are more bothered by
dysphonia).
4
In contrast, other studies have found that experi-
enced clinicians judge voice disorders less severely than inex-
perienced listeners, because increased exposure may reduce
a clinicians reactions to disordered speech or voices.
5
Other
studies have shown no differences in how inexperienced and
experienced listeners judge dysphonic voices.
6
A number of reasons explain why results from these studies
are variable. First, most data are based on very few participants
in each listener group (eg, typical range of experienced listeners
is 13), thereby reducing the external validity of the results.
1
Second, some researchers do not provide denitions of percep-
tual dimensions for their participant-listeners; in addition, they
may use rating scales that are inappropriate for making these
judgments, which may contribute error to the data.
7,8
Finally,
all of these studies have been performed on listeners who
have different levels of training with varied populations of voice
and/or speech disorders, making it difcult to compare results
across studies.
A limited number of studies have investigated differences
between patients self-ratings of voice quality and inexperi-
enced and experienced listeners judgments of their voices.
3,6,9
An individuals perception of the severity of his or her dyspho-
nia is critical, because the success of voice treatment is largely
based on that individuals perception of his or her voice, regard-
less of the clinicians perceptions of that same voice.
1
Lee et al
3
performed one of the few studies comparing patients self-
ratings of voice quality with experienced listeners judgments.
In that study, 28 individuals with dysphonia were found to
Accepted for publication December 5, 2008.
Portions of this paper were presented at the Voice Foundations 37th Annual Sympo-
sium: Care of the Professional Voice, May 2008, Philadelphia, PA.
From the *Department of Speech and Hearing Sciences, University of Washington,
Seattle, Washington; yDepartment of Rehabilitation Medicine, University of Washington,
Seattle, Washington; and the zDepartment of OtolaryngologyHead & Neck Surgery,
University of Washington, Seattle, Washington.
Address correspondence and reprint requests to Tanya L. Eadie, Department of Speech
and Hearing Sciences, University of Washington, 1417 NE, 42nd Sreet, Seattle, WA98105.
E-mail: teadie@u.washington.edu
Journal of Voice, Vol. 24, No. 5, pp. 564-573
0892-1997/$36.00
2010 The Voice Foundation
doi:10.1016/j.jvoice.2008.12.005
consistently rate their voices as more severe than either of the
two experienced clinicians who judged the same speech sam-
ples. However, these results again are difcult to generalize be-
cause of the small number of experienced listeners who
participated.
Differences among listeners with varied experience levels
might be expected because experience is one factor by which
listeners develop internal templates for various voice qualities.
For example, Kreiman et al
1
have argued that all listeners have
similar, relatively stable internal standards for normal voice
quality, because all listeners have approximately equal experi-
ence with normal voices. They suggest that when inexperienced
listeners are asked to judge dysphonic voices, they base their
judgments on standards suited to normal voices. In contrast,
experienced listeners compare dysphonic voices to variable
templates developed through repeated exposure to pathological
qualities; in fact, variable experience among experienced lis-
teners may even render them more variable as a group than
inexperienced listeners.
1
Lee et al
3
suggest that it is unlikely that individuals with dys-
phonia have internal templates of disordered voices beyond
their own. They showed that individuals with dysphonia were
reliable in self-judgments of voice. However, agreement
between self-perceived dysphonia and experienced listeners
judgments of their voices was poor, suggesting that different
types of listeners base their judgments on different cues. These
results are supported in a recent investigation of individuals
with adductor spasmodic dysphonia (ADSD).
6
In that study,
evaluations of OS and VE were investigated when judgments
were made by 10 experienced listeners, 20 inexperienced lis-
teners, and 20 individuals with ADSD. They found that group
averages did not differ among the three groups. However,
results showed that relationships between speakers ratings of
their own voices and other listeners ratings were only moder-
ate, whereas inexperienced and experienced listeners judg-
ments were strongly correlated.
6
These results support the
contention that speakers base their judgments on cues that are
different than other listeners, and that one might expect judg-
ments to vary as a function of listener experience. Further, if
speakers and other listeners base their judgments on different
perceptual cues, then the predicted relationship between audi-
tory-perceptual measures and other outcomes, such as voice-
related quality of life (V-RQOL), might also be expected to
differ as a function of experience.
Relationships between perceived dysphonia and
quality of life
In addition to auditory-perceptual outcomes, recent investiga-
tions have focused on measuring the impact of voice disorders
on V-RQOL using validated questionnaires.
1012
Although one
might hypothesize a strong relationship between perceived
severity of dysphonia and V-RQOL, results have indicated
that only weak to moderate relationships exist. For example,
Murry et al
13
found that patient-reported V-RQOL
11
scores
from 50 individuals with dysphonia were only moderately
related (r 0.44) to two clinicians ratings of perceived dys-
phonia. Similarly, Behrman et al
14
found a moderate relation-
ship (r 0.48) between 50 patients V-RQOL scores using
the Voice Handicap Index (VHI)
10
and one experienced clini-
cians ratings of their voices. Other studies have found weaker
relationships between experienced clinicians judgments of
voice and V-RQOL scores (eg, range r 0.230.32).
3,6
Relationships between V-RQOL and inexperienced listeners
judgments of dysphonia have been found to be even weaker
than the same relationships with experienced listeners. For
example, Ma and Yiu
12
found that three novice listeners judg-
ments of OS, roughness, or breathiness did not predict more
than 8% of 40 dysphonic individuals V-RQOL scores. Further-
more, even when patients self-rate their own voice quality (OS),
these ratings are only moderately related to V-RQOL outcomes
(r 0.600.65).
3,10
Results from these studies have led investi-
gators to conclude that voice quality and psychosocial impact
are two different constructs, and that both types of evaluations
(auditory-perceptual judgments of voice, patient-reported
QOL) are valuable as independent outcomes.
3,13,14
Two reasons may explain why weak relationships have been
previously found to exist between voice quality dimensions and
self-reported QOL scores. First, the relationship may depend on
the rated perceptual dimension (eg, OS, breathiness, rough-
ness). Second, the strength of the relationship may be affected
by the experience of the listener. For example, although audi-
tory-perceptual dimensions relate to the auditory signal, they
also may be interpreted using sensory cues that are only avail-
able to the speaker. If these dimensions are interpreted by
speakers differently than other listeners, discrepancies between
patients and clinicians perceptual judgments may result. Lee
et al
3
suggested that, although clinicians are likely to base judg-
ments on what they hear and see, patients may make judgments
based on what they hear and feel. Thus, one dimension that may
be a more sensitive predictor of V-RQOL is perceived effort (ie,
a subjective index of vocal fatigue),
15
because individuals with
dysphonia may perceive this dimension differently than other
listeners.
14
For example, in their study, Eadie et al
6
found the
only signicant predictor of V-RQOL in individuals with
ADSD was the patients self-ratings of perceived VE. In con-
trast, ratings made by experienced and inexperienced listeners,
including OS of voice and perceived effort, were only weakly
related to QOL scores.
Purpose of the study
Although the ndings by Eadie et al
6
may be important for clin-
ical decision making in individuals with ADSD, it may be that
these ndings are unique to ADSD, because perceived effort/
strain is one dimension that characterizes the perceptual symp-
toms of dysphonia in this clinical population.
16
However, it is
important to determine whether similar effects related to lis-
tener experience and relationships between perceived dyspho-
nia and V-RQOL are demonstrated in individuals with other
types of dysphonia. Therefore, the purposes of this study
were (1) to determine differences in listener judgments of OS
and VE when judgments were made by experienced listeners,
inexperienced listeners, and individuals with dysphonia; and
(2) to determine relationships between auditory-perceptual rat-
ings of dysphonia and voice-related QOL using the VHI.
10
Tanya L. Eadie, et al Experience and Judgments of Dysphonia 565
METHODS
Participants
Three groups of participants included: (1) individuals with dys-
phonia; (2) inexperienced listeners; and (3) experienced lis-
teners. All participants were native English speakers and
reported no other speech, language, hearing, or voice symptoms
apart from those reported by participants in the dysphonia
group. Individuals with dysphonia passed hearing screening
tests at 40 dB for the octave frequencies of 2508000 Hz.
This screening level was selected to maximize the number of
eligible older individuals who could participate, while not
affecting the validity of the listening procedure.
17
Inexperi-
enced and experienced listeners passed hearing screening tests
at 20 dB for the octave frequencies of 2508000 Hz. All partic-
ipants were paid, and procedures were approved by the Univer-
sity of Washington Human Subjects Committee.
Individuals with dysphonia. Participants included 20
individuals (10 males, 10 females) diagnosed with a variety
of larynx-based voice disorders by one of two experienced
laryngologists (A.H., A.M.). Individuals were among those
who were evaluated for voice-related complaints in the Depart-
ment of OtolaryngologyHead and Neck Surgery at the Uni-
versity of Washington Medical Center. Individuals with
diagnoses of spasmodic dysphonia or those who had undergone
total laryngectomy were excluded. The mean age of all
speakers was 52.5 years (age range: 2585 years), with mean
ages of 51.8 years (range: 2578 years) for men, and 53.2 years
(range: 3185 years) for women. In addition, four age- and sex-
matched controls with no known history of voice impairment
were included in the group.
Inexperienced listeners. Twenty-four (seven males, 17
females) individuals (mean age: 23.9 years; range: 1927 years)
were recruited from the student population and broader com-
munity at the University of Washington. Listeners had no known
prior experience with or formal exposure to voice pathology.
Experienced listeners. Experienced listeners (mean age:
50.3 years; range: 3162 years) included 10 certied speech-
language pathologists (two males, eight females). These partic-
ipants reported working with individuals with voice disorders
aged 17.1 years on average (range: 638 years).
Data collection
Individuals with dysphonia. Individuals with dysphonia
completed a demographic form and a validated V-RQOL ques-
tionnaire, the VHI.
10
The VHI consists of 30 items ranked on
a 5-point scale, with the VHI total scores ranging from 0 to
120 (eg, 0 best score; 120 maximum perceived voice hand-
icap). Individuals were then asked to provide speech samples of
Fairbanks Rainbow Passage.
18
A common reading passage
was chosen as the stimulus to control phonetic context, which
may affect dysphonia and perception of voice quality.
19
Control
of this variable was important to make valid comparisons across
speakers. Speech samples were recorded using a headset micro-
phone (AKG C420, AKG Acoustics, Vienna, Austria) routed to
a digital audiotape recorder (Tascam DAP1, TEAC Corpora-
tion, Tokyo, Japan) and digitized at sampling rate of
44 100 kHz. All samples were recorded in a quiet environment
with low amounts of ambient noise. Immediately after record-
ing the speech sample, participants were provided with deni-
tions of OS and VE. In this study, overall voice severity (OS)
was dened as a comprehensive measure of how good or
poor the voice sample is judged to be by the listener. This
judgment is based on multiple factors which ultimately range
from normal to profoundly impaired on a continuum.
7
Vocal
effort was dened as the perceived effort in producing
voice.
20
Participants were familiarized with 100-mm visual
analog scales (VAS) and were asked to listen to their recordings
of the second sentence of the Rainbow Passage, The Rainbow
is a division of white light into many beautiful colors through
headphones (Samson RH600, Samson Technologies,
Hauppauge, NY). Judgments of the second sentence of the Rain-
bow Passage relate to listener judgments of the entire passage,
and thus, the second sentence was selected as the stimulus.
18
Based on the perception of their speech samples, participants
were then asked to make judgments of OS and VE using the rat-
ing scales provided. To control for order effects, the rated dimen-
sions (OS, VE) were randomized across participants.
Stimulus preparation
To prepare the stimuli for the experienced and inexperienced
listeners, speech samples were transferred from digital audio-
tape (DAT) to a desktop computer, and the second sentence
of the Rainbow Passage was extracted using sound-editing
software. The samples were entered into a custom-made soft-
ware program that randomly generates speaker order, presents
perceptual rating scales, and records responses. Twenty-ve
percent (n 6) of the voices were randomly repeated in each
condition (OS and VE) to assess intrarater reliability for a total
of 60 speaker samples across the dimensions (N 24 + 6 sam-
ples per condition 32 conditions 60 stimuli).
Auditory-perceptual procedure for inexperienced
and experienced listeners
Twenty-four inexperienced and 10 experienced listeners com-
pleted a demographic form and participated in similar listening
protocols. Listeners were provided the same OS and VE deni-
tions as those presented to the individuals with dysphonia, and
were familiarized with the rating scales. Listeners then were
asked to make judgments of OS and VE for all of the stimuli
using 100-mm VAS. Listeners made judgments after a single
presentation of each stimulus. The order of speech samples
and dimensions (OS, VE) were randomized across listeners to
counter for learning effects.
Statistical methods
Group means of listeners ratings for OS and VE were calcu-
lated for each speech sample. Differences across OS and VE
judgments for the three groups (individuals with dysphonia,
inexperienced listeners, experienced listeners) were determined
using two repeated-measures analysis of variance (ANOVA)
tests and corrected post hoc tests, when appropriate. Relation-
ships among perceived OS and VE were determined using
Journal of Voice, Vol. 24, No. 5, 2010 566
multiple Pearson product-moment correlation coefcients.
Scoring algorithms were used to calculate total VHI scores. Al-
though subscales can be calculated for physical, social, and
functional handicap, only total VHI scores were used in this
study. VHI scores were examined for their relationship with lis-
teners ratings of auditory-perceptual dimensions using Pear-
sons correlations. A predetermined level of statistical
signicance (P < 0.01) was used for all correlational analyses,
because multiple correlations were performed.
Reliability
To determine intrarater reliability for the inexperienced and
experienced listeners, six speaker samples (25%) for each
dimension were randomly repeated. Pearson product-moment
correlation coefcients were calculated by comparing the orig-
inal and repeated VAS values. Because correlations do not indi-
cate the degree to which any two judgments agree, a measure of
intrarater agreement also was calculated, whereby two judg-
ments were considered to agree if they fell within 10 mm on
the 100-mm VAS.
21
The percentage agreement was calculated
for each listener in the inexperienced and experienced groups
across the two rating conditions (OS and VE). Group means
are reported for each listener group in Table 1.
Interrater reliability of auditory-perceptual ratings was ana-
lyzed using Cronbachs alpha and a measure of interrater agree-
ment. This measure was determined by calculating the
difference between each listeners rating for every stimulus
with the average listeners judgment for the same stimulus
(ie, the mean score across listeners). Ratings within 10 mm
on the 100-mm VAS were considered to be in agreement.
21
The mean percentage agreement was calculated for each lis-
tener group across the rating conditions (Table 1).
RESULTS
Individuals with dysphonia: demographics
Demographic information for the 20 participants with dyspho-
nia and the four normal control speakers is presented in Table 2.
The data are presented in order of least to most severe voice
handicap, as measured by the VHI total scores across the
speaker group.
The mean VHI score for all 24 participants was 31.3 (stan-
dard deviation 24.3) (mild to moderate voice handicap);
scores ranged from 2 (essentially no voice handicap) to 85
(ie, severe voice handicap, out of a maximum handicap score
of 120). Although one normal control participant (S21) reported
a score of 19 on the VHI and a self-perceived dysphonia score
(OS) of 31, he did not report any specic voice symptoms or
complaints at the time of recording. He also was informally
judged by the rst author (T.L.E.), an experienced voice clini-
cian with more than 9 years of experience, to exhibit normal
voice quality; these data were conrmed by the experienced lis-
teners (mean OS for S21 10.50 on the 100-mm VAS). Thus,
his data were deemed acceptable and included in the study.
Differences between the VHI scores for men and women
were nonsignicant. Relationships between VHI scores and de-
mographic variables, such as age and gender, did not reveal any
signicant relationships (P > 0.05). In addition, there were no
differences found in VHI totals as a function of pathology
(eg, paralysis vs. benign lesions, and others.). As a result,
data from individuals with all types of laryngeal pathologies
were treated as a single group in the analysis.
Differences across listener judgments
To determine whether judgments of OS and VEdiffered with re-
spect to listener group, two repeated-measures ANOVAs were
performed. Data frommale and female participants were treated
as a single group in the analysis, because no signicant differ-
ences were found for any of the rated dimensions with respect
to gender. Results revealed no signicant differences across
groups for judgments of either OS (F(2,38) 1.063, P > 0.05)
or VE (F(2,38) 2.655, P > 0.05). No signicant relationships
were found between any demographic factors (eg, age, number
of years of voice-related clinical experience, laryngeal pathol-
ogy) and listeners judgments. A boxplot and whiskers graph
is plotted in Figure 1. In this graph, median scores are indicated
by the line within each box, with 25th and 75th percentiles (rst
and third quartiles) forming the borders of the box. Minimum
and maximum values are indicated by the bottom and top whis-
kers, respectively. Means and standard errors of listener judg-
ments of OS and VE also are plotted in Figure 1. The data
show relatively equal variation in listener responses across the
three groups, and similar use of the rating scales for both judg-
ments of OS and VE.
Relationships among listener judgments
The relationships among listener groups was investigated using
Pearsons correlation coefcients; the strength of the relation-
ships is indicated by the variance scores (r
2
). Judgments made
TABLE 1.
Means and SDs for Intra- and Interrater Reliability Measures of OS and VE Judgments
Reliability Measures
Inexperienced
Listeners: OS
Experienced
Listeners: OS
Inexperienced
Listeners: VE
Experienced
Listeners: VE
Mean intrarater correlation (r) 0.87 (SD0.12) 0.91 (SD0.12) 0.84 (SD0.18) 0.94 (SD0.05)
Mean intrarater agreement (%) 63.89 (SD21.23) 63.33 (SD29.19) 63.19 (SD16.28) 76.67 (SD17.92)
Mean ICC (alpha) 0.99 0.98 0.97 0.97
Mean interrater agreement (%) 78.83 (SD18.93) 80.40 (SD19.36) 77.00 (SD18.47) 76.00 (SD13.47)
Abbreviations: SD, standard deviation; ICC, intraclass correlation.
SDs are not available for ICC values.
Tanya L. Eadie, et al Experience and Judgments of Dysphonia 567
by inexperienced and experienced listeners were strongly and
signicantly correlated for judgments of OS and VE, and pre-
dicted between 82% and 88% of the variance. In contrast, rela-
tionships between self-rated OS and VE by individuals with
dysphonia with those made by other listeners (both experienced
and inexperienced listeners) were moderate in strength (Table
3), and only predicted between 23% and 35% of the variance.
Figures 2 and 3 also show the weaker relationships between
judgments made by individuals with dysphonia with those
made by other listeners; these relationships contrast with the
strong relationships between the inexperienced and experi-
enced listeners. Figures 2 and 3 also show even, but randomly
distributed residuals about the regression lines, suggesting
that there is no response bias in the use of the VAS by any lis-
tener group.
Relationships between perceptual dimensions are shown in
Table 4. Dimensions judged by the inexperienced and experi-
enced listeners were strongly and signicantly correlated (ie,
OS and VE judgments were strongly related). In contrast,
self-rated OS judgments were only weakly related to self-rated
VE in individuals with dysphonia.
Relationships between voice handicap index scores
and auditory-perceptual dimensions
Auditory-perceptual judgments made by all listener groups
were signicantly related to VHI total scores (P < 0.01) (Table
5). All relationships were moderate in strength (predicting
between 30% and 46% of the variance in the data). Empirically,
the strongest relationships were found between self-rated VE
and experienced listeners judgments of OS to VHI totals. No
TABLE 2.
Demographic Characteristics of Individuals With Dysphonia and Control Speakers (N24), Including Participant
Identication (ID) Code, Gender, Age (y), Diagnosis, VHI Totals, and Self-Perceived Dysphonia
(OS as Judged on a 100-mm VAS)
Participant ID Code Gender Age (y) Diagnosis VHI Total Self-Perceived OS
Dysphonic speakers
1 M 78 Bowing BL VFP 2 2
14 F 81 UL VFP, muscle tension
dysphonia
2 66
9 M 33 BL edema secondary to reux 4 35
2 M 64 VFP, laryngopharyngeal reux 6 28
15 F 63 BL Reinkes edema 14 68
8 M 70 Laryngeal cancer, s/p partial
laryngectomy and radiation
therapy
18 58
10 M 60 s/p Zenkers diverticulectomy 24 29
20 F 38 UL VF pedunculated polyp 29 60
6 M 64 LT VFP s/p mediastinoscopy and
upper lobectomy for lung
cancer
31 26
18 F 44 VF cyst, VF edema 31 14
7 M 27 LT VFP 35 12
16 F 60 VF cyst 37 41
5 M 53 Papilloma 39 61
3 M 25 UL VFP, possible dislocated vocal
process, lesion of unknown
etiology
47 25
19 F 56 Clear larynx s/p resolved VF
nodules
50 54
13 F 47 VF cyst, chronic cough for 6 years 55 59
17 F 31 VF nodules, subglottic stenosis 63 56
12 F 45 RT VF cyst, LT Reinkes edema 66 85
11 F 85 BL VFP, receiving collagen
injections
75 87
4 M 57 RT VFP s/p parathyroid surgery 83 55
Control speakers
24 F 53 Normal voice 6 11
22 M 37 Normal voice 8 14
23 F 35 Normal voice 8 14
21 M 53 Normal voice 19 31
Abbreviations: M, male; F, female; BL, bilateral; LT, left; RT, right; s/p, status post; UL, unilateral; VF, vocal fold; VFP, vocal foldparesis; VHI, Voice HardicapIndex.
Speakers are listed in order of increasing VHI totals.
Journal of Voice, Vol. 24, No. 5, 2010 568
differences in relationships were observed as a function of
laryngeal pathology or any demographic variables. Correla-
tions are shown in Table 5.
DISCUSSION
The purpose of this study was twofold. The rst objective was
to determine the effect of experience on auditory-perceptual
judgments of dysphonia in three different listener groups,
including individuals with dysphonia, inexperienced listeners,
and experienced listeners. These effects were examined using
larger numbers of subjects, including both inexperienced and
experienced listeners, than previous studies to increase the val-
idity of the ndings. The second objective was to evaluate the
relationships between auditory-perceptual ratings of dysphonia
and patient-reported voice handicap. Results indicated that
although judgments of OS and VE were not signicantly differ-
ent across groups, individuals with dysphonia appeared to use
different strategies for evaluating perceptual dimensions than
other listeners. In addition, voice handicap was moderately pre-
dicted by all listener groups perceptions of dysphonia. These
results are discussed relative to models of voice perception
and clinical evaluation of voice disorders.
Differences across listener judgments
In the present study, no signicant differences were found for
ratings of OS across all listener groups, with similar variability
and use of rating scales by all listeners (Figure 1). These results
are consistent with those found by Damrose et al,
4
who found
no signicant differences between three experienced and three
novice listeners severity judgments of spasmodic dysphonia.
Results also are consistent with those reported by Eadie et
al,
6
who found no effect of listener experience on OS judgments
made by eight voice clinicians, 20 inexperienced listeners, and
20 individuals with spasmodic dysphonia. However, the results
contrast with those who found differences between patients
self-perceptions of voice and auditory-perceptual evaluations
performed by other listeners. For example, Lee et al
3
found
that 28 dysphonic individuals consistently rated their voices
more severely than either of the two voice clinicians, whereas
Sapir et al
9
found that self-rated dysphonia by patients with
spasmodic dysphonia was less severe than clinicians.
In addition to judgments of OS, listeners in the present study
evaluated dysphonic speech samples for VE. Like judgments of
OS, no signicant differences were found as a function of listener
experience, which is consistent with previous results in
FIGURE 1. A boxplot and whiskers graph illustrating the variance in judgments of OS and VE across the listener groups (inexperienced listeners,
experienced listeners, individuals with dysphonia). The midportion of the box indicates the median values. The 25th and 75th percentiles of the data
are indicated by the borders of the box. The whiskers show the minimum (lower) and maximum (upper) values. Means (diamond shapes) and stan-
dard errors (error bars) are located within the box for each listener group.
TABLE 3.
Pearson Correlation Coefcients and Variance (r
2
) Scores Across Listener Groups (Inexperienced, Experienced, and
Individuals With Dysphonia) for Ratings of Overall Severity and Vocal Effort
Correlated Listener Groups
Overall
Severity (r)
Overall
Severity (r
2
)
Vocal
Effort (r)
Vocal
Effort (r
2
)
Inexperienced vs. experienced 0.94* 0.88 0.91* 0.82
Experienced vs. individuals with dysphonia 0.59* 0.35 0.56* 0.31
Inexperienced vs. individuals with dysphonia 0.48 0.23 0.51 0.26
*Correlation signicant at the P < 0.01 level (2-tailed).
Tanya L. Eadie, et al Experience and Judgments of Dysphonia 569
a spasmodic dysphonia population.
6
Collectively, results fromthe
present study suggest that when listeners judge speech samples
frompatients with larynx-based dysphonias, the exposure to pre-
vious dysphonic samples does not affect a listeners sensitivity to
the voice signal. These results appear to be robust compared with
previous studies. First, reliability and agreement measures among
all listeners judgments in the present study were strong, support-
ingthe internal validityof the measures. Second, the present study
included a larger number (n 10) of experienced voice clinicians
and inexperienced listeners (n 24) than most previous studies.
The inclusion of a larger number of participants increases the
external validity of the results. In addition, by including a large
number of listeners, groupaverages derivedfor eachspeaker sam-
ple more closely approximate those of a typical listener in that
group; that is, increased number of listeners in a group reduces
random error associated with perceptual outcomes and increases
the validity of the results.
8
Finally, voice clinicians who partici-
pated in the present study reported on average 17 years of experi-
ence with individuals with dysphonia (range: 638 years).
Although the average age of the experienced listener group
(avg 50.3 years) was greater than the inexperienced listeners
(avg 23.9 years), age was not a signicant predictor of voice
quality ratings, indicating that any differences between the two
groups (or lack thereof) were solely because of exposure to dys-
phonia. Although the results did not showthe effect of experience
on judgments of OS and perceived effort, future research should
include investigations of the effect of experience on different
voice dimensions, such as breathiness and roughness, which
may be differentially inuenced.
Relationships among listener judgments
Results from the present study showed strong relationships
between inexperienced and experienced listeners judgments
for all dimensions (8288% of shared variance). However, rela-
tionships between patients self-judgments and other listeners
were moderate (2536% of shared variance). These results
are consistent with those found by Eadie et al,
6
who performed
a similar study in spasmodic dysphonia. In their study, strong
relationships were found across auditory-perceptual dimen-
sions for experienced and inexperienced listeners; in contrast,
the relationship between perceived effort and severity was
weak for the individuals with dysphonia. Overall, results sug-
gest that although the group means for severity and effort
were not signicantly different across listeners, individuals
with dysphonia may be using different strategies to evaluate
perceived severity and effort than other listeners. This interpre-
tation is supported by similar variability in the use of the VAS
by all types of listeners (Figure 1), in combination with the
even, but random error about the regression lines in Figures 2
and 3. These results suggest that ratings were not affected by
bias in using the scales. In addition, data indicate that individ-
uals with dysphonia may be somewhat sensitive to the differ-
ences between dimensions (ie, perceived effort means
something different to a person with dysphonia than perceived
quality).
One explanation for this nding might relate to differences
among how different types of listeners make auditory-
perceptual judgments. For example, speakers have access to
cues above and beyond those found in the acoustic signal,
FIGURE 2. Relationships among the listener groups (inexperienced listeners, experienced listeners, individuals with dysphonia) judgments
of OS.
Journal of Voice, Vol. 24, No. 5, 2010 570
which are not available to other listeners. Thus, it may be
hypothesized that speakers use these additional cues when rat-
ing their own voices, even when they are asked to make judg-
ments on an audio signal.
6,22
For example, previous studies
have found that perceived effort may be differentially evaluated
by individuals who self-evaluate their own voices when com-
pared with other listeners.
6,22
These results are consistent
with those found in the present study, in which individuals
with dysphonia appeared to differentiate between judgments
of OS and perceived effort (r 0.38).
Brandt et al
22
suggested that, although speakers have access
to kinesthetic, physiologic (eg, increased tension in the vocal
folds and increased subglottal air pressure), and acoustic
cues, other listeners depend on increased high-frequency
energy found in the acoustic signal of effortful speech. They
suggested that some listeners also internalize what they hear
and relate their judgments to their own perceived effort in voice
production. These hypotheses also may explain why relation-
ships between judgments made by experienced clinicians and
individuals with dysphonia were empirically stronger in the
present study than relationships between individuals with dys-
phonia and inexperienced listeners. With increased experience,
clinicians may have learned to calibrate judgments of both OS
and effort and relate them to voice production in dysphonic
speakers. However, it is clear that even with experience, lis-
teners (vs. speakers) seemed to perceive OS and perceived
effort as relatively equivalent dimensions when performing
evaluations of the present studys speaker population. These
results may lead one to question whether perceived effort,
which has sometimes been equated with perceived strain,
23
is
a salient dimension that validly can be used by clinicians to
evaluate individuals with these types of voice disorders. These
issues need further investigation and clarication using physio-
logical, acoustical, and psychophysical (eg, multidimensional
scaling) methods.
Results from the present study have implications for models
of voice perception. For example, Kreiman et al
1
have proposed
that listeners make auditory-perceptual judgments based on
FIGURE 3. Relationships among the listener groups (inexperienced listeners, experienced listeners, individuals with dysphonia) judgments
of VE.
TABLE 4.
Pearson Correlation Coefcients Across Auditory-Perceptual Dimensions as a Function of Listener Group (Inexperienced
Listeners, Experienced Listeners, Individuals With Dysphonia)
Correlated Dimensions
Inexperienced
Listeners
Experienced
Listeners
Individuals
With Dysphonia
Overall severity vs. vocal effort 0.98* 0.96* 0.38
*Correlation signicant at the P < 0.01 level (2-tailed).
Tanya L. Eadie, et al Experience and Judgments of Dysphonia 571
internal standards that are affected by listener experience with
stimuli. In their model, Kreiman et al
1
suggest that the internal
standards of individuals with dysphonia should be mainly like
inexperienced listeners standards, because patients presum-
ably do not have much exposure to pathological voices other
than their own. Moderate relationships among self-rated dys-
phonia with other listeners judgments suggest that voice
patients internal standards may actually differ from other
listeners standards. Future studies should include patients
judgments of other speakers with dysphonia in addition to
self-judgments to help determine whether individuals with
voice disorders are more similar to experienced or inexperi-
enced listeners in making these types of evaluations.
Relationships between voice handicap index scores
and auditory-perceptual dimensions
In contrast to previous studies, listeners perceptions of dyspho-
nia similarly predicted patient-rated voice handicap, that is,
regardless of listener experience (ie, individuals with dyspho-
nia, inexperienced listeners, experienced listeners), relation-
ships between judgments of voice quality or effort and
patient-rated VHI scores were moderate (Table 5). The strength
of the relationships between other listeners (experienced and
inexperienced listeners) judgments and VHI totals were some-
what stronger than those found in previous studies. For exam-
ple, Eadie et al
6
found that inexperienced and experienced
listeners judgments of spasmodic dysphonia were only weakly
(r 0.300.38) related to VHI totals. Somewhat stronger rela-
tionships have been found between experienced clinicians
judgments and V-RQOL reported by individuals with benign
vocal fold lesions (r 0.440.48),
13,14
whereas weaker rela-
tionships have been found between inexperienced listeners
judgments of voice and V-RQOL.
6,12
Similar relationships
were found in this study, with experienced listeners judgments
being stronger predictors of VHI totals than inexperienced lis-
teners, although the differences were not tested for their clinical
or statistical signicance.
In the present study, the strongest predictor of VHI totals
made by other listeners was for judgments of OS made by
experienced voice clinicians (r 0.66). The stronger relation-
ship between experienced listeners judgments and VHI totals
compared with previous results may relate to the increased
number of experienced judges who participated in this
study.
3,13,14
As already explained, an increased number of lis-
teners also increases the number of judgments to base percep-
tual averages for each speaker sample (ie, representing the
typical listener) on; this procedure has been found to
decrease random errors in previous perceptual studies.
8
In addi-
tion, OS of dysphonia is a dimension typically evaluated in for-
mal rating scales of voice, and is usually the most reliably
judged dimension of dysphonia for individuals with most lar-
ynx-based voice disorders.
24
Thus, stronger relationships
between OS and VHI totals also may be a function of the clin-
ical population included in this investigation (ie, when com-
pared with individuals diagnosed with spasmodic dysphonia).
6
In this study, patient-perceived VE also was among the stron-
gest predictors of V-RQOL (r 0.68) in individuals with a vari-
ety of larynx-based voice disorders. These results are consistent
with those found in a previous study of ADSD, in which patient-
perceived VE was the only signicant predictor of VHI totals.
Thus, it appears that self-perceived effort is a measure which
is meaningful for individuals with dysphonia in general. These
results highlight the fact that although disruptions of voice qual-
ity might be of concern to some individuals with dysphonia,
perception of increased VE might be differentiated from voice
quality, and may be a more sensitive indicator of V-RQOL.
Given that one important goal of treating voice disorders is to
reduce the psychosocial impact, data from this study indicates
that patient-rated VE should be included as a part of standard
voice evaluation.
CONCLUSIONS
Auditory-perceptual ratings are used for assessing and measur-
ing treatment outcomes in voice disorders.
1,8
Many listeners
make these evaluations in clinical and research practice; how-
ever, the role of experience in making these judgments is not
clear. Results from this study indicate that a listeners experi-
ence with a voice disorder does not systematically inuence
a listeners sensitivity to dysphonia. However, strategies used
by individuals with dysphonia may differ from other listeners
when making judgments of OS or perceived effort. Regardless
of experience, listeners judgments of OS and effort may only
moderately predict V-RQOL. These results continue to support
the use of auditory-perceptual measures and V-RQOL as mean-
ingful, but complementary, measures of vocal function in indi-
viduals with voice disorders.
Acknowledgments
The authors wish to acknowledge fundingsupport of the Royalty
Research Fund at the University of Washington. They also thank
TABLE 5.
Pearson Correlation Coefcients of VHI Total Scores With Listeners Judgments Across Auditory-Perceptual Dimensions
Perceptual Dimension
Correlation of VHI Totals With
Inexperienced Listeners
Judgments
Correlation of VHI Totals With
Experienced Listeners
Judgments
Correlation of VHI Totals With
Dysphonic Individuals
Judgments
Overall severity 0.55* 0.66* 0.56*
Vocal effort 0.59* 0.63* 0.68*
*Correlation signicant at the P < 0.01 level (2-tailed).
Journal of Voice, Vol. 24, No. 5, 2010 572
all the participants in this study and the student researchers in the
Vocal Function Laboratory who helped with data collection.
REFERENCES
1. Kreiman J, Gerratt BR, Kempster GB, Erman A, Berke GS. Perceptual eval-
uation of voice quality: review, tutorial, and a framework for future
research. J Speech Hear Res. 1993;36:2140.
2. HiranoM. Clinical Examinationof Voice. NewYork: Springer-Verlag; 1981.
3. Lee M, Drinnan M, Carding P. The reliability and validity of patient self-
rating of their own voice quality. Clin Otolaryngol. 2005;30:357361.
4. Damrose JF, Goldman SN, Groessl EJ, Orloff LA. The impact of long-term
botulinum toxin injections on symptom severity in patients with spasmodic
dysphonia. J Voice. 2004;18:415422.
5. Laczi E, Sussman JE, Stathopoulos ET, Huber J. Perceptual evaluation of
hypernasality compared to HONC measures: the role of experience. Cleft
Palate Craniofac J. 2005;42:202211.
6. Eadie TL, Nicolici C, Baylor C, Almand K, Waugh P, Maronian N. Effect of
experience on judgments of adductor spasmodic dysphonia. Ann Otol Rhi-
nol Laryngol. 2007;116:695701.
7. Eadie TL, Doyle PC. Direct magnitude estimation and interval scaling of
pleasantness and severity in dysphonic and normal speakers. J Acoust
Soc Am. 2002;112:30143021.
8. Shrivastav R, Sapienza CM, Nandur V. Application of psychometric theory
to the measurement of voice quality using rating scales. J Speech Lang
Hear Res. 2005;48:323335.
9. Sapir S, Aronson AE, Thomas JE. Judgment of voice improvement after re-
current laryngeal nerve section for spastic dysphonia: clinicians versus
patients. Ann Otol Rhinol Laryngol. 1986;95:137141.
10. Jacobson BH, Johnson A, Grywalski C, et al. The Voice Handicap Index
(VHI): development andvalidation. AmJ SpeechLangPathol. 1997;6:6670.
11. Hogikyan ND, Sethuraman G. Validation of an instrument to measure voi-
ce-related quality of life (V-RQOL). J Voice. 1999;13:557569.
12. Ma EPM, Yiu EML. Voice activity and participation prole: assessing the
impact of voice disorders on daily activities. J Speech Hear Lang Res.
2001;44:511524.
13. Murry T, Medrado R, Hogikyan ND, Aviv JE. The relationship between rat-
ings of voice quality and quality of life measures. J Voice. 2004;18:
183192.
14. Behrman A, Sulica L, He T. Factors predicting patient perception of dys-
phonia caused by benign vocal fold lesions. Laryngoscope. 2004;114:
16931700.
15. Chang A, Karnell MP. Perceived phonatory effort and phonation threshold
pressure across a prolonged voice loading task: a study of vocal fatigue.
J Voice. 2004;18:454466.
16. Stewart CF, Allen EL, Tureen P, Diamond BE, Blitzer A, Brin MF. Adduc-
tor spasmodic dysphonia: standard evaluation of symptoms and severity.
J Voice. 1997;11:95103.
17. Ventry IM, Weinstein BE. Identication of elderly people with hearing
problems. ASHA. 1983;25:3742.
18. Fairbanks G. Voice and Articulation Drillbook. 2nd ed. New York: Harper
and Row; 1960.
19. DeKrom G. Some spectral correlates of pathological breathy and rough
voice quality for different types of vowel fragments. J Speech Hear Res.
1995;38:794811.
20. Verdolini K, Titze IR, Fennell A. Dependence of phonatory effort on hydra-
tion level. J Speech Hear Res. 1994;37:10011007.
21. Chan KM, Yiu EM. The effect of anchors and training on the reliability of
perceptual voice evaluation. J Speech Lang Hear Res. 2002;45:111126.
22. Brandt JF, Ruder KF, Shipp T. Vocal loudness and effort in continuous
speech. J Acoust Soc Am. 1969;46:15431548.
23. American-Speech-Language-Hearing-Association. Consensus Auditory-
Perceptual Evaluation of Voice (CAPE-V). Rockville Pike, MD; 2002.
24. Webb AL, Carding PN, Deary IJ, MacKenzie K, Steen N, Wilson JA. The
reliability of three perceptual evaluation scales for dysphonia. Eur Arch
Otorhinolaryngol. 2004;261:429434.
Tanya L. Eadie, et al Experience and Judgments of Dysphonia 573