2004, 29, 538–544
The reliability and sensitivity to change of acoustic measures of voice quality
P. N . C A R D I N G , * I . N . S T E E N , A . W E B B , * K . M A C K E N Z I E , à I . J . D E A RY § & J . A . W I L S O N *
*Department of Otolaryngology, Head and Neck Surgery, University of Newcastle, UK, Centre for Health Services Research, University of Newcastle, Newcastle - Upon - Tyne, UK, àDepartment of Otolaryngology, Head and Neck Surgery, Glasgow Royal Inﬁrmary, Glasgow, UK, and §Department of Psychology, University of Edinburgh, Edinburgh, UK
Accepted for publication 1 October 2003
carding p.n., steen i.n., webb a., mackenzie k., deary i.j. (2004) Clin. Otolaryngol. 29, 538–544
The reliability and sensitivity to change of acoustic measures of voice quality
This study aimed to evaluate the reliability and sensitivity to change of three commonly used acoustic parameters as measured by the Multi-Dimensional Voice Programme (MDVP); jitter, shimmer and noise-to-harmonic ratio. A total of 231 subjects’ voices were recorded and analysed. The sample comprised 145 dysphonic patients who received intervention (surgery or voice therapy), 36 dysphonic patients who received no intervention, and 50 non-dysphonic (normal) subjects. All voices were recorded and analysed on two occasions (before and after treatment, or test–retest assessment) using a standard procedure. These data were analysed using standard psychometric procedures for assessing reliability and responsiveness. The acoustic analysis measures demonstrated poor to moderate reliability and effect size with respect to their sensitivity to change. Caution should be exercised in the injudicious use of computer-based acoustic analysis systems as an isolated measure of voice outcome in any clinical trial of interventions aimed at improving voice quality. Keywords voice disorders dysphonia voice quality acoustic measurement perturbation
The need for instrumental ‘objective’ assessment of voice quality can be seen in the increasing use of acoustic analysis methods for clinical diagnosis, outcome measurement and research. Acoustic analysis of the vocal signal has been applied to almost every area of voice care including the evaluation of surgical procedures,1–5 voice therapy,6,7 radiotherapy,8,9 medical therapy, 10–12 screening of laryngeal diseases,13,14 and vocal pathology differential diagnosis.15–17 However, there remains no standardization of technique methodology and considerable variability in which acoustic parameters to measure (as demonstrated by the references above). Even in one computer analysis system (e.g. The Computerized Speech Laboratory), the clinician is faced with a perplexing number of parameters from which to select. One useful means of informing the choice would be robust evidence of a parameter’s reliability and sensitivity to change as applied to a typical voice pathology clinical
Correspondence: Paul Carding, Department of Otolaryngology, Head and Neck Surgery, Freeman Hospital, Newcastle upon Tyne, NE7 7DN, UK (e-mail: email@example.com).
population. However, this information is surprisingly sparse and incomplete.18–20 This study aimed to evaluate the discriminatory power, reliability, and effect size of the sensitivity to change of three commonly used acoustic parameters as measured by the Multi-Dimensional Voice Programme (MDVP); jitter, shimmer and noise-to-harmonic ratio (NHR).
The patient sample comprised three cohorts: 1. A total of 145 subjects complaining of hoarseness and attending two university otolaryngology clinics (Newcastle and Glasgow) who received an intervention [90 had speech and language therapy (SLT) and 55 had surgery]: Their voice was assessed on two occasions, once before and once after the intervention. Fourteen of these subjects also participated in the test–retest assessment of the acoustic measures. This retest was performed within 2 h following the initial assessment and prior to any intervention.
Ó 2004 Blackwell Publishing Ltd
A total of 50 subjects with a normal voice: The voices of these subjects were also recorded twice within 2 h of each other.3 Sensitivity and responsiveness to change were assessed by comparing change in mean scores before and after interventions known to be effective in improving voice quality with the sd of baseline scores and the sd of change scores.0)
All subjects 231 50. patients were instructed to produce a prolonged /a:/ sound at a comfortable pitch and attempting to keep the sound as stable as possible for »6 s (or as long as they could manage).9) (13. Sample demographics are given in Table 1.4) 29 (80. 29. One purpose of this study was to determine the extent to which acoustic analysis could be applied to a range of dysphonic voices commonly encountered in clinical practice.8) 127 (70.Change of acoustic measures of voice quality 539
2. A total of 36 dysphonic subjects who received no intervention: All these subjects participated in the test–retest assessment of the acoustic measures. Sample demographics Cohort Subjects with abnormal voice Intervention cohort Sample size Mean age (range) Sex [n (%)] Male Female Diagnosis [n (%)] Organic Non-organic Movement disorder Systemic disease Not recorded Treatment [n (%)] Voice therapy Surgery 145 52. data analysis The distributions of the acoustic indices were assessed graphically and through consideration of summary statistics. NJ.
One main issue is the proportion of voices for which the Computer Speech Laboratory (CSL) acoustic analysis is even possible. USA).6) 17 (47.7) 0
All 181 52.0) (12. Patients were allowed to practice the task several times before the recorded sample /a:/ was performed. Discriminatory power of the indices was assessed by calculating conﬁdence intervals for the differences between groups in mean scores using normal distribution theory and application of the central limit theorem. 538–544
90 (62.4 (23–76) 19 (38.2) 2 (5. Measures of frequency perturbation (jitter). USA) and a Sony electret microphone (omnidirectional and sensitivity of 60 dB).4) 98 (67.9) (39. A total of 86 subjects (36 non-intervention dysphonics and 50 normal subjects) were also recorded a second time within 2 h of the ﬁrst recording following an identical recording procedure. The microphone-to-mouth distance was 10 cm and careful positioning minimized aerodynamic noise.0 (20–88) 7 (19.4) (13. No effort was made to control the absolute sound pressure levels of phonation. It was recognized that some voices may be too severely dysphonic (too aperiodic) to produce meaningful acoustic analysis values (i.0) 31 (62.7 (17–88) 54 (29.7) (41.6) 158 (68.5 (17–87) 47 (32.6) 6 (16.6) 46 57 23 18 1 (31. The whole vowel sample was digitized into a computer and the exact middle second of the prolonged vowel sample was identiﬁed and analysed using the MDVP of The Computer Speech Laboratory Model 4300B (Kay Elemetrics. NJ.2 (17–88) 73 (31.6) 11 (30. Lincoln Park.3)
Subjects with normal voice 50 41. Clinical Otolaryngology. the two assessment recordings were made within 2 h of each other to reduce the effect of the inherent variability of voice quality in dysphonic patients. These
Table 1.2) 57 74 25 24 1 (31. However.2 The difference between two groups in mean score divided by the standard deviation (sd) of individual scores was used as a measure of effect size. The analyses of test–retest reliability and responsiveness to change were restricted to subjects for whom there were no missing data.5)
measures represent three of the most commonly used acoustic parameters from the MDVP. Sony Corporation. These data were used for test–retest reliability analysis.9)
Ó 2004 Blackwell Publishing Ltd. 3. type III acoustic voice signals21). The analysis sampling rate was set at 25 kHz. The intraclass correlation coefﬁcient was used as a measure of test–retest reliability. All patients were recorded in a sound-proof room (ambient noise <40 dB) using a Sony DCT690 DAT recorder (sampling frequency ¼ 48 kHz.e.1) 55 (37. amplitude perturbation (shimmer) and NHR were noted. The number of subjects for whom an assessment was attempted and the number of subjects who
Non-intervention cohort 36 53.6) (16. Allandale.
73 (8.98 (2.25 (0. number of subjects able to provide an analysable voice sample.7) 4.35) 14 (100) 13 (92.74) 88 (95. the box extends from the 25th percentile (lower edge) to the 75th percentile (upper edge).06) 132 (93.75 (7.9) 110 (77.23 (0.49 (2.01 (7.13) Subjects with normal voice 50 (50) 49 (98.0) 188 (83. Clinical Otolaryngology.0) 88 (95.56) 43 (91.0) 43 (91. Carding et al.5–20.83) 134 (95.72) 45 (100) 8. (Mean scores for pre.27) 45 (90.5 IQRs from the edge of the box) are individually plotted (as a circle).75 (2. Although the scores tend to be positively skewed the distributions suggest that it would be reasonable to consider parametric procedures such as the t-test when analysing these variables.)
distribution of scores The ﬁrst three plots in Fig.43 (2. and mean scores by index by assessment by cohort Cohort Subjects with abnormal voice Intervention cohort Total sample size (n in retest study) Initial acoustic assessment Number of subjects [n (%)] Jitter Analysable [n (%)] Mean (SD) Shimmer Analysable [n (%)] Mean (SD) NHR Analysable [n (%)] Mean (SD) Retest assessment Number of subjects [n (%)] Jitter Analysable [n (%)] Mean (SD) Shimmer Analysable [n (%)] Mean (SD) NHR Analysable [n (%)] Mean (SD) Post-treatment assessment Number of subjects [n (%)] Jitter Analysable [n (%)] Mean (SD) Shimmer Analysable [n (%)] Mean (SD) NHR Analysable [n (%)] Mean (SD) 145 (14) 142 (97.08) 105 (76.78) 29 (87.08) 45 (100) 0.18) 141 (97. Finally.24) Non-intervention cohort 36 (36) 33 (91.5) 11.01) 134 (78.42 (5.12) 33 (91.95 (7.6) 0.24) 49 (100) 0.88) 29 (87.N.09 (2. The horizontal line towards the centre of each box denotes the median score.03 (2.and post-treatment assessments based on completed pairs are given in Table 5.540 P.07 (9.5) 11.9) 0.9) 5.98) 13 (92.9) 5.6).2) 134 (95. Subjects in the intervention cohort were more likely to be able to provide an ‘analysable’ voice sample after treatment than before (odds ratio ¼ 7.13) 110 (77.3.0) 45 (100) 4.68) 183 (83. and thus indicates the interquartile range (IQR).4) 11.10 (3.5) 6.7) 0.72 (7.0) 49 (100) 4.63 (4.78) 43 (91. All normal voices were analysable for all acoustic parameters. 29.33 (0.
Ó 2004 Blackwell Publishing Ltd. Number of assessments attempted.5) 5. Mean and sd scores for the three elements of the acoustic assessment are given.31 (5. The lines emerging from the boxes (‘whiskers’) give an indication of the range of the data.9) 11.7) 30 (90. Outliers (observations which are more than 1.05 (2.7) 139 (79.8) 0.24 (0.27 (0. comprising 181 dysphonics and 50 normal people.71) 49 (100) 8. Similarly.4) 6. Jitter.05 (6. 1 show the distribution of jitter.9) 4.6) 0.24 (0.9) 5.05 (4.9) 12.9) 0.82 (6.6) 0. acoustic retest values are available for all subject cohorts. A total of 231 subjects were recruited to the study. 95% CI: 2.0) 9.9) 10. Approximately 20% dysphonic voices were not analysable at initial acoustic assessment.79) 30 (90.9) 11. These acoustic indices do not suffer from any obvious ﬂoor or ceiling effects.36) 188 (83. 538–544
.33 (0.74 (6.7) 29 (87.36 (0.
Table 2.91) 30 (90.59) 88 (95. posttreatment acoustic values are presented for the patients who received intervention.32) 47 (94. shimmer and NHR scores obtained at the initial assessment.5) 0.10) All 181 (50) 175 (96.9) 0.85) 13 (92.65 (8.28) All subjects 231 (100) 224 (97.26)
were able to provide a voice sample that could be analysed is given in Table 2.7) 10. shimmer and NHR values at initial assessment can be compared for the intervention and non-intervention cohorts and for subjects with a normal voice.35 (0.0) 4.75) 139 (79.31) 92 (92.23 (0.35 (0.
43.43). Although these differences were statistically signiﬁcant (P < 0. reliability was at best moderate (for jitter and shimmer) and
Ó 2004 Blackwell Publishing Ltd. It does not discriminate between the non-intervention (dysphonic) cohort and the normal voice (effect size ¼ 0. Both cohorts of subjects with abnormal voices have signiﬁcantly higher mean shimmer scores than the cohort of subjects with a healthy voice (effect sizes ¼ 0.5 0 SLT Surgery Post-intervention
Shimmer by intervention by time
NHR by intervention by time
Figure 1. Distributionof jitter.Change of acoustic measures of voice quality 541
Initial jitter score 30 25 20 15 10 5 0 Intervention Non-intervention Normal voice 20 10 0 50 40 30
Initial shimmer score
Intervention Non-intervention Normal voice
Jitter by cohort
Initial noise to harmonic ratio 2 1. the actual effect sizes involved were fairly small (0. Clinical Otolaryngology.15) cohort and it does not discriminate between the two cohorts of subjects with abnormal voices (effect size ¼ 0.
discriminatory power The jitter score discriminates only between intervention subjects and subjects with a normal voice (effect size ¼ 0. the reduced mean score in the non-intervention cohort is probably just a chance occurrence. For subjects with dysphonia. 538–544
. 29.5 1 .5 1 . Subjects with an abnormal voice in the non-
intervention cohort had a lower mean NHR than subjects in the other two cohorts. shimmer and NHR scores by cohort and intervention. reliability The intraclass correlation coefﬁcient was used to assess test– retest reliability (Table 4).28).05).5 0
Shimmer by cohort
Intervention Non-intervention Normal voice
NHR by cohort
Pre-intervention 30 25 20 15 10 5 0 SLT Surgery 20 10 0 SLT Surgery Post-intervention 50 40 30 Pre-intervention Post-intervention
Jitter by intervention by time
Pre-intervention 2 1. Table 3).38 and 0.36 in both cases).
ES1 ¼ change in mean score divided by sd of baseline scores (Cohen’s D). Change in acoustic indices following intervention Mean score (sd) Intervention SLT Surgery Index Jitter Shimmer HNR Jitter Shimmer HNR n 63 63 58 39 39 38 Pre-treatment 6. because change applies to reliable variance and not error variance. jitter. shimmer and NHR.3 to 0.91 (0.10 ()0. 538–544
. according to Titze23 further limits the utility and reliability of perturbation measurement.542 P.96) 0.2) (8.4) (8.5 0. 29.4) )1.
Table 5.8 ()5.23 ()0.34) Post-treatment 4.23 to 0.18) )0.9 ()4. 1 give a visual indication of the change in each of the acoustic indices following each of the interventions.18 0.82)
This study aimed to evaluate the discriminatory power.88 (0.
Table 3.18 to 0.63) Normal voice (n ¼ 45) 0.36 0. and the effect size of the sensitivity to change of three commonly used acoustic parameters as measured by the MDVP.8 ()5.1 to 1.48) )0. responsiveness to change The lower set of box plots in Fig.01.11 ()0.70) 0.47 0.68 to 2.22 to )0.46 (0.32 0.7 0.35 to 0. for each intervention. Reliability of the measures is better when used with subjects with a normal voice but this is the sample to whom the measures are least relevant.1 ()3.3) (0.11 to 0.11 to 0.4) (0.28 (0.74) 0. previous authors have suggested a close correlation between speciﬁc acoustic
poor for NHR. Carding et al.4) )0. reliability.42) 2. There was a small to moderate size effect of all three acoustic parameters following effective clinical intervention (as measured by perceptual ratings and voice quality of life measures).32 0.34 ES2 0.25 0.49) )0.1 9.1 9.85) 0.2 7. Test–retest reliability coefﬁcients (95% CI) Subject group Acoustic measure Jitter Shimmer Noise to harmonic ratio Dysphonic (n ¼ 39) 0. This point will be discussed later. It is also very likely that the data sample included a large number of type II acoustic voice signals which.34 to 5.11) Non-intervention group versus normal subjects 0.01) Effect size ES1 0.1) (9.34 to 3.34 6.68 (0.5 to )0.5 to )0.32 (4. **P < 0.1) (6. The effect sizes are similar – mostly around one-third of a sd – for both interventions but the difference in mean scores reaches statistical signiﬁcance only for SLT (most likely because of the larger sample size).73 (0.4) (0.11 ()0.32) (6.11 ()0. The clinical sample comprised unselected dysphonic outpatients and hence the study was able to determine the applicability of these acoustic measures in a normal clinical setting.40 (0.51 to 0.and post-treatment.33 (0.31
n ¼ number of subjects with a score both pre. Our results report that »20% of dysphonic voices cannot be analysed at all for acoustic measurement.58 to 0.63 to 1.00) Intervention subjects versus non-intervention subjects 1.23 4.36 ()3.28) (4.34 0.6)** )2. It is possible that we could have achieved larger size effects with sub-groups of data.24 to 0.00 ()0.5 0. For example.18) Difference (95% CI) )2.55 (0. there was a reduction in scores following the intervention (Table 5). The modest period-free reliability will limit the attainment of the best possible change effect size estimate.N. Pairwise comparison of the three cohorts for each acoustic measure: difference in mean scores (95% CI) Comparison Acoustic measure Jitter Shimmer Noise-to-harmonic ratio Intervention group versus normal subjects 1.02 to 0.24)
Table 4. Clinical Otolaryngology.93) 3.27 0. The changes in mean scores represent effect sizes that can be considered as small to moderate. These voices represent type III signals23 which have insufﬁcient periodic structure to allow the software to produce any measurement of cycle-to-cycle perturbation.26 0. Ó 2004 Blackwell Publishing Ltd.1 12.21 to 0. For each index.06 to 6. *P < 0.28 0.67) 0.23 (4.73 to 3.5) (0.05.65 ()0.03)* )0.3 0. ES2 ¼ change in means score divided by sd of change scores.002) )1.
The sensitivity-to-change data illustrate the danger of routinely applying isolated acoustic measurement values to measure change in voice quality following intervention. This poor reliability may partly explain the large sd of these acoustic indices which will. One main aim of this report will be to discuss the relative sensitivity of each measure in comparison with each other. however. assessment of sensitivity to change of these acoustic measures is difﬁcult to do without a gold standard against which to make comparisons.25 Therefore. perceptual and patient self-report voice outcome measurements.20 However. affect the measurements of sensitivity to change. Finally. A total of 32 patients had type III voice signals prior to treatment and could not therefore be analysed. Our results suggest that perturbation measures may have speciﬁc value with particular voice quality types. there are many sources of potential variability within the process of acoustic analysis. previous work by our group. these relationships remain unclear and recent work22 suggests that the mapping of acoustic features to different voice qualities is highly complex and multidimensional. The actual change was signiﬁcant only for jitter and shimmer scores following voice therapy but statistical signiﬁcance depends upon the sample size and there were more subjects who received voice therapy than surgery.26
Ó 2004 Blackwell Publishing Ltd. The mean scores for all three acoustic measures decreased following intervention. Furthermore. It is well recognized that automated acoustic analysis systems may provide perturbation measures even when the underlying signal is too ‘noisy’ to provide reliable period detection. the effect size was approximately one-third of sd. exclusion of ‘atypical characteristics’ in the selected sample and selection of the ‘most stable’ segment. Clinical Otolaryngology. there is a lot of inherent variability even within subjects resulting in large sd for each measure in comparison with mean scores. the acoustic analysis measures used in this study did not demonstrate adequate reliability. in turn.e. We are currently preparing publication of further data that allows comparison of acoustic. 29. this relationship requires further investigation. This is a fairly small change in populations where there is reason to expect considerable improvements in voice quality. Acoustic measurements of the speech signal are perhaps best used as evidence to support clinical impression and. However. The evidence from this study suggests that these acoustic measures have limited sensitivity to change when used indeterminately on all voices. Furthermore. in our study. where appropriate. Furthermore. Titze23 provides guidelines to determine the appropriateness of voice samples for acoustic analysis. We recognized that dysphonic voices are inherently variable and therefore attempted to minimize this effect by allowing subjects to practice prior to the recording sample and to perform the repeated measures within 2 h.Change of acoustic measures of voice quality 543
parameters and certain perceptual voice quality types (e.18–20 This reliability would appear to be best in near-to-normal (type I) voice signals but decrease further as the voices become more severely dysphonic (less periodic). It is possible that recording multiple voice samples and averaging the results (as suggested by Titze23) may improve the reliability scores and this could be the subject of a further study. this practice is against clinical practicality and routine use.19 From the analysis of the reliability of our data.g. For each intervention and for each acoustic measure. It is important to note that. Voice outcome measurement should reﬂect the multidimensional aspects of voice quality27 and techniques should be selected speciﬁcally to elucidate and support other clinical measures. a number of patients demonstrated clinically signiﬁcant improvement following intervention but could not be measured using acoustic analysis methods. it is not possible to recommend acoustic analysis as an independent measure of voice outcome. because the measures are not particularly reliable. Reliability of the test–retest data were poor for the clinical population and moderate for the normal voice samples. One possible reason for the small effect size is that. using the same data set suggests limited reliability of perceptual ratings of voice quality. It is possible that the reliability of acoustic measurement of the voice may be enhanced by adopting strict signal selection criteria and multiple sampling analysis.24. In conclusion. effect size or sensitivity to change to recommend their routine use to measure outcome in clinical trials of interventions aimed at improving voice quality. signals that contain intermittancy and aperiodic segments) the routine applicability of acoustic analysis should be further questioned. correlation analysis between the two independent measures of voice quality may only serve to further compound the matter. jitter and roughness). Within-subject (repeated measures) analysis has also been found to be only moderately reliable even in nondysphonic control subjects. Previous studies have examined the intraand inter-system reliability of a number of computerized acoustic analysis programmes and found them to be only moderately reliable. 538–544
. 91% of these patients had analysable voice signals post-treatment but could not be included in these data for obvious reasons. However. Our study would suggest that these selection criteria would render a large proportion of our clinical voice sample as unsuitable for acoustic analysis. These include visual inspection of the signal to ensure sufﬁcient signal periodicity. as part of an integrated and complementary clinical evaluation. given the likelihood
that the clinical voice sample contains a large number of type II acoustic voice signals23 (i. This has additional implications in the potential employment of acoustic analysis measurement as a clinical measure of intervention outcome.26 This acoustic analysis data are totally erroneous.
& Rihkanen H.. Ryker K. (2001) The effect of levodopa on vocal function in Parkinson’s disease. 868–872 13 Balter M. (1996) Intrasubject variability of objective voice measures. 45. & Mitev P.. J. 121. Clin.. A voice analysis system for the screening of laryngeal diseases. 3rd edn.. Head Neck Surg. (2001) Postthyroidectomy superior laryngeal nerve injury.. 5. Kreiman J. Rad. (1997) Objective evaluation of the quality of voice following radiation therapy for T1 glottic cancer. 256–268 21 Bough D. Ortega A.. 38. et al.. (1997) Acoustic analysis of pathological voices. (2000) An Introduction to Medical Statistics. 121. Cherovac C. 418–429 20 Gonzalez J. J.M. 502– 520 19 Dixon H.S. Arch. Phys. Voice 15. (2001) Acoustic analysis of voice quality with or without false vocal fold displacement after cordectomy.. Otol. 189–193 18 Sapienza C.. (1989) Design and Analysis of Reliability Studies: the Statistical Evaluation of Measurement Errors.N. Ruiz P.. (2000) Acoustic analysis after radiotherapy in T1 vocal cord carcinoma: a new approach to the analysis of voice quality. (1999) Effects of voice by endolaryngeal microsurgery. (2000) Vocal violence in actors: an investigation into its acoustic consequences and the effects of hygienic laryngeal release training. Ann. Dworkin J. IA 24 Frohilch M.. & Chapman K. 395–412 10 Rovirosa A. Rhinol. 451–454 7 Uloza V. Piazza C.. Whurr Publishers. et al. OtoRhinoLaryngol.R. Laryngol. J. Sataloff R. 1829–1843
14 Sanabria J. J.W. Younes A. Rhinol. et al. Deary I.S. 149–153 12 Hamdan A.M. 24. (2001) Functional results after type I thyroplasty with the Montgomery’s prosthesis. Walton S. Biol.E. The reliability of three perceptual evaluation rating scales for dysphonia. Mihcaelis D. 172–175 27 Carding P. (2000) Evaluating Voice Therapy. (2000) Acoustic voice analysis by means of the hoarseness diagram. Oxford University Press. Sharara A. et al. Voice 14.C. (2001) Acoustic analysis in the diagnosis of voice disorders in children.E. 312–315 8 Roy N. Eng. acoustic analysis of sustained phonation and reading.. Cervera T.G.R.. Bellomom A. Provenzano L.E. London 3 Dunn G.. et al. Voice 14.
1 Bertino G. 538–544
.W. Hammarberg B. Arch. Acta Otorhinolaryngolologica Italica 21. Neuropharmacol. J.N. 131–140 2 Bland M. (2001) Inhaled beclomethasone dipropionate improves acoustic measures of voice in patients with asthma.. OtoRhino-Laryngol. (2002) Acoustic voice analysis: reliability of a set of multidimensional parameters..S. Bio.. J. Acta Otorhinolaringologica Espanola 53. Rad.. 16. Clinical Otolaryngology. et al. 24. Policarpo M. 29. Chest 120. Gerratt B. 258. (in press) 26 Hillman R. Curr. Ped. Int. Med. (2001) Effects of aggressive therapy on laryngeal symptoms and voice characteristics in patients with gastro-esophageal reﬂux. Adams S. 166–174 22 Rabinov R. Measuring the Effectiveness of Treatment. 47.. Otol. London 4 Reijonen P. & Miralles J. (1995) Workshop on Acoustic Voice Analysis. (2001) Efﬁcacy of a behavioural based voice therapy protocol for vocal nodules.. Strube H. & Zietels S.. J. Ferrero F. Med. 215–230 9 Holmberg E. Reinhardt J. Voice 10. 419–429 16 Boyanov B.. 11. (2002) A computer system for acoustic analysis of pathological voices and laryngeal disease screening.B. University Press. Otolaryngol. Martinez-Celdran E. Gutierrez R. et al.. Lehikoinen-Soderlund S. Speech Hear Res. (2000) Adductor spasmodic dysphonia and muscular tension dysphonia.. 74–82 17 Niedzielska G. 706–720 25 Webb A. et al. 99–102 15 Hadjitodorov S..544 P. J.G. 256. Speech Hear Res. Phys. Carding P.L. Arch. Eur. Voice 15. Carding et al. 26–32 23 Titze I. (1997) Appropriate use of objective measures of vocal function in the multidisciplinary management of voice disorders. Eur. et al.
This study was funded by a Health Services Research Grant from the Wellcome Trust. 73–79 11 Aref A. Opin. Hillman R.I. 43. (2002) Results of fascial augmentation in unilateral vocal fold paralysis. & Murray T. Montgomery W. 156–162 6 Aluffi P.M. et al. Laryngol. Acta OtoLaryngol.. Otol. Devi S. (1995) Comparing reliability of perceptual ratings of roughness and acoustic measures of jitter. London
Ó 2004 Blackwell Publishing Ltd. Eng. 256–259 5 Peretti G. Summary Statement. Oncol. Head Neck Surg. Eur. Int.T. et al. National Centre for Voice and Speech. J. 57.. & Hadjitodorov S.H. (1999) Dysphonia and delayed food allergy: a provocation/neutralization study with videolarngostroboscopy. Oncol.L. OtoRhinoLaryngol.. & Bless D.. et al. The University of Iowa.