J Neurol (2008) 255:1545–1548

DOI 10.1007/s00415-008-0978-4 ORIGINAL COMMUNICATION

S. Fonville Accuracy and inter-observer variation

H. B. van der Worp
P. Maat in the classification of dysarthria
M. Aldenhoven
A. Algra from speech recordings
J. van Gijn

Received: 2 January 2008
Received in revised form: 3 April 2008
Accepted: 18 April 2008
Published online: 3 September 2008
S. Fonville, MD
Dept. of Neurology
Onze Lieve Vrouwe Gasthuis
Amsterdam, The Netherlands
Dr. H. B. van der Worp, MD, PhD (쾷) ·
M. Aldenhoven, MD · A. Algra, MD, PhD ·
J. van Gijn, MD, FRCP, FRCPE
Dept. of Neurology
Rudolf Magnus Institute of Neuroscience
University Medical Center Utrecht
Heidelberglaan 100
3584 CX Utrecht, The Netherlands
Tel.: +31-30/2509111
Fax: +31-30/2522782
E-Mail: h.b.vanderworp@umcutrecht.nl
P. Maat, MD
Dept. of Neurology
St. Elisabeth Hospital Tilburg
Tilburg, The Netherlands
A. Algra, MD, PhD
Julius Centre for Health Sciences
and Primary Care
University Medical Center Utrecht
Utrecht, The Netherlands
■ Abstract Background Dysar-
thria may be classified as flaccid,
spastic, ataxic, hypokinetic, chore-
atic, dystonic, or mixed. We hypo-
thesized that in routine neurologi-
cal practice the reliability and
accuracy of perceptual analysis
alone in the classification of dysar-
thria is low and that this classifica-
tion is mainly based on the clinical
context rather than on the percep-
tion of speech. We therefore stud-
ied the accuracy and the inter-
observer agreement in the
classification of dysarthrias on the
basis of perceptual analysis alone.
Methods Seventy two neurologists
and neurological trainees classified
recorded speech samples of 100 pa-
tients as flaccid, spastic, ataxic,
extrapyramidal, or mixed dysar-
thria, or as not dysarthric. All
observers were blinded to the pa-
tients’ final diagnosis, which was
based on all clinical features and
investigations. In the analysis the
observers were arranged in eight
groups of nine observers, or four
paired groups with similar levels of
clinical experience. Together, the
observers in a given group rated all
100 recordings. Results The accu-
racy of the classification was poor
(35 % were classified correctly) and
the inter-observer agreement be-
tween paired groups low (κ 0.16 to
0.32). The level of experience in
neurology did not have a signifi-
cant influence. Conclusion Neuro-
logical trainees as well as experi-
enced neurologists have great
difficulty in identifying specific
types of dysarthria on the basis of
perceptual analysis alone. In clini-
cal practice this probably means
that most neurologists will classify
dysarthria in the context of other
features from neurological exami-
nation or ancillary investigations.
■ Key words dysarthria ·
interobserver variation ·
neurological examination

flaccid, spastic, ataxic, hypokinetic, choreatic, dystonic,

Introduction
Dysarthria is defined as a disturbance in the pronuncia-
tion of spoken language. Articulation, respiration, reso-
nance, prosody and phonation may be compromised by
a range of inborn or acquired structural anomalies af-
fecting the vocal tract, but the term “dysarthria” is usu-
or mixed [7]. This classification rests largely on clinico-
anatomic distinctions proposed by Darley, Aronson and
Brown [3, 4].
The perception of dysarthria depends on the analytic
ear of the clinician. In practice, however, dysarthria will
usually be classified in the context of other disturbances
found at neurological examination or imaging. Although
JON 2978

ally confined to articulation disorders caused by a neu- neurological textbooks stress the importance of dysar-
rological impairment. Dysarthria may be classified as thria as an aid in the diagnostic process, little is known
1546
about the accuracy of perceptual analysis alone for the
identification of different types of dysarthria. Among
experienced speech/language pathologists and graduate
students, a study from the USA found an overall accu-
racy in identifying the type of dysarthria in only 19 %
and 56 % of the cases [9]. An accuracy of 71 % was found
in a French study, but the investigators only distin-
guished hypokinetic, spastic, and ataxic dysarthrias [1].
We hypothesized that in routine neurological prac-
tice the reliability and accuracy of perceptual analysis to
identify the different types of dysarthria are low. We
therefore studied the inter-observer variability and ac-
curacy of perceptual analysis alone to assess its value as
an aid in the diagnosis and classification of dysarthria.
In addition, we investigated the influence of neurologi-
cal experience.
Patients and methods
■ Patients
We included native Dutch-speaking patients with any type of dysar-
thria who presented at the Departments of Neurology or Neurosur-
gery of the University Medical Center of Utrecht, the Netherlands,
between August 2002 and January 2004. We excluded all patients suf-
fering from aphasia or apraxia of speech, patients with inborn or ac-
quired anatomical deformities in the oropharyngeal region, and pa-
tients who were unable to read a text. We also included control patients
without dysarthria. All patients gave consent for participation in the
study. In every patient, two authors (second and last) decided the
‘true’ classification of the dysarthria by agreeing on the type of dys-
arthria based on perceptual analysis and comparing this with all
other clinical features and with the results of imaging and neurophys-
iological studies.
Approval from the local medical ethics review board was not re-
quired, given the present criteria.
■ Recordings
All subjects were asked to read a standardized Dutch text, entitled
“Loos alarm op het strand” (“False alarm on the beach”) and contain-
ing 142 words. Of all 172 samples recorded, 100 were selected on ac-
count of the technical quality of the recording, the audibility of the
speech disorder, and the presence of a certain clinical diagnosis. The
100 recordings were divided randomly into nine groups, each consist-
ing of either 11 or 12 recordings, which were transferred to separate
compact discs.
■ Observers
A total of 36 neurologists and 36 neurological trainees from three
regional hospitals and four university hospitals participated in the
study. On the basis of their clinical experience, we divided them into
four pairs of observer groups, each group consisting of 9 observers:
1) two groups of neurological trainees with less than 36 months of
experience, 2) two with neurological trainees who had more than 36
months of experience (in the Netherlands, neurological training takes
6 years), 3) two with neurologists who had less than 10 years of expe-
rience after board certification, and 4) two with neurologists having
more than 10 years of experience after board certification. None of
them had a specific training in distinguishing different types of dys-
arthria. All observers were blinded to the patients’ clinical diagnosis.
Each observer listened in a separate session to the recordings on
one compact disc and classified the types of dysarthria by filling out
a standard form. Each recording could be repeated once before a final
classification was made. Within any observer group, each of the nine
compact discs was evaluated by a single member of that group. Each
compact disc was therefore evaluated by eight different observers
(four pairs with a similar level of experience).
■ Classification
Because of the low prevalence of the choreatic and dystonic dysar-
thrias in our centre, we combined hypokinetic, choreatic, and dys-
tonic dysarthrias into one category of ‘extrapyramidal’ dysarthrias. A
sixth group of samples without dysarthria was added. This group in-
cluded patients with different Dutch dialects, as well as patients with
speech disorders without a physical cause (such as those with a con-
version disorder). Each sample could therefore be classified as a “flac-
cid”, “spastic”, “ataxic”, “extrapyramidal”, or “mixed dysarthria”, or as
“no dysarthria”. Patients with mixed dysarthria were those with dif-
ferent types of dysarthria combined, such as a combined flaccid and
spastic dysarthria in patients with ALS.
■ Statistical analysis
For each level of neurological experience, the inter-observer agree-
ment between the two groups was calculated with kappa statistics [2,
5]. In general, κ represents the strength of agreement above the agree-
ment expected by chance according to the following rating: < 0, poor;
0–0.20, slight; 0.21–0.40, fair; 0.41–0.60, moderate; 0.61–0.80, substan-
tial; and > 0.80, excellent [8]. We used the formula [95 % confidence
interval (CI) = κ ± (1.96*standard error)] to calculate the correspond-
ing 95 % CIs.
Results
On the basis of the combination of all clinical findings
and the results of ancillary investigations, 19 of the 100
samples were classified as flaccid dysarthria, 17 as spas-
tic, 15 as ataxic, 12 as extrapyramidal, and 20 as mixed.
Seventeen samples were rated as not dysarthric. Overall,
the observers classified 35 % of the samples correctly
(Table 1).
Neurological trainees with less than 36 months of ex-
perience and neurologists with more than 10 years of
experience classified the samples slightly more often in-
correctly, but differences with the other observer groups
were not statistically significant. There was no consis-
tent pattern in the confusion of one type of dysarthria
for the other (Table 2).
Inter-observer agreement in the classification of dys-
arthria was fair in neurological trainees with more than
36 months of experience and slight in the other observer
groups (Table 3). There was considerable overlap be-
tween the confidence intervals.
 1547

Table 1 Accuracy of the classification of dysarthria

related to the level of neurological experience Final classification* Neurological Neurological Neurologists Neurologists Mean
trainees trainees < 10 years of > 10 years of (%)
< 36 months of > 36 months of experience (%) experience (%)
experience (%) experience (%)

Flaccid 40 40 53 37 42
Spastic 29 38 38 29 34
Ataxic 33 40 47 37 39
Extrapyramidal 29 42 25 20 29
Mixed 13 10 20 13 14
No dysarthria 50 56 50 59 54

Total 32 37 39 33 35
* The final classification of dysarthria was based on the combination of all clinical findings and ancillary investiga-
tions

Table 2 Confusion matrix: classification of dysar-

thria by all observers combined compared with the Final classification* Classification by observers
final classification
Flaccid Spastic Ataxic Extrapyra-midal Mixed No dysarthria
(%) (%) (%) (%) (%) (%)

Flaccid 42 6 16 11 13 13
Spastic 17 34 11 16 15 8
Ataxic 10 18 39 4 12 18
Extrapyramidal 15 12 16 29 5 24
Mixed 19 25 20 15 14 9
No dysarthria 14 19 8 4 3 54
* The final classification of dysarthria was based on the combination of all clinical findings and ancillary investiga-
tions

Table 3 Inter-observer agreement in the classification of dysarthrias

Observer groups Cohen’s
Kappa
Neurological trainees < 36 months of experience 0.16
Neurological trainees > 36 months of experience 0.32
Neurologists < 10 years of experience 0.20
Neurologists > 10 years of experience 0.19
Discussion
The 72 neurologists and neurologists in training par-
ticipating in the present study classified only 35 % (range
32–39 %) of the speech samples correctly, which demon-
strates that classifying dysarthria based on perceptual
analysis alone is highly unreliable. This is in keeping
with a previous study in the USA of two groups of expe-
rienced speech/language pathologists, who correctly
identified only 19 % and 56 % of the dysarthria types,
despite the use of a standardized form including differ-
ent characteristics of each type of dysarthria [9]. A
slightly higher accuracy was found in a French study, but
this included only hypokinetic, spastic, and ataxic dys-
arthrias [1].
In a British study testing the inter-rater reliability for
95 % CI the clinical assessment of stroke, the inter-observer vari-
ation was smaller among physicians than among medi-
0.06–0.27
cal students [6]. We had also expected to find a large ef-
fect of the variable ‘clinical experience’ on the accuracy
0.21–0.44
of the classification as well as on the inter-observer vari-
0.09–0.31
ation. Although there was a trend towards a better inter-
0.09–0.30 observer agreement among neurological trainees with
more than 36 months of experience, neither the accu-
racy of identification nor the inter-observer agreement
differed significantly between groups. This finding is
difficult to explain. One factor may be that regardless of
the level of experience, neurologists tend to classify dys-
arthria in the context of other characteristics found on
neurological examination or imaging, and they do not
receive specific training or accumulate experience in the
identification of speech disorders from perceptual anal-
ysis alone.
Remarkably, the same applies to speech pathologists,
because in the American study mentioned above no dif-
ference was found in the accuracy of identification be-
tween those with a minimum of five years of clinical
experience and those in training [9].
The observers distinguished 54 % (range 50–59 %) of
the subjects without dysarthria correctly from subjects
1548

with dysarthria. The fact that this number did not ap-
proach 100 % in experienced neurologists is probably
explained by the inclusion of subjects with different
Dutch dialects and those with speech disorders without
a localisable (physical) cause. ‘Mixed dysarthria’ ap-
peared to be the most difficult type to classify (14 % clas-
sified correctly, range 10–20 %), probably because par-
ticipants classified speech samples according to the type
of dysarthria that predominated.
In conclusion, neurological trainees as well as expe-
rienced neurologists have great difficulty in identifying
specific types of dysarthria on the basis of perceptual
analysis alone. In clinical practice this means that most
neurologists will classify dysarthria in the context of
other features from the neurological examination or an-
cillary investigations. In general clinical practice, the
classification of the type of dysarthria should therefore
not be overestimated.
■ Conflict of interest The authors declare no conflict of interest.
■ Acknowledgement We thank all patients, neurologists, and neuro-
logical trainees who participated in this study.

References
1. Auzou P, Ozsancak C, Jan M, Menard JF,
Eustache F, Hannequin D (2000) Evalua-
tion of motor speech function in the
diagnosis of various forms of dysar-
thria. Rev Neurol (Paris) 156:47–52
2. Cohen J (1960) A coefficient of agree-
ment for nominal scales. Educ Psychol
Meas 20:37–46
3. Darley FL, Aronson AE, Brown JR
(1969) Differential diagnostic patterns
of dysarthria. J Speech Hear Res 12:
246–296
4. Darley FL, Aronson AE, Brown JR
(1969) Clusters of deviant speech
dimensions in the dysarthrias. J Speech
Hear Res 12:462–496
5. Fleiss JL (1971) Measuring nominal
scale agreement among many raters.
Psychol Bull 76:378–382
6. Hand PJ, Haisma JA, Kwan J, Lindley RI,
Lamont B, Dennis MS, Wardlaw JM
(2006) Interobserver agreement for the
bedside clinical assessment of sus-
pected stroke. Stroke 37:776–780
7. Kent RD (2000) Research on speech
motor control and its disorders: a
review and prospective. J Commun
Disord 33:391–428
8. Landis JR, Koch GG (1977) The mea-
surement of observer agreement for
categorial data. Biometrics 33(1):
159–174
9. Zyski BJ, Weisiger BE (1987) Identifica-
tion of dysarthria types based on per-
ceptual analysis. J Commun Disord 20:
367–378