AJSLP
Research Article
Part of Speech Production in Patients
With Primary Progressive Aphasia:
An Analysis Based on Natural
Language Processing
Charalambos Themistocleous,a Kimberly Webster,b
Alexandros Afthinos,a and Kyrana Tsapkinia,c
Background: Primary progressive aphasia (PPA) is a
neurodegenerative disorder characterized by a progressive
decline of language functions. Its symptoms are grouped
into three PPA variants: nonfluent PPA, logopenic PPA,
and semantic PPA. Grammatical deficiencies differ depending
on the PPA variant.
Aims: This study aims to determine the differences between
PPA variants with respect to part of speech (POS) production
and to identify morphological markers that classify PPA
variants using machine learning. By fulfilling these aims, the
overarching goal is to provide objective measures that can
facilitate clinical diagnosis, evaluation, and prognosis.
Method and Procedure: Connected speech productions
from PPA patients produced in a picture description task
were transcribed, and the POS class of each word was
estimated using natural language processing, namely, POS
M
orphosyntactic deficits have been identified as
one of the key symptoms of primary progres-
sive aphasia (PPA; Mesulam et al., 2014, 2012;
Mesulam & Weintraub, 2014), a neurodegenerative condi-
tion that results in a considerable deterioration of speech
and language skills (Thompson, Lukic, et al., 2012; Thompson
a
Department of Neurology, Johns Hopkins School of Medicine,
Baltimore, MD
b
Department of Otolaryngology, Johns Hopkins Medicine,
Baltimore MD
c
Department of Cognitive Science, Johns Hopkins University,
Baltimore MD
Correspondence to Charalambos Themistocleous: cthemis1@jhu.edu
Editor-in-Chief: Melissa Duff
Editor: Dirk B. Den Ouden
Received September 16, 2019
Revision received February 14, 2020
Accepted May 13, 2020
https://doi.org/10.1044/2020_AJSLP-19-00114
Publisher Note: This article is part of the Special Issue: Select Papers
From the 49th Clinical Aphasiology Conference.
466 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021 • Copyright © 2020 American Speech-Language-Hearing Association
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
tagging. We then implemented a twofold analysis: (a) linear
regression to determine how patients with nonfluent PPA,
semantic PPA, and logopenic PPA variants differ in their
POS productions and (b) a supervised classification analysis
based on POS using machine learning models (i.e., random
forests, decision trees, and support vector machines) to
subtype PPA variants and generate feature importance (FI).
Outcome and Results: Using an automated analysis of a
short picture description task, this study showed that content
versus function words can distinguish patients with nonfluent
PPA, semantic PPA, and logopenic PPA variants. Verbs
were less important as distinguishing features of patients
with different PPA variants than earlier thought. Finally, the
study showed that among the most important distinguishing
features of PPA variants were elaborative speech elements,
such as adjectives and adverbs.
& Mack, 2014). PPA is characterized by substantial vari-
ability of symptoms as an effect of the degree of neuro-
degenerative decline, underlying pathology, and areas of
brain damage (Mesulam, 2013; Thompson, Lukic, et al.,
2012; Thompson & Mack, 2014). To understand the symp-
toms, recently established consensus criteria classify pa-
tients into three main PPA variants: the nonfluent PPA
variant (nfvPPA), the logopenic PPA variant (lvPPA), and
the semantic PPA variant (svPPA; Gorno-Tempini et al.,
2011; Gorno-Tempini & Pressman, 2016).
Morphosyntactic production is key for language
communication, and as it becomes impaired in PPA, it can
provide objective markers for the classification of patients
with PPA into variants, for clinical evaluation, prognosis
of the condition, and intervention. Patients with nfvPPA
are characterized by sparsity of function words and ab-
normal syntax due to peak atrophy at the posterior inferior
frontal gyrus (Broca’s area; Gorno-Tempini et al., 2011;
Thompson et al., 1997, 2013; Thompson, Cho, et al., 2012).
Disclosure: The authors have declared that no competing interests existed at the time
of publication.
 Patients with lvPPA are characterized by deficits in single-
word retrieval and naming. Repetition of long words and
phrases is also impaired, and their speech is characterized
by phonological errors, such as paraphrases. Atrophy in
individuals with lvPPA is related to brain volume loss in
the left posterior temporal and inferior parietal regions
(Gorno-Tempini et al., 2008, 2011; Gorno-Tempini &
Pressman, 2016). Patients with svPPA are characterized
by impaired object knowledge, impaired confrontation
naming, and limited single-word comprehension. They
are also characterized by impaired semantic memory of
familiar objects, “empty speech” in verbal production,
and surface dyslexia, as an effect of damage at the anterior
temporal lobe (Gorno-Tempini et al., 2011; Gorno-Tempini
& Pressman, 2016; Wilson et al., 2010).
Morphosyntactic production in PPA has been ana-
lyzed qualitatively (Kertesz et al., 1998; Mesulam, 2001)
and quantitatively (Thompson, 2012; Thompson et al.,
1997, 2014; Wilson, Brandt, et al., 2014; Wilson, DeMarco,
et al., 2014; Wilson et al., 2010). Several early studies
have shown that patients with nfvPPA produce propor-
tionally fewer function (also known as grammatical) words
than content words from patients with lvPPA and svPPA
(Thompson, Cho, et al., 2012; Thompson & Mack, 2014;
Thompson et al., 2013; Wilson et al., 2010). Function
words are words with grammatical meaning and syntactic
purpose, such as conjunctions and articles. In contrast,
content words have semantic meaning, such as nouns and
verbs. However, studies, such as Fraser et al. (2014), did
not find a pronounced deficit in function words in individ-
uals with nfvPPA in comparison to controls and patients
with svPPA.
In addition to a specific deficit in content versus func-
tion words, patients with nfvPPA have been characterized
by selective impairment in naming verbs relative to naming
nouns (Thompson, Lukic, et al., 2012). The opposite of
this pattern was found to exist in patients with svPPA, who
are characterized by selective impairment in naming nouns
relative to verbs (Cappa et al., 1998; Daniele et al., 1994;
Hillis et al., 2004; Kertesz et al., 1998; Thompson, Lukic,
et al., 2012). Overall, patients with PPA display substantial
heterogeneity with respect to content word production, espe-
cially nouns and verbs. For example, Thompson, Cho, et al.
(2012) did not find significant differences between patients
with nfvPPA, patients with lvPPA, and healthy controls in
noun-to-verb ratio. Only patients with svPPA displayed dif-
ferences in noun-to-verb ratio from both patients with
nfvPPA and controls (Thompson, Cho, et al., 2012).
Previous studies examined connected speech produc-
tions in PPA using picture description tasks (see Faroqi-
Shah et al., 2020; Wilson et al., 2010). Connected speech
reveals a wide array of naturally occurring morphosyn-
tactic content and function word productions, such as the
production of adjectives, adverbs, pronouns, and conjunc-
tions, and can determine the distribution of content words
and function words across PPA variants (Ash et al., 2006;
Graham et al., 2004; Kavé et al., 2007; Meteyard & Patterson,
2009). Nevertheless, the manual analysis of connected
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
speech productions is labor intensive, as it requires the
identification of the part of speech (POS) for every word
that appears in the transcriptions of connected speech.
Our study offers a less labor-intensive automatic analy-
sis of POS from connected speech productions and em-
ploys the analysis to classify patients into lvPPA, svPPA,
and nfvPPA. Automatic methods were employed success-
fully in earlier research for Alzheimer’s disease diagnosis,
classification, and so forth (Bucks et al., 2000; Fraser
et al., 2018; Guinn & Habash, 2012; Orimaye et al., 2014;
Rentoumi et al., 2014; Themistocleous et al., 2018; Thomas
et al., 2005) and in the automatic identification of patients with
svPPA, nfvPPA, and healthy controls (Fraser et al., 2014).
The aims of this study were (a) to quantify the pro-
duction of POS in patients with different PPA variants,
(b) to determine whether and which POS markers can clas-
sify patients into PPA variants automatically, and (c) to
provide a methodological approach that can be employed
in clinical settings to enable clinicians to quantify mor-
phosyntactic productions in patients with nfvPPA, lvPPA,
and svPPA in a timely manner. Our overarching goal is to
identify objective morphosyntactic measures that can be
employed in clinical diagnosis, evaluation, and prognosis.
We employed samples of connected speech from picture
description tasks from all three PPA variants and auto-
matically extracted measures of POS production. Subse-
quently, we employed computational morphosyntactic
analysis, namely, morphosyntactic tagging, to identify the
morphosyntactic categories automatically.
Method
Participants
Recordings of the Cookie Theft picture description
task from 52 patients diagnosed with PPA were analyzed
(Goodglass et al., 2001). Diagnosis of PPA was provided
by neurologist(s) according to consensus criteria (Gorno-
Tempini et al., 2011). Variant subtyping was based on
magnetic resonance imaging results, clinical and neuropsy-
chological examination, and speech and language evalua-
tions following consensus criteria. Nineteen participants were
subtyped as nfvPPA, 18 were subtyped as lvPPA, and 15
were subtyped as svPPA. Unclassified patients with PPA and
apraxia of speech–only cases were excluded (see Table 1).
Data were collected as part of a clinical trial conducted
at Johns Hopkins University (NCT02606422; Tsapkini
et al., 2018). Inclusion criteria comprised progressive speech
and language deficits, native English speakers, minimum of
high school education, and normal or corrected vision.
Exclusion criteria comprised absence of any developmental
or nondegenerative neurological disorder (e.g., stroke)
and absence of primary deficits in other cognitive domains,
such as memory. The present data are from baseline neuro-
psychological and neurolinguistic evaluations.
One-way analysis of variance tests showed that there
were no significant differences between patients with nfvPPA,
lvPPA, and svPPA for gender (F = 5.82, df = 2, p =.09), age
Themistocleous et al.: Part of Speech Production in PPA 467
 Table 1. Demographic and neuropsychological data for each participant group.

Variable nfvPPA lvPPA svPPA

Demographics
No. of participants 19 18 15
Gender 9F 10M 8F 10M 7F 8M
Age 69 (6) 68 (8) 67 (6)
Education 16 (1) 17 (2) 16 (2)
Severity
Language severity 2.77 (0.5) 1.39 (0.8) 2.27 (0.6)
Total severity 10 (6.5) 11 (6.3) 12 (5.5)
Language production
Boston Naming Test (30) 17 (5.6) 14 (6.5) 13 (11.1)
Category fluency (fruits, animals, vegetables) 22.9 (10.1) 20.8 (14.3) 7.8 (6.3)
Learning and memory
Rey Auditory Verbal Learning Test 33 (14) 19 (12) 18 (8)

Note. Recorded materials and speakers’ demographic information. Shown are the number of participants,
gender, age, education, language severity, and total severity. The numbers in parentheses indicate the standard
deviation. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant;
svPPA = semantic PPA variant; F = female; M = male.

(F = 0.354, df = 2, p = .705), education (F = 0.162, df = 2,
p = .853), language severity (F = 0.154, df = 2, p =.86),
and total severity (F = 1.162, df = 2, p = .33; see Table 1).
Participants’ age, education, and language severity were
considered in data analysis (Riello et al., 2018). Language
severity in Table 1 is a five-level rating subscale in the Clin-
ical Dementia Rating–Modified Frontotemporal Lobar
Degeneration Scale (Knopman et al., 2008). This scale was
found effective in representing the progression of language
deficits due to neurodegeneration, distinguishing patients
based on a corresponding language severity outcome measure
(Knopman et al., 2008). The Clinical Dementia Rating–
Modified Frontotemporal Lobar Degeneration Scale showed
that participants in this study were matched for overall lan-
guage severity.
Materials
The Cookie Theft picture description task was part
of the oral evaluation (Goodglass et al., 2001). This word-
less picture shows three people performing different activi-
ties: two children trying to steal cookies from a cupboard
as their mother is washing the dishes. A clinician presented
the picture to the participant and prompted the partici-
pant following the standard Boston Diagnostic Aphasia
Examination–Third Edition (Goodglass et al., 2001) instruc-
tions (“Tell me everything you see going on in this picture”).
The patient is instructed to speak in complete sentences,
making reference to the objects, people, and activities
shown. No prompts or cues are provided. Speech produc-
tions were audio-recorded with a digital audio recorder.
The audio recordings were all converted into 16000 Hz
mono format and transcribed using speech-to-text technol-
ogy implemented in an in-house software platform, Themis
(Themistocleous et al., 2018; Themistocleous & Kokkinakis,
2018). Twenty percent of the transcriptions were selected
randomly and evaluated by two independent evaluators.
The automatic transcriptions were compared word by word
with the transcripts of the two independent evaluators and
marked as true (“T”) or false (“F”) for each word based
on whether the word was correctly transcribed by the soft-
ware (T) or whether the software provided a different tran-
scription or no transcription (F). Using Cohen’s kappa,
the agreement of the automatic transcription with the first
transcriber was .95 ( p < .0001), and the agreement with
the second transcriber was .96 ( p < .0001); the agreement
between the two transcribers was .97 ( p < .0001). After
establishing that transcriptions were close to human tran-
scriptions, no further transcription modifications were
employed as we were interested to see how well the tran-
scriptions perform without manual modifications. Most
fillers (e.g., “um” and “uh”) were transcribed (and ana-
lyzed) but were not considered word utterances, that is,
they were not included in the total word count. Repetitions
and false starts were transcribed in Roman alphabet; repe-
titions of words were included in the total word count.
Neologisms were transcribed using standard orthography
in Roman alphabet. Table 2 shows the mean number of
words per variant. Individuals with nfvPPA, lvPPA, and
svPPA produced different numbers of words; individuals
Table 2. Number of words and mean number of words per participant
in each variant in Cookie Theft picture description task.
Materials produced nfvPPA lvPPA svPPA
Total no. of words 1,816 4,121 3,126
Mean words 121 (85)*,a 217 (165) 165 (97)
Note. PPA = primary progressive aphasia; nfvPPA = nonfluent
PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic
PPA variant.
a
Significantly different from the logopenic primary progressive
aphasia variant. *p < .05.

468 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 with nfvPPA produced fewer words than patients with
svPPA and lvPPA.
Natural Language Processing
and Morphosyntactic Measures
The transcripts were analyzed morphosyntactically
using the NLTK Python library (Bird et al., 2009). Morpho-
syntactic analysis involved the following steps: (a) identi-
fication of words and (b) sentence structure analysis and
POS identification in the text (for an overview of these
methods, see Jurafsky & Martin, 2009). From the tagged
corpus, both content words and function words were
counted, and their corresponding proportions of the main
POS categories (POS category/number of word count)
were estimated. In addition to proportions, the ratios of
all POS classes were calculated. Specifically, we calculated
the following POS measures:
1. Content words, namely, the proportion of nouns,
verbs, adjectives, and adverbs in the text; content
words have a specific lexical meaning as they refer
to objects, actions, and so forth.
2. Function words, namely, the proportion of conjunc-
tions (e.g., and, or, but); prepositions (e.g., in, of );
determiners and predeterminers (e.g., the, a/an, both);
pronouns such as he, she, and it and wh– pronouns
such as what, who, and whom; modal verbs (e.g.,
can, should, will); possessive ending (’s); adverbial
particles (e.g., about, off, up); and conjunction (e.g.,
infinitival to, as in to do).
3. The corresponding ratio of nouns and verbs with
other POS: noun/verb ratio, noun/adjective ratio, noun/
adverb ratio, noun/pronoun ratio, noun/preposition
ratio, noun/conjunction ratio, verb/adjective ratio,
verb/adverb ratio, verb/pronoun ratio, verb/preposi-
tion ratio, verb/conjunction ratio, adjective/adverb
ratio, and content word/function word ratio.
The analysis of ratios identifies the morphosyntactic
selectivity in the speech, as ratios can determine specific
preferences in the use of one POS category over another
POS category. For example, the reduction of attributes
assigned to nouns (e.g., “the red car” vs. “the car”) can
be expressed by the noun/adjective ratio. The verb/adverb
ratio provides information about the use of adverbs with
respect to verbs in PPA productions “(she) reads” versus
“(she) reads fast,” and the adverb/adjective ratio provides
information about the selective use of adverbs with respect
to adjectives. As the overall data size was small, POS ra-
tios provide further information that can improve the
automatic classification. The automatic POS tagging was
compared to the manual tagging provided by three raters
with background in linguistics. Cohen’s kappa shows that
there was substantial agreement between the automatic
morphological analysis and the manual one provided from
Rater 1 (κ = .983, p < .0001), Rater 2 (κ = .985, p < .0001),
and Rater 3 (κ = .983, p < .0001).
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
Statistical Analysis
To determine whether patients with nfvPPA, svPPA,
and lvPPA differ in the production of POS, generalized
linear models were conducted with the PPA variants as pre-
dictors and the POS measures as response variables (see also
Lien et al., 2017). Estimated marginal means (also known
as least squares means) provided contrasts of the three
PPA variants. The statistical analysis was performed in
R (R Core Team, 2016), using lmer4 (Bates et al., 2015),
lmerTest (Kuznetsova et al., 2017), and emmeans (Russell,
2018) packages in R.
Machine Learning Classification
To determine whether variants differed with respect
to morphosyntactic measures and identify the morpho-
syntactic measures that enable variant classification, a
machine learning classification task was conducted. Specif-
ically, a binary one-against-all classification was employed,
where the model distinguishes one variant from the other
two variants: nfvPPA versus (lvPPA and svPPA), svPPA
versus (nfvPPA and lvPPA), and lvPPA versus (nfvPPA
and svPPA). Three machine learning models were employed:
decision trees (DTs), random forests (RFs; Breiman,
2001; Breiman et al., 1984), and support vector machines
(SVMs; Cortes & Vapnik, 1995). DTs split the data in
a binary manner and provide a classification output rep-
resented as a dendrogram (tree diagram). The main dis-
advantage of DTs is that they often tend to create very
large and complicated trees that cannot generalize to un-
known data. RFs aim to address DTs’ disadvantage by
producing several tree models that are subsequently av-
eraged. SVMs employ hyperspaces, that is, demarcating
planes, to distinguish different groups, using both linear
and nonlinear kernels and often perform better than DTs
and RFs.
For the classification task, predictors were first indi-
vidually scaled, such that they ranged between zero and
one. To find the best performing models during evaluation,
several models were tested using multiple cross-validations
from 5-, 10- (…) to 35-folds (Hastie et al., 2009). We opti-
mized the number of trees in RF, the size of the kernels
in SVM models, and the initial random state for DTs. The
optimization process showed that the output of the 35 cross-
validation provided better results overall.
Machine learning models were trained and evaluated
35 times using a different set of 34 folds as a training set
and a different single fold as an evaluation set. During the
evaluation phase, the machine learning models provided
a prediction of the PPA variant of each patient. After a
trained model predicts the variant of a new and unknown
(to the model) patient, the prediction of the model is com-
pared to the actual PPA variant from the neurological
evaluation to estimate whether the model made a correct
prediction. From the comparison of the predicted variant
and the actual variant, the following cases are estimated:
(a) the true positive (TP) cases (i.e., the cases when patients
Themistocleous et al.: Part of Speech Production in PPA 469
 belong to variant X [e.g., nfvPPA] and the model predicts
correctly that they belong to that variant X), (b) the false
negative (FN) cases (the cases that the model predicts that
patients do not belong to variant X, when in actuality their
variant is X), (c) the false positive (FP) cases (the cases
that the model predicts that patients belong to variant X,
when in actuality they do not belong to variant X), and
(d) the true negative (TN) cases (i.e., when the model pre-
dicts that a patient does not belong to variant X correctly).
These cases are employed to calculate the following evalua-
tion metrics:
1. Precision (Precision = TP/[TP + FP]): The precision
is a measure of the true positives divided by the sum
of true positives and FPs; if there are many FPs, the
precision is lower.
2. Recall (Recall = TP/[TP + FN]): The recall is a mea-
sure of the true positives divided by the sum of true
positives and FNs, that is, a low recall indicates that
there are many FNs.
3. Accuracy and balanced accuracy (Accuracy =
[TP + TN]/[TP + TN + FP + FN]): The accuracy is
a measure of the correct predictions made by the
model divided by the total number of all cases. The
balanced accuracy is the average of recall obtained
on each class. The accuracy is a number from 0 to
100; an accuracy of 50 means that the model is cor-
rect half of the times, and a 100% accuracy means
that the model is always correct.
4. The F1 score (F1 score = 2 × [(Precision × Recall)/
(Precision + Recall)]): The F1 score measures the
weighted average of Precision and Recall. It is a score
from 0 to 100 that is similar to accuracy, which is
adjusted if the design is unbalanced.
5. ROC/AUC curve: The receiver operating character-
istic (ROC) and the area under the curve (AUC)
are two evaluation measures that display the per-
formance of a model. The ROC is a curve that is
created by plotting the TP rate against the FP rate.
The ROC/AUC measures the entire two-dimensional
area under the entire ROC curve from (0, 0) to (1, 1).
An optimal model has an ROC/AUC closer to 1,
whereas a bad model whose predictions are 100%
wrong has an ROC/AUC of 0 (see also Baldi et al.,
2000; Chicco, 2017, for a discussion on the selection
of evaluation metrics in machine learning).
Results
First, we present the overall distribution of morpho-
syntactic words into content words and function words.
Second, we present the distribution of POS features in PPA
variants and the results of the regression analysis. Third,
we present the machine learning models mentioned above
(RF, SVM, and DT) that classify patients into PPA vari-
ants using these features.
470 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
Distribution of Content Words and Function Words
Content Versus Function Word Production
In nfvPPA, the proportion of content words is higher
than the proportion of function words (see Figures 1A and
1B and Table 3). Patients with nfvPPA also produce a sig-
nificantly higher content word/function word ratio com-
pared to patients with svPPA and lvPPA, β = 1.720, t(47) =
13.97, p < .0001 (see Figure 2C and Table 3). Estimated
marginal means (also known as least squares means) for
pairwise differences for the content word/function word
ratio show significant differences for nfvPPA and lvPPA
( p < .05), and nfvPPA and svPPA ( p < .05), but there is
no significant contrast between lvPPA and svPPA. These
findings demonstrate that content word/function word ra-
tio distinguishes patients with nfvPPA from svPPA and
lvPPA.
Noun Production
Patients with nfvPPA produce proportionally more
nouns than patients with lvPPA and svPPA (see Figure 2).
Consequently, patients with PPA differ significantly in
the proportion of nouns, F(2, 47) = 15.7, p < .01. Both
lvPPA (t = −4.06, p < .01) and svPPA (t = −5.30, p < .01)
are statistically different from nfvPPA. However, pairwise
comparisons reveal that patients with lvPPA and svPPA
do not differ in the proportion of nouns (nonsignificant).
Verb Production
Patients with PPA do not significantly differ in verb
production (nfvPPA, M = 0.24, SD = 0.06; lvPPA, M = 0.26,
SD = 0.6; svPPA, M = 0.28, SD = 0.05). The comparison
of verb production in patients with nfvPPA and svPPA
shows that patients with nfvPPA produce fewer verbs (t =
2.2, p = .03), but the overall model was not significant (see
Figure 3).
Adjective and Adverb Production
Patients with nfvPPA produced fewer adjectives and
adverbs than patients with svPPA and lvPPA (see Figures 4A
and 4B). This resulted in an overall statistically significant
effect of the variant on both adjectives and adverbs (see
Figures 4A and 4B and Table 3).
Modal Verb, Conjunction, Determiner,
and Preposition Production
There were fewer productions with modal verbs in
patients with nfvPPA (M = 0.002, SD = 0.003); patients
with lvPPA (M = 0.012, SD = 0.014) and patients with
svPPA (M = 0.012, SD = 0.013) produced approximately
the same number of modal verbs. Patients with lvPPA
(β = 0.011, t = 2.94, p < .01) and patients with svPPA (β =
0.01056, t = 2.76, p < .01) produced significantly more
modal verbs than patients with nfvPPA. Although patients
with nfvPPA produce fewer conjunctions, determiners,
particles, and prepositions from patients with svPPA and
lvPPA, the statistical models for these morphonsyntactic
categories were not significant.
 Figure 1. Violin plots with the proportion of content (A), function words (B), and the content/function word ratio (C). The shape of the plots
indicates the probability density of the data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive
aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

The Appendix shows the noun/verb ratio, noun/
adjective ratio, noun/adverb ratio, noun/pronoun ratio, noun/
preposition ratio, and noun/conjunction ratio. Figure 5
shows the results for the noun/adverb ratio (Figure 5A) and
the verb/adverb ratio (Figure 5B). These ratios distinguish
patients with nfvPPA, svPPA, and lvPPA significantly
manifesting different structural effects in the patterns of
POS in PPA individuals.
Machine Learning Classification
To evaluate the contribution of POS markers for
identifying patients with different PPA variants and deter-
mine POS features that enable an automatic classification
of patients with nfvPPA, lvPPA, and svPPA, we employed
three machine learning models, RFs, SVMs, and DTs, in a
supervised one-against-all classification task. The accuracy,
balanced accuracy, F1 score, precision, recall, and ROC/
AUC produced in supervised one-against-all classification
from RF, SVM, and DT are shown in Table 4; the ROC/
AUC plots are shown in Figure 6. Patients with nfvPPA
and svPPA are identified better with 79% and 77% classifi-
cation accuracy, respectively, than patients with lvPPA who
are identified with 64% classification accuracy. Although,
SVM achieved 84% classification accuracy, it was accom-
panied with a lower ROC/AUC. As such, we are presenting
the feature hierarchy results from the RF model, which
indicated both the highest ROC/AUC and accuracy. It is

Themistocleous et al.: Part of Speech Production in PPA 471

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 Table 3. Regression models showing the statistical effects of primary progressive aphasia (PPA) variants on the distribution of part of speech
(POS).

POS PPA Estimate SE t Pr(>|t|) F test

Content word/function word ratio nfvPPA 1.72 0.123 13.97 < .0001*** F(2, 47) = 4.9, p < .01, R2 = 17%
lvPPA −0.466 0.174 −2.67 .0103*
svPPA −0.502 0.186 −2.69 .0098**
Content word nfvPPA 0.6401 0.0234 27.36 .0001*** F(2, 49) = 1.47, p < .24, R2 = 2%
lvPPA −0.0571 0.0335 −1.7 .095
svPPA −0.0342 0.0352 −0.97 .336
Function word nfvPPA 0.3800 0.0261 14.54 .0001*** F(2, 49) = 3.74, p < .05, R2 = 13%
lvPPA 0.0996 0.0375 2.66 .011*
svPPA 0.0706 0.0393 1.79 .079
Noun nfvPPA 0.3606 0.0284 12.71 .0001*** F(2, 49) = 11.4, p < .001, R2 = 32%
lvPPA −0.1468 0.0407 −3.61 .0007***
svPPA −0.1901 0.0427 −4.45 .0001***
Adverb nfvPPA 0.0337 0.0113 2.98 .0044** F(2, 49) = 6.99, p < .01, R2 = 22%
lvPPA 0.0315 0.0162 1.94 .0577
svPPA 0.0636 0.0170 3.73 .0005***
Adjective nfvPPA 0.0215 0.0142 1.51 .138 F(2, 49) = 3.16, p < .05, R2 = 11%
lvPPA 0.0232 0.0204 1.13 .263
svPPA 0.0539 0.0214 2.51 .015*
Pronoun nfvPPA 0.2243 0.0172 13.04 .0001*** F(2, 49) = 1.45, p = .25, R2 = 6%
lvPPA 0.0351 0.0247 1.42 .16
svPPA 0.0385 0.0259 1.49 .14
Verb nfvPPA 0.2243 0.0172 13.04 .0001*** F(2, 49) = 1.45, p = .24, R2 = 6%
lvPPA 0.0351 0.0247 1.42 .16
svPPA 0.0385 0.0259 1.49 .14
Determiner nfvPPA 0.1172 0.0130 8.99 .0001*** F(2, 49) = 2.66, p < .01, R2 = 10%
lvPPA 0.0344 0.0187 1.84 .072
svPPA −0.0076 0.0196 −0.38 .702
Modal verb nfvPPA 0.0017 0.0025 0.68 .501 F(2, 49) = 5.6, p < .001, R2 = 19%
lvPPA 0.0106 0.0035 3.02 .004**
svPPA 0.0098 0.0037 2.67 .01*
Preposition nfvPPA 0.1049 0.0112 9.41 .0001*** F(2, 49) = 1.53, p = .2, R2 = 6%
lvPPA 0.0279 0.0160 1.74 .088
svPPA 0.0119 0.0168 0.71 .481
Conjunction nfvPPA 0.0521 0.0078 6.69 .0001*** F(2, 49) = 0.78, p < .46, R2 = 3%
lvPPA 0.0063 0.0112 0.56 .58
svPPA 0.0147 0.0117 1.25 .22
Particle nfvPPA 0.0144 0.0037 3.88 .0003*** F(2, 49) = 0.54, p = .58, R2 = 2%
lvPPA −0.0039 0.0053 −0.74 .4657
svPPA 0.0017 0.0056 0.31 .76
Possessives nfvPPA 0.0040 0.0015 2.67 .01* F(2, 49) = 0.94, p = .39, R2 = 3%
lvPPA −0.0006 0.0021 −0.27 .79
svPPA −0.0029 0.0022 −1.32 .19

Note. The intercept of the model corresponds to the nonfluent PPA variant (nfvPPA). It also provides the standard error (SE), the t value, and
the p value, Pr(>|t|). The last column provides the overall analysis of variance score of the model and its corresponding R2. lvPPA = logopenic
PPA variant; svPPA = semantic PPA variant.
*p < .05, **p < .01, ***p < .001.

important to note that feature hierarchies depend on the
respective machine learning model. Therefore, different
models might result into different featural rankings. Regard-
less of rankings, they demonstrate the properties that
influence the model, which can be indicative of which mor-
phological features influenced the classification outcome
of the machine learning model.
The POS features with the greatest impact on the
machine learning classification model for the identification
of each PPA variant are shown in Table 5. The relative
importance of POS as a measure of PPA variance is calcu-
lated from the relative FI ranking, which is employed as
a decision node in trees generated by the RF algorithm. FI
indicates that highly ranked features have a greater contri-
bution to the final classification output; this denotes the
relative importance of a POS measure. The FIs shown in
Table 5 are standardized and add up to one. The highest six
rankings for each variant are shown.
This output demonstrates the morphosyntactic selec-
tivity in patients with nfvPPA, lvPPA, and svPPA variants.
The noun/adverb ratio; the proportion of nouns, adjectives,
and adverbs; and the noun/adjective ratio distinguish pa-
tients with nfvPPA from patients with the other two PPA
variants. The high predominance of content words, especially
nouns in the productions of patients with nfvPPA, demon-
strates the overall impact of content words for the classification

472 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 Figure 2. Violin plots with the distribution of nouns normalized to the total number of words produced by
patients with nfvPPA, lvPPA, and svPPA. The shape of the plots indicates the probability density of the
data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive
aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

of this variant. The machine learning model employed dif-
ferent POS measures for the identification of lvPPA. Specif-
ically, the proportion of determiners, the adverb/preposition
ratio, the verb/pronoun ratio, the content word/function
word ratio, and the proportion of adjectives contribute to
the classification of patients with lvPPA. Finally, the svPPA
classification model employed the proportions of nouns and
adverbs, the noun/adverb ratio, the adverb/preposition ra-
tio, and the noun/verb ratio as predictors. This FI suggests
that the proportion of nouns, the noun/adverb ratio, and
the noun/verb ratio have a high impact for the classification
of patients with svPPA.
Discussion
Grammatical production in PPA (Mesulam et al., 2014,
2012; Mesulam & Weintraub, 2014) varies considerably
between patients with different variants. Despite the long-
standing interest in PPA morphosyntactic production, re-
search has not led to conclusive morphosyntactic patterns
for the systematic classification of PPA variants. This arti-
cle aimed to address the need to determine morphosyntactic
production in patients with PPA using POS measures and
highlighted their role for subtyping PPA using machine
learning. This was accomplished by charting a procedure
for the quantitative assessment of connected speech and the
comparative evaluation of morphosyntax in all three PPA
variants using natural language processing. Our findings
show substantial heterogeneity of POS production in patients
with PPA. Grammatical selectivity is manifested in the dis-
tribution of content words (particularly nouns) and func-
tion words in distinguishing patients with different variants.
In short, automatic morphosyntactic tagging of connected
speech samples from an isolated picture task corroborated
earlier findings that were elicited using a time-consuming
battery of tasks. It also provided novel results that iden-
tify the effects of PPA on the production of adjectives and
adverbs and highlights differences in verb production in
patients with PPA compared to previous studies.
POS Production in nfvPPA
Patients with nfvPPA produced more content words
than function words. Specifically, nouns were more fre-
quent in patients with nfvPPA than in patients with lvPPA
and svPPA. This finding corroborates earlier studies that
suggest naming is not as impaired in nfvPPA as in svPPA
(i.e., making reference to objects, people, etc., in a picture
description task; e.g., Bastiaanse & Jonkers, 2012; Gorno-
Tempini et al., 2011). By contrast, the production of function
words was significantly diminished in patients with nfvPPA

Themistocleous et al.: Part of Speech Production in PPA 473

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 Figure 3. Violin plots with the distribution of verbs normalized to the total number of words produced by
patients with nfvPPA, lvPPA, and svPPA. The shape of the plots indicates the probability density of the
data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive
aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

compared to patients with lvPPA and svPPA. Our findings
showed that patients with nfvPPA produced fewer adjec-
tives and adverbs (recognized as content words) than patients
with svPPA and lvPPA. Patients with nfvPPA are known
to have impaired production of function words, so this find-
ing suggests that they have impairment with certain content
words as well, namely, adverbs and adjectives. However,
the reduced use of adverbs by patients with nfvPPA com-
pared to patients with svPPA does not imply that the lan-
guage of patients with nfvPPA is grammatically less dense
or less meaningful than that of patients with svPPA. In
fact, speech production in individuals with svPPA is often
indefinite and recognized as “empty.” The linear model
showed that the ratios of nouns with other POS (i.e., noun/
adjective, noun/preposition, noun/adverb, noun/verb) can
reliably and significantly distinguish patients with nfvPPA
from other variants. The contribution of these ratios to
the identification of patients with PPA was further sup-
ported by the subsequent classification task. This finding
highlights the significant predominance of nouns and the
interplay of nouns with adjectives, prepositions, adverbs,
and verbs as a characteristic to the speech of patients with
nfvPPA.
Given the central role of verbs in sentence structure,
it was surprising that patients with nfvPPA did not differ in
the production of verbs compared to patients with svPPA
and lvPPA. This suggests that verb production might not
be a good predictor for PPA variant classification (Fraser
et al., 2014; Kim & Thompson, 2000). However, when
verbs are taken together with nouns, the noun/verb ratio
is higher in patients with nfvPPA than in patients with
lvPPA and svPPA. This ratio can distinguish patients with
all three PPA variants (for a different finding, see Thompson,
Cho, et al., 2012).
Modal verbs (e.g., can, should) were analyzed sepa-
rately from matrix verbs, as modal verbs are grammatical
words that constitute a closed class and are employed in
speech to convey deontic modalities (e.g., the expression
of wishes, commands; Bybee & Fleischman, 1995; Bybee
et al., 1994). We found a different distribution of modal
verbs from matrix verbs in PPA. Patients with nfvPPA
produced significantly fewer modal verbs than patients with
lvPPA and svPPA. This finding on modal verbs can be inter-
preted as a manifestation of the agrammatism that charac-
terizes patients with nfvPPA (Knibb et al., 2009; Thompson
& Bastiaanse, 2012; Thompson, Cho, et al., 2012; Wilson
et al., 2010).
POS Production in svPPA
SvPPA is characterized by a combination of word
comprehension deficits, fluent aphasia, and a particularly

474 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 Figure 4. Violin plots with the distribution of adjectives (A) and the distribution of adverbs (B) normalized to the total number of words produced
by patients with nfvPPA, lvPPA, and svPPA. The shape of the plots indicates the probability density of the data. Dots indicate the means
and error bars indicate 95% confidence intervals. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic
PPA variant; svPPA = semantic PPA variant.

Figure 5. Violin plots showing the noun/adverb ratio (A) and the verb/adverb ratio (B). The shape of the plots indicates the probability density
of the data. Superimposed are error bars showing 95% confidence intervals from the mean (black circle in the middle). PPA = primary progressive
aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Themistocleous et al.: Part of Speech Production in PPA 475

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 Table 4. Mean and standard deviation (in square brackets) of model accuracy, balanced accuracy, F1 score, precision, recall, and area under the
curve (AUC) produced in supervised one-against-all classification from random forests (RF), support vector machines (SVM), and decision trees (DT).

PPA RF SVM DT

nfvPPA *Accuracy 79% [36%] 84% [31%] 79% [36%]

Balanced Accuracy 71% [40%] 71% [40%] 68% [41%]
F1 score 71% [41%] 70% [41%] 68% [41%]
Precision 71% [40%] 71% [42%] 71% [42%]
Recall 71% [40%] 70% [40%] 68% [41%]
AUC 78% [25%] 71% [36%] 78% [25%]
lvPPA *Accuracy 64% [42%] 57% [43%] 57% [42%]
Balanced Accuracy 64% [42%] 57% [43%] 57% [42%]
F1 score 63% [43%] 55% [44%] 58% [42%]
Precision 64% [45%] 56% [46%] 63% [46%]
Recall 64% [42%] 57% [43%] 57% [41%]
AUC 64% [23%] 57% [17%] 43% [32%]
svPPA *Accuracy 77% [40%] 69% [43%] 73% [38%]
Balanced Accuracy 77% [40%] 69% [43%] 73% [38%]
F1 score 76% [41%] 70% [43%] 72% [39%]
Precision 76% [42%] 72% [44%] 74% [41%]
Recall 77% [40%] 69% [43%] 73% [38%]
AUC 87% [22%] 81% [35%] 75% [25%]

Note. Shown with boldface are the models that provided the highest classification accuracy. The star symbol (*) indicates nonweighted
measures; weighted measures consider the unbalance in a design. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant;
lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Figure 6. Mean receiver operating characteristic (ROC) curves from
the classification models of nvfPPA, lvPPA, and svPPA. We run
three separate supervised one-against-all classification models
using rain forests (RFs), support vector machines (SVMs), and decision
trees (DTs). Shown in the figure is the ROC with the mean area under
the curve (AUC) for the DT, SVM, RF models and the corresponding
SD. The best classification is the one whose ROC approaches the
right upper corner of the plot and that has the highest AUC; here,
the best mean AUC resulted from RFs for nfvPPA, RFs for lvPPA,
and SVM models for svPPA.PPA = primary progressive aphasia;
nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant;
svPPA = semantic PPA variant.
severe anomia, whereas the acoustic or phonological char-
acteristics are viewed as unimpaired (Wilson et al., 2010).
We found that patients with svPPA produced fewer nouns
than patients with nfvPPA and lvPPA, which aligns with
our expectations: Patients with svPPA are characterized by
impaired naming abilities (Gorno-Tempini et al., 2011). Also,
they frequently substituted nouns with pronouns, which
was manifested in the noun/pronoun ratio (e.g., I see him).
This can be attributed to accommodation or compensation
strategies for noun (e.g., I see a child) production difficul-
ties (Wilson et al., 2010). As such, the noun/pronoun ratio
can be employed to distinguish patients with svPPA from
patients with nfvPPA.
Verb production did not constitute a marker that
could differentiate patients with svPPA from patients with
lvPPA and svPPA. This finding replicates earlier studies on
verb production in patients with svPPA (Fraser et al., 2014;
Thompson, Cho, et al., 2012). Patients with svPPA pro-
duced more adverbs than patients with nfvPPA and lvPPA.
Adverbs modify the sentence and/or the verb phrase by
providing information about manner, place, time, frequency,
degree, and so forth. Adverbs were found to be important
in distinguishing individuals with svPPA from healthy con-
trols, but not from nfvPPA (Fraser et al., 2014). Unlike
patients with nfvPPA, patients with svPPA produced a
higher number of function words than patients with other
PPA variants (Meteyard & Patterson, 2009).

POS Production in lvPPA

Patients with lvPPA were found to differ significantly
from nfvPPA with regard to function words, modal verbs,
and nouns. Patients with lvPPA produced more modals

476 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
 Table 5. Feature importance (FI) of the classification models for patients with nfvPPA, lvPPA, and svPPA from random forests.
nfvPPA
Ranking Features FI
1 Noun/adverb ratio 0.11
2 Nouns 0.09
3 Adjectives 0.07
4 Adverbs 0.07
5 Noun/adjective ratio 0.06
6 Content/function word ratio 0.04
Note. The FI ranking is a standardized measure with positive values that sum to 1; the table shows the first six highest ranked features.
PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.
and function words compared to patients with nfvPPA.
Interestingly, patients with lvPPA produced morphosyntac-
tic constituents that overlapped with both patients with
nfvPPA patients and patients with svPPA (i.e., their pro-
portions were usually in-between those with svPPA and
nfvPPA). Therefore, it is not surprising that the classifi-
cation accuracy of patients with lvPPA was lower—just
64% classification accuracy—than in patients with nfvPPA
(71% classification accuracy) and svPPA (77% classifica-
tion accuracy).
Diagnostic Utility of the Automatic Analysis of POS
The automatic analysis of POS employed in this study
and in previous studies (e.g., Fraser et al., 2014) highlights
the contribution of computational methods for analyzing
language production in patients with PPA and bridging the
gap between computational linguistic analysis and manual
evaluation of language productions in PPA. In clinical set-
tings, the automatic analysis of POS in the form of a com-
puter application enables physicians, neuropsychologists,
and speech-language pathologists to elicit POS measures of
connected speech, discourse, and so forth. At the same time,
it provides objective markers that estimate patients’ gram-
matical competence. Clinicians can employ these objective
markers for diagnosisng PPA, subtyping variants, and esti-
mating linguistic changes due to speech-language therapy
and/or the natural progression of degenerative condition.
It also enables clinicians to measure additional POS catego-
ries that can inform individualized therapeutic programs
(e.g., targeting increased use of specific POS productions).
Finally, the methodological approach of this study as a
diagnostic tool can be employed in other conditions affect-
ing language, such as aphasia due to stroke (Fyndanis &
Themistocleous, 2019).
Limitations and Future Directions
A major limitation in determining the difference
between PPA variants in POS production is the type of
task. Grammatical productions, specifically the distribution
of POS, verb tense, and so forth, depend on the nature
of the task. A common criticism of picture description
tasks (e.g., Cookie Theft) that instruct patients to discuss
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
lvPPA svPPA
Features FI Features FI
Determiners 0.10 Nouns 0.12
Adverb/preposition ratio 0.08 Noun/adverb ratio 0.09
Verb/pronoun ratio 0.07 Adverbs 0.07
Content/function word ratio 0.07 Adverb/preposition ratio 0.05
Adjectives 0.06 Noun/verb ratio 0.05
Noun/verb ratio 0.06 Noun/adjective ratio 0.05
everything they see is that patients are inclined to elicit
labeling rather than narration and verbs produced are
mostly in the present tense (past tense and future tense are
relatively uncommon). Storytelling and discourse settings
(e.g., free conversation) can facilitate the production of
POS and also differ in their POS distributions as shown
by several studies of speech genres (Bhatia, 1993; Swales,
1990). Future research is important to employ computa-
tional analysis of morphosyntactic production in different
types of discourse settings.
With regard to the aim of classifying PPA variants
based on morphological (POS) markers, this study is lim-
ited by the small corpus used. Cookie Theft description
of 52 participants with PPA may be a respectable sample
for PPA given the rarity of the syndrome, but it is a small
sample for totally unbiased unsupervised machine learn-
ing methods. A larger corpus would enable us to character-
ize POS productions with greater precision and reduce
variability. A related methodological limitation is that the
machine learning models do not classify all PPA variants
at the same time but classify one variant against the other
two. The decision for one against all classification was ne-
cessitated by the limited number of data.
Future work should employ unsupervised machine
learning models that offer simultaneous classification of all
PPA variants against each other. Outputs should be used
in different types of discourse settings, and multifactorial
designs should be implemented with predictors that include
speech acoustics, grammar, and so forth, to capture both
speech and overall language deficits.
Acknowledgments
We are grateful to our participants and referring physicians
for their interest in our study. We thank Olivia Hermann and
Bronte Ficek for their assistance in our study. This work was sup-
ported through grants from the National Institutes of Health (Grant
NIH/NIDCD R01 DC014475) and the Science of Learning Insti-
tute at Johns Hopkins University to K. T.
References
Ash, S., Moore, P., Antani, S., McCawley, G., Work, M., &
Grossman, M. (2006). Trying to tell a tale: Discourse impairments
Themistocleous et al.: Part of Speech Production in PPA 477
 in progressive aphasia and frontotemporal dementia. Neurol-
ogy, 66(9), 1405–1413. https://doi.org/10.1212/01.wnl.0000210435.
72614.38
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A., & Nielsen, H.
(2000). Assessing the accuracy of prediction algorithms for
classification: An overview. Bioinformatics, 16(5), 412–424.
https://doi.org/10.1093/bioinformatics/16.5.412
Bastiaanse, R., & Jonkers, R. (2012). Linguistic accounts of agram-
matic aphasia. In R. Bastiaanse & C. K. Thompson (Eds.),
Perspectives on agrammatism (pp. 17–33). https://doi.org/10.4324/
9780203120378
Bates, D., Mächler, M., Bolker, B. M., & Walker, S. C. (2015).
Fitting linear mixed-effects models using lme4. Journal of Sta-
tistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.
v067.i01
Bhatia, V. K. (1993). Analysing genre: Language use in professional
settings. Longman.
Bird, S., Klein, E., & Loper, E. (2009). Natural language process-
ing with Python: Analyzing text with the natural language toolkit.
O’Reilly Media.
Breiman, L. (2001). Random forests. Machine Learning, 45(1),
5–32. https://doi.org/10.1023/A:1010933404324
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984).
Classification and regression trees. Wadsworth.
Bucks, R. S., Singh, S., Cuerden, J. M., & Wilcock, G. K. (2000).
Analysis of spontaneous, conversational speech in dementia of
Alzheimer type: Evaluation of an objective technique for ana-
lysing lexical performance. Aphasiology, 14(1), 71–91. https://
doi.org/10.1080/026870300401603
Bybee, J., & Fleischman, S. (1995). Modality in grammar and dis-
course. Benjamins.
Bybee, J., Perkins, R., & Pagliuca, W. (1994). The evolution of
grammar. Tense, aspect, and modality in the languages of the
world. The University of Chicago Press.
Cappa, S. F., Binetti, G., Pezzini, A., Padovani, A., Rozzini, L., &
Trabucchi, M. (1998). Object and action naming in Alzheimer’s
disease and frontotemporal dementia. Neurology, 50(2), 351–355.
https://doi.org/10.1212/wnl.50.2.351
Chicco, D. (2017). Ten quick tips for machine learning in compu-
tational biology. BioData Mining, 10(1), 35. https://doi.org/
10.1186/s13040-017-0155-3
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine
Learning, 20, 273–297. http://doi.org/10.1007/BF00994018
Daniele, A., Giustolisi, L., Silveri, M. C., Colosimo, C., & Gainotti, G.
(1994). Evidence for a possible neuroanatomical basis for
lexical processing of nouns and verbs. Neuropsychologia,
32(11), 1325–1341. https://doi.org/10.1016/0028-3932(94)
00066-2
Faroqi-Shah, Y., Treanor, A., Ratner, N. B., Ficek, B., Webster, K.,
& Tsapkini, K. (2020). Using narratives in differential diagnosis
of neurodegenerative syndromes. Journal of Communication
Disorders, 85, 105994. https://doi.org/10.1016/j.jcomdis.2020.
105994
Fraser, K. C., Lundholm Fors, K., Eckerström, M., Themistocleous,
C., & Kokkinakis, D. (2018). Improving the sensitivity and
specificity of MCI screening with linguistic information. In Pro-
ceedings of the LREC 2018 Workshop “Resources and process-
ing of linguistic, para-linguistic and extra-linguistic data from
people with various forms of cognitive/psychiatric impairments
(RaPID-2)” (2015) (pp. 19–26). http://lrec-conf.org/workshops/
lrec2018/W31/pdf/3_W31.pdf
Fraser, K. C., Meltzer, J. A., Graham, N. L., Leonard, C., Hirst, G.,
Black, S. E., & Rochon, E. (2014). Automated classification
of primary progressive aphasia subtypes from narrative speech
478 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
transcripts. Cortex, 55, 43–60. https://doi.org/10.1016/j.cortex.
2012.12.006
Fyndanis, V., & Themistocleous, C. (2019). Are there prototypical
associations between time frames and aspectual values? Evi-
dence from Greek aphasia and healthy ageing. Clinical Lin-
guistics & Phonetics, 33(1–2), 191–217. https://doi.org/10.1080/
02699206.2018.1480657
Goodglass, H., Kaplan, E., & Barresi, B. (2001). BDAE-3: Boston
Diagnostic Aphasia Examination–Third Edition. Lippincott
Williams & Wilkins.
Gorno-Tempini, M. L., Brambati, S. M., Ginex, V., Ogar, J.,
Dronkers, N. F., Marcone, A., Perani, D., Garibotto, V., &
Miller, B. L. (2008). The logopenic/phonological variant of
primary progressive aphasia. Neurology, 71(16), 1227–1234.
https://doi.org/10.1212/01.wnl.0000320506.79811.da
Gorno-Tempini, M. L., Hillis, A. E., Weintraub, S., Kertesz, A.,
Mendez, M., Cappa, S. F., Ogar, J. M., Rohrer, J. D., Black,
S., Boeve, B. F., Manes, F., Dronkers, N. F., Vandenberghe, R.,
Rascovsky, K., Patterson, K., Miller, B. L., Knopman, D. S.,
Hodges, J. R., Mesulam, M. M., & Grossman, M. (2011).
Classification of primary progressive aphasia and its variants.
Neurology, 76(11), 1006–1014. https://doi.org/10.1212/WNL.
0b013e31821103e6
Gorno-Tempini, M. L., & Pressman, P. (2016). Introduction to
primary progressive aphasia. In S. L. Small & G. Hickok (Eds.),
Neurobiology of language (pp. 935–952). Elsevier. https://
doi.org/10.1016/B978-0-12-407794-2.00075-4
Graham, N. L., Patterson, K., & Hodges, J. R. (2004). When more
yields less: Speaking and writing deficits in nonfluent pro-
gressive aphasia. Neurocase, 10(2), 141–155. https://doi.org/
10.1080/13554790490497256
Guinn, C., & Habash, A. (2012). Language analysis of speakers with
dementia of the Alzheimer’s type. AAAI Fall Symposium—
Technical Report, FS-12-01 (Yaruss 1998), 8–13.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements
of statistical learning: Data mining, inference, and prediction
(2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7
Hillis, A. E., Oh, S., & Ken, L. (2004). Deterioration of naming
nouns versus verbs in primary progressive aphasia. Annals of
Neurology, 55(2), 268–275. https://doi.org/10.1002/ana.10812
Jurafsky, D., & Martin, J. H. (2009). Speech and language pro-
cessing: An introduction to natural language processing, compu-
tational linguistics, and speech recognition (2nd ed.). Pearson
Prentice Hall.
Kavé, G., Leonard, C., Cupit, J., & Rochon, E. (2007). Structur-
ally well-formed narrative production in the face of severe con-
ceptual deterioration: A longitudinal case study of a woman
with semantic dementia. Journal of Neurolinguistics, 20(2),
161–177. https://doi.org/10.1016/j.jneuroling.2006.06.003
Kertesz, A., Davidson, W., & McCabe, P. (1998). Primary progres-
sive semantic aphasia: A case study. Journal of the Interna-
tional Neuropsychological Society, 4(4), 388–398. https://www.
ncbi.nlm.nih.gov/pubmed/9656612
Kim, M., & Thompson, C. K. (2000). Patterns of comprehension
and production of nouns and verbs in agrammatism: Implica-
tions for lexical organization. Brain and Language, 74(1), 1–25.
https://doi.org/10.1006/brln.2000.2315
Knibb, J. A., Woollams, A. M., Hodges, J. R., & Patterson, K.
(2009). Making sense of progressive non-fluent aphasia: An
analysis of conversational speech. Brain, 132(10), 2734–2746.
https://doi.org/10.1093/brain/awp207
Knopman, D. S., Kramer, J. H., Boeve, B. F., Caselli, R. J., Graff-
Radford, N. R., Mendez, M. F., Miller, B. L., & Mercaldo, N.
(2008). Development of methodology for conducting clinical
 trials in frontotemporal lobar degeneration. Brain, 131(11),
2957–2968. https://doi.org/10.1093/brain/awn234
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017).
lmertest package: Tests in linear mixed effects models. Journal
of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/
jss.v082.i13
Lien, D., Hu, Y., & Liu, L. (2017). A note on using ratio variables
in regression analysis. Economics Letters, 150, 114–117. https://
doi.org/10.1016/j.econlet.2016.11.019
Mesulam, M. M. (2001). Primary progressive aphasia. Annals of
Neurology, 49(4), 425–432. https://www.ncbi.nlm.nih.gov/
pubmed/11310619
Mesulam, M. M. (2013). Primary progressive aphasia and the
language network: The 2013 H. Houston merritt lecture.
Neurology, 81(5), 456–462. https://doi.org/10.1212/WNL.
0b013e31829d87df
Mesulam, M. M., Rogalski, E. J., Wieneke, C., Hurley, R. S., Geula,
C., Bigio, E. H., Thompson, C. K., & Weintraub, S. (2014). Pri-
mary progressive aphasia and the evolving neurology of the
language network. Nature Reviews Neurology, 10(10), 554–569.
https://doi.org/10.1038/nrneurol.2014.159
Mesulam, M. M., & Weintraub, S. (2014). Is it time to revisit
the classification guidelines for primary progressive aphasia.
Neurology, 82(13), 1108–1109. https://doi.org/10.1212/WNL.
0000000000000272
Mesulam, M. M., Wieneke, C., Thompson, C., Rogalski, E., &
Weintraub, S. (2012). Quantitative classification of primary
progressive aphasia at early and mild impairment stages. Brain,
135(5), 1537–1553. https://doi.org/10.1093/brain/aws080
Meteyard, L., & Patterson, K. (2009). The relation between con-
tent and structure in language production: An analysis of speech
errors in semantic dementia. Brain and Language, 110(3), 121–134.
https://doi.org/10.1016/j.bandl.2009.03.007
Orimaye, S. O., Wong, J. S.-M., & Golden, K. J. (2014). Learning
predictive linguistic features for alzheimer’s disease and related
dementias using verbal utterances. In Proceedings of the Work-
shop on Computational Linguistics and Clinical Psychology:
From linguistic signal to clinical reality (pp. 78–87). Association
for Computational Linguistics. https://doi.org/10.3115/v1/W14-
3210
R Core Team. (2016). R: A language and environment for statistical
computing. R Foundation for Statistical Computing. https://
www.R-project.org/
Rentoumi, V., Raoufian, L., Ahmed, S., de Jager, C. A., & Garrard, P.
(2014). Features and machine learning classification of connected
speech samples from patients with autopsy proven Alzheimer’s
disease with and without additional vascular pathology. Journal
of Alzheimer’s Disease, 42, S3–S17. https://doi.org/10.3233/JAD-
140555
Riello, M., Faria, A. V., Ficek, B., Webster, K., Onyike, C. U.,
Desmond, J., Frangakis, C., & Tsapkini, K. (2018). The role
of language severity and education in explaining performance
on object and action naming in primary progressive aphasia.
Frontiers in Aging Neuroscience, 10, 346. https://doi.org/10.3389/
fnagi.2018.00346
Russell, L. (2018). Emmeans: Estimated marginal means, aka least-
squares means. R package version 1(2). https://cran.r-project.org/
web/packages/emmeans/index.html
Swales, J. M. (1990). Genre analysis: English in academic and re-
search settings. Cambridge University Press.
Themistocleous, C., Eckerstrom, M., & Kokkinakis, D. (2018).
Identification of mild cognitive impairment from speech in
Swedish using deep sequential neural networks. Frontiers in
Neurology, 9, 975. https://doi.org/10.3389/fneur.2018.00975
Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions
Themistocleous, C., & Kokkinakis, D. (2018). THEMIS-SV:
Automatic classification of language disorders from speech
signals. ESOC 2018: European Stroke Organisation Con-
ference, Gothenburg. https://doi.org/10.26226/morressier.
5ab8f55bd462b8029238c82d
Thomas, C., Keselj, V., Cercone, N., Rockwood, K., & Asp, E. (2005).
Automatic detection and rating of dementia of Alzheimer type
through lexical analysis of spontaneous speech. In IEEE Inter-
national Conference on Mechatronics and Automation (Vol. 3,,
pp. 1569–1574). https://doi.org/10.1109/icma.2005.1626789
Thompson, C. K. (2012). Northwestern Assessment of Verbs and
Sentences (NAVS). Northwestern University.
Thompson, C. K., Ballard, K. J., Tait, M. E., Weintraub, S., &
Mesulam, M. M. (1997). Patterns of language decline in non-
fluent primary progressive aphasia. Aphasiology, 11(4–5),
297–321. https://doi.org/10.1080/02687039708248473
Thompson, C. K., & Bastiaanse, R. (2012). Introduction to agram-
matism. In Perspectives on agrammatism. Psychology Press.
https://doi.org/10.4324/9780203120378
Thompson, C. K., Cho, S., Hsu, C. J., Wieneke, C., Rademaker,
A., Weitner, B. B., Mesulam, M. M., & Weintraub, S. (2012).
Dissociations between fluency and agrammatism in primary
progressive aphasia. Aphasiology, 26(1), 20–43. https://doi.org/
10.1080/02687038.2011.584691
Thompson, C. K., Lukic, S., King, M. C., Mesulam, M. M., &
Weintraub, S. (2012). Verb and noun deficits in stroke-induced
and primary progressive aphasia: The Northwestern Naming
Battery. Aphasiology, 26(5), 632–655. https://doi.org/10.1080/
02687038.2012.676852
Thompson, C. K., & Mack, J. E. (2014). Grammatical impairments
in PPA. Aphasiology, 28(8–9), 1018–1037. https://doi.org/10.1080/
02687038.2014.912744
Thompson, C. K., Meltzer-Asscher, A., Bachrach, A., Roy, I., &
Stockall, L. (2014). Neurocognitive mechanisms of verb argument
structure processing. John Benjamins. https://doi.org/10.1075/
lfab.10.07tho
Thompson, C. K., Meltzer-Asscher, A., Cho, S., Lee, J., Wieneke, C.,
Weintraub, S., & Mesulam, M. M. (2013). Neurocognitive mech-
anisms of verb argument structure processing. Behavioural
Neurology, 26(1–2), 35–54. https://doi.org/10.1155/2013/749412
Tsapkini, K., Webster, K. T., Ficek, B. N., Desmond, J. E., Onyike,
C. U., Rapp, B., Frangakis, C. E., & Hillis, A. E. (2018). Elec-
trical brain stimulation in different variants of primary progres-
sive aphasia: A randomized clinical trial. Alzheimer’s & Dementia
(New York, N. Y.), 4(1), 461–472. https://doi.org/10.1016/
j.trci.2018.08.002
Wilson, S. M., Brandt, T. H., Henry, M. L., Babiak, M., Ogar,
J. M., Salli, C., Wilson, L., Peralta, K., Miller, B. L., & Gorno-
Tempini, M. L. (2014). Inflectional morphology in primary
progressive aphasia: An elicited production study. Brain and
Language, 136, 58–68. https://doi.org/10.1016/j.bandl.2014.
07.001
Wilson, S. M., DeMarco, A. T., Henry, M. L., Gesierich, B., Babiak,
M., Mandelli, M. L., Miller, B. L., & Gorno-Tempini, M. L.
(2014). What role does the anterior temporal lobe play in sentence-
level processing? Neural correlates of syntactic processing in
semantic variant primary progressive aphasia. Journal of Cog-
nitive Neuroscience, 26(5), 970–985. https://doi.org/10.1162/
jocn_a_00550
Wilson, S. M., Henry, M. L., Besbris, M., Ogar, J. M., Dronkers,
N. F., Jarrold, W., Miller, B. L., & Gorno-Tempini, M. L.
(2010). Connected speech production in three variants of primary
progressive aphasia. Brain, 133(7), 2069–2088. https://doi.org/
10.1093/brain/awq129
Themistocleous et al.: Part of Speech Production in PPA 479
 Appendix
Regression Models Showing the Statistical Effects of Primary Progressive Aphasia (PPA) Variants on the Distribution of Part
of Speech

POS ratio PPA Estimate SE t Pr(>|t|) F test

Noun/verb ratio nfvPPA 1.583 0.172 9.21 < .0001*** F(2, 47) = 7.3, p < .01, R2 = 24%
lvPPA −0.678 0.243 −2.79 .0076**
svPPA −0.939 0.26 −3.61 .0007***
Noun/adjective ratio nfvPPA 11.87 1.36 8.7 < .0001*** F(2, 43) = 7.58, p < .01, R2 = 26%
lvPPA −5.5 1.79 −3.07 .0037**
svPPA −6.89 1.86 −3.7 .0006***
Noun/adverb ratio nfvPPA 12.69 1.9 6.67 < .0001*** F(2, 46) = 8.09, p < .001, R2 = 26%
lvPPA −8.16 2.69 −3.03 .0040**
svPPA −10.51 2.78 −3.78 .0005***
Noun/pronoun ratio nfvPPA 5.414 0.952 5.69 < .0001*** F(2, 44) = 4.37, p < .05, R2 = 17%
lvPPA −2.544 1.327 −1.92 .0617
svPPA −4.043 1.394 −2.9 .0058**
Noun/preposition ratio nfvPPA 3.29 0.384 8.56 < .0001*** F(2, 46) = 5.36, p < .01, R2 = 19%
lvPPA −1.442 0.536 −2.69 .0099**
svPPA −1.678 0.572 −2.93 .0052**
Noun/conjunction ratio nfvPPA 7.302 0.941 7.76 < .0001*** F(2, 45) = 6.01, p < .01, R2 = 21%
lvPPA −2.708 1.293 −2.09 .0419*
svPPA −4.735 1.377 −3.44 .0013**
Verb/adjective ratio nfvPPA 9.223 1.585 5.82 < .0001*** F(2, 43) = 0.142, p = .86, R2 = 0.6%
lvPPA −1.021 2.079 −0.49 .6300
svPPA −0.969 2.165 −0.45 .6600
Verb/adverb ratio nfvPPA 9.25 1.25 7.4 .0000*** F(2, 46) = 5.34, p < .001, R2 = 19%
lvPPA −4.29 1.77 −2.43 .0191*
svPPA −5.64 1.83 −3.09 .0034**
Verb/pronoun ratio nfvPPA 3.779 0.699 5.41 < .0001*** F(2, 44) = 1.19, p = .3, R2 = 5%
lvPPA −0.529 0.973 −0.54 .5900
svPPA −1.558 1.023 −1.52 .1300
Verb/preposition ratio nfvPPA 2.254 0.256 8.8 < .0001*** F(2, 46) = 0.15, p = .86, R2 = 0.6%
lvPPA −0.016 0.357 −0.04 .9600
svPPA 0.174 0.381 0.46 .6500
Verb/conjunction ratio nfvPPA 5.071 0.642 7.9 .0000*** F(2, 45) = 0.31, p = = .73, R2 = 1%
lvPPA 0.234 0.882 0.27 .7900
svPPA −0.487 0.939 −0.52 .6100
Adjective/adverb ratio nfvPPA 0.7450 0.1506 4.95 0.000011*** F(2, 46) = 0.23, p = .79, R2 = 1%
lvPPA 0.0748 0.2130 0.35 0.73
svPPA −0.0753 0.2200 −0.34 0.73
Adjective/pronoun ratio nfvPPA 0.3438 0.1186 2.90 0.0058** F(2, 47) = 0.9, p = .41, R2 = 3%
lvPPA 0.2086 0.1653 1.26 0.2137
svPPA 0.0365 0.1737 0.21 0.8346
Adjective/preposition ratio nfvPPA 0.2478 0.0948 2.62 0.012* F(2, 46) = 1.74, p = .19, R2 = 7%
lvPPA 0.1579 0.1321 1.20 0.238
svPPA 0.2579 0.1410 1.83 0.074
Adjective/conjunction ratio nfvPPA 0.626 0.266 2.35 0.023* F(2, 45) = 0.87, p = .4, R2 = 3%
lvPPA 0.474 0.366 1.30 0.202
svPPA 0.165 0.390 0.42 0.673
Adverb/pronoun ratio nfvPPA 0.430 0.244 1.77 0.084 F(2, 44) = 0.93, p = .4, R2 = 4%
lvPPA 0.458 0.339 1.35 0.184
svPPA 0.185 0.357 0.52 0.607
Adverb/preposition ratio nfvPPA 0.309 0.131 2.36 0.022* F(2, 46) = 2.83, p = .07, R2 = 10%
lvPPA 0.350 0.182 1.92 0.061
svPPA 0.419 0.194 2.16 0.036*
Adverb/preposition ratio nfvPPA 0.736 0.180 4.09 0.00018*** F(2, 45) = 2.88, p = .07, R2 = 11%
lvPPA 0.571 0.248 2.31 0.02570
svPPA 0.452 0.264 1.71 0.09348

Note. The intercept of the model corresponds to the nonfluent PPA variant (nfvPPA). It also provides the standard error (SE), the t value, and
the p value, Pr(>|t|). The last column provides the overall analysis of variance score of the model and its corresponding R2. lvPPA = logopenic
PPA variant; svPPA = semantic PPA variant.
*p < .05. **p < .01. ***p < .001.

480 American Journal of Speech-Language Pathology • Vol. 30 • 466–480 • February 2021

Downloaded from: https://pubs.asha.org Iberoamericana- Instituto Universitaria on 03/24/2024, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions