Psychological Assessment

© 2019 American Psychological Association 2019, Vol. 31, No. 6, 730 –740
1040-3590/19/$12.00 http://dx.doi.org/10.1037/pas0000695

Ecological Validity of a Quantitative Classification System for Mental

Illness in Treatment-Seeking Adults

Christopher C. Conway Maxwell Mansolf and Steven P. Reise

College of William and Mary University of California, Los Angeles

Quantitative models of mental illness, such as the Hierarchical Taxonomy of Psychopathology (HiTOP),
aim to optimize clinical assessment, which conventionally follows categorical diagnostic rubrics. The
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

evidence base for these models is robust, but also uniform; available data come mostly from structured
This document is copyrighted by the American Psychological Association or one of its allied publishers.

diagnostic interviews in nationally representative samples. It remains to be seen whether HiTOP

adequately reflects mental illness as evaluated in routine clinical care, where diagnosis is often unsys-
tematic and incomplete, relative to controlled research conditions. To test the generalizability of a
quantitative nosology to real-world assessment contexts, we fit the HiTOP model to diagnoses in a large
sample (N ⫽ 25,002) of treatment-seeking university students who were seen by health professionals in
everyday practice. We then examined the criterion validity of model components in relation to clinically
relevant outcomes (i.e., suicide attempts, self-injury, and binge drinking). Three related structures fit the
data well: a correlated-factor model with internalizing, externalizing, and eating pathology dimensions;
a higher-order model that added a general factor of psychopathology that spanned these 3 first-order
factors; and a bifactor model that partitioned diagnostic (co)variance across a general factor and 3
orthogonal group factors. The first-order factors had expected patterns of criterion validity, and the
general factor was a strong predictor of suicidality and self-injury, paralleling past research. Bifactor
model group factors had interpretative problems, however. Across models, categorical diagnoses con-
sistently offered minimal incremental validity relative to the transdiagnostic factors. We conclude that
HiTOP is ecologically valid— explaining comorbidity patterns among diagnoses assigned “in the
field”—and is poised to enhance clinical assessment and decision-making in routine care.

Public Significance Statement

Patterns in practitioners’ diagnostic judgments could reveal dimensions of psychopathology that are
useful for clinical assessment and decision-making.

Keywords: confirmatory factor analysis, dimensional model, Hierarchical Taxonomy of Psychopathology

(HiTOP), p-factor, transdiagnostic

Supplemental materials: http://dx.doi.org/10.1037/pas0000695.supp

A valid classification system for mental illness is fundamental to
all aspects of clinical practice. Psychiatric diagnoses orient assess-
ment procedures, prognosis, and treatment selection. Contempo-
rary categorical diagnoses, however, such as those catalogued in
the Diagnostic and Statistical Manual of Mental Disorder (DSM),
have several well-documented defects. Decades of research attest
This article was published Online First January 21, 2019.
Christopher C. Conway, Department of Psychology, College of William
and Mary; Maxwell Mansolf and Steven P. Reise, Department of Psychol-
ogy, University of California, Los Angeles.
The authors have no conflicts of interest to declare. They adhered to
APA ethical standards in the conduct of this study.
Correspondence concerning this article should be addressed to Christo-
pher C. Conway, who is now at Department of Psychology, Fordham
University, 226 Dealy Hall, 441 East Fordham Road, Bronx, NY 10458.
E-mail: cconway26@fordham.edu
730
to problems with within-category heterogeneity, excessive comor-
bidity of putatively independent conditions, unreliability, and poor
diagnostic coverage that collectively undercut the validity of the
categorical perspective and, by extension, its clinical utility
(Chmielewski, Clark, Bagby, & Watson, 2015; Clark, Cuthbert,
Lewis-Fernández, Narrow, & Reed, 2017; Harkavy-Friedman,
2009; Regier et al., 2013).
A quantitative classification solves many of the problems en-
demic to categorical rubrics like DSM. Quantitative models are
based on patterns of covariation among signs and symptoms of
mental illness. Dimensions, as opposed to categories, are the basic
units of these systems, such that people are presumed to vary
continuously, not qualitatively, with respect to clinical problems
(Krueger et al., 2018). The dimensions composing a quantitative
classification are derived empirically, as opposed to authoritatively
or via clinical heuristics, as was the case during some phases of
DSM development (Clark et al., 2017). Building on early factor
 DIMENSIONAL MODEL OF PSYCHOPATHOLOGY
analyses of youth psychopathology symptoms (e.g., Achenbach,
1966), researchers are now working to describe the dimensional
structure of mental illness across the life span (Lahey, Krueger,
Rathouz, Waldman, & Zald, 2017).
A Hierarchical Taxonomy of
Psychopathology (HiTOP)
The Hierarchical Taxonomy of Psychopathology (HiTOP1) is an
emerging quantitative classification that brings together years of
research on the latent dimensions that characterize mental illness.
It is based on the idea that mental illness can be understood
hierarchically. Thus, there are various levels of resolution in the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
HiTOP model, ranging from narrow, homogeneous symptom com-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
ponents and maladaptive traits at the base of the hierarchy up to
broad spectra and superspectra at the apex (Kotov et al., 2017).
Near the top of the hierarchy, internalizing and externalizing
dimensions anchor the “spectrum” level. Internalizing reflects the
common features of anxiety and depressive disorders, whereas
externalizing represents the substrate of antisocial behavior and
substance use disorders. These two factors have been replicated
consistently across developmental, clinical, and cultural groups
(Eaton, Rodriguez-Seijas, Carragher, & Krueger, 2015). Data are
more limited for other spectrum-level constructs in the HiTOP
structure: thought disorder, detachment, and somatization. Drilling
down to lower levels of the hierarchy, spectra can be dissected into
subfactors, such as fear (marking panic and the phobias) and
distress (depression and generalized anxiety) dimensions that are
often found to undergird the broader internalizing spectrum (e.g.,
Watson, 2005). In turn, subfactors comprise syndromes (e.g.,
panic, social phobia), which can themselves be decomposed into
fine-grain symptom components (e.g., autonomic arousal, perfor-
mance anxiety) and maladaptive traits (e.g., anxiousness, separa-
tion insecurity).
A general factor of psychopathology, dubbed the p-factor after
its g-factor counterpart in the intelligence field, is theorized to rest
at the very top of the dimensional hierarchy. Though the meaning
of this factor remains unclear, it may represent negative affectivity,
poor impulse control, demoralization secondary to mental illness,
or some admixture of these processes (Caspi & Moffitt, 2018).
There has been a recent surge of reports on the presence of the
p-factor in large-scale research on mental illness comorbidity
(Lahey et al., 2017). This research generally shows that models
including the p-factor fit well according to conventional indices in
youth (Laceulle, Vollebergh, & Ormel, 2015; Lahey et al., 2015;
Tackett et al., 2013) and adults (Caspi et al., 2014; Lahey et al.,
2012). The p-factor also appears to effectively predict key clinical
outcomes, such as suicide and treatment use (Caspi et al., 2014;
Lahey et al., 2015; Patalay et al., 2015; Snyder, Young, & Hankin,
2017). Studies in this area represent the p-factor as a superordinate
factor in a higher-order model or as a general dimension on a
bifactor model (Caspi & Moffitt, 2018).
Relative to categorical diagnosis, HiTOP implies a significantly
different perspective on clinical assessment. As mentioned above,
patients are presumed to vary quantitatively, rather than qualita-
tively, on illness dimensions. Notably, these dimensions conserve
information about patient severity that is lost when assigning
binary diagnoses in the DSM tradition, and they are consistent with
the importance of subsyndromal symptom presentations for prog-
731
nosis and treatment (e.g., Waszczuk et al., 2017). Additionally, a
dimensional system avoids the conceptually problematic “other
specified/unspecified” designation by allowing assessors to char-
acterize patients using homogeneous symptom and trait dimen-
sions that can describe clinical cases that either just miss DSM
diagnostic thresholds or do not fit neatly into any DSM diagnostic
category (e.g., Verheul & Widiger, 2004). Further, practitioners
can take advantage of the hierarchical structure of mental illness to
target a level of resolution dictated by the purpose of the assess-
ment (e.g., screening, treatment planning), setting (e.g., the fear
subfactor might be of greatest interest at a panic disorder clinic),
and patient preferences. More generally, the hierarchical orienta-
tion permits “telescoping” assessment, such that assessors can
progress from higher-order spectra to narrower subfactors, syn-
dromes, and symptom components depending on resources and
patient needs.
Recent data point to the utility of the HiTOP approach to
assessment for prognosis and treatment. Two reports from nation-
ally representative samples showed that scores on the internalizing
spectrum were strongly linked to suicide risk, whereas categorical
disorders (e.g., major depression) were mostly unrelated to suicide
potential, net the internalizing dimension (Eaton et al., 2013;
Sunderland & Slade, 2015). Also, research in the World Mental
Health Surveys found that the persistence and new onset of
DSM–IV diagnoses could be explained almost entirely by variation
in higher-order internalizing and externalizing spectra (Kessler,
Petukhova, & Zaslavsky, 2011; see also Kotov, Perlman, Gámez,
& Watson, 2015). Regarding intervention, factor analytic studies
in clinical samples suggest that treatment decisions by both prac-
titioners (Waszczuk et al., 2017) and patients (Rodriguez-Seijas,
Eaton, Stohl, Mauro, & Hasin, 2017) are based more on individual
differences on higher-order dimensions (e.g., thought disorder)
than DSM categories (e.g., schizotypal personality disorder). Thus,
there are preliminary indications that dimensional assessment,
especially at higher-order levels of the hierarchical framework,
outperforms categorical assessment in forecasting key clinical
outcomes (reviewed in Conway et al., in press).
Ecological Validity of the HiTOP Model
Research to date generally has supported the HiTOP model, and
there is reason to believe that it could yield major assessment
benefits. However, HiTOP undoubtedly remains a work in prog-
ress. One understandable limitation in the evidence base is that
data collection efforts have almost exclusively been oriented
around standardized diagnostic interviews. Many comprehensive
studies on the structure of psychopathology involve factor analysis
of diagnostic information from nationally representative samples
assessed by lay interviewers using structured interview measures
(e.g., Eaton et al., 2013; Krueger, 1999). This is not a method-
ological deficiency per se, given that clinical interviews are gen-
erally the most reliable source of symptom information. However,
while these standardized assessments are customary in research
settings, they are much less common—indeed, they are rare—in
routine clinical care. This discrepancy raises the question of
whether the HiTOP model largely constructed from interview data
1
http://medicine.stonybrookmedicine.edu/HITOP
 732 CONWAY, MANSOLF, AND REISE
applies to diagnoses assigned in the course of everyday clinical
practice.
Diagnosis in ordinary practice, as compared to the controlled
conditions of research settings, is unsystematic. Clinicians tend to
home in on the most urgent or problematic features of the clinical
picture and select a diagnosis that best matches those prominent
symptoms (Wilk et al., 2006). This heuristic leads to underdetec-
tion of clinical problems and, as a result, imperfect diagnosis and
an incomplete representation of comorbidity. Research suggests
that a more unstructured assessment style in clinical practice is to
blame for this bias (Zimmerman & Mattia, 1999). Aside from time
limitations that might preclude comprehensive evaluations, a lack
of specialist training among practitioners in primary care settings
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
also contributes to variability in diagnostic practices (Clark et al.,
This document is copyrighted by the American Psychological Association or one of its allied publishers.
2017).
In sum, HiTOP is intended to guide diagnostic assessment in
diverse clinical settings, but there is little evidence that data from
real-world clinical practice conform to the HiTOP model. It is
possible that diagnoses rendered in routine care do not conform to
the structure based on data from specialized research settings. In
the current study, we advanced HiTOP research by using a novel
source of assessment data: diagnoses assigned by mental health
professionals “in the field.” In so doing, we aimed to evaluate the
ecological validity, or generalizability, of the HiTOP system.
Current Study
We analyzed responses to a large-scale survey of university
students (n ⫽ 25,002) who had sought mental health treatment in
the past year. Each student provided a lifetime history of diagnoses
that had been assigned by a mental health professional. We hy-
pothesized that a correlated 2-factor (internalizing and externaliz-
ing) model consistently supported in prior studies would fit these
student-reported diagnostic data well (Eaton et al., 2015; Lahey et
al., 2017). We also expected that a general psychopathology factor
(i.e., p-factor) would emerge in bifactor and higher-order models
to explain the covariation among the full array of clinician-
assigned diagnoses (Carragher et al., 2016; Caspi et al., 2014;
Lahey et al., 2012, 2015; Tackett et al., 2013). Finally, we hypoth-
esized that the higher-order dimensions would be robustly associ-
ated with suicide potential, self-injury, and binge drinking,
whereas diagnostic categories would have weak incremental va-
lidity relative to the cross-cutting dimensions.
Method
Participants
Participants were a subsample of respondents to the Healthy
Minds Study (HMS; Eisenberg, Hunt, & Speer, 2013), a national
survey of mental health complaints among university students.
There were 98,971 survey responses between 2007 and 2014. We
focus here on a subset of 25,002 students (i.e., 25% of the original
sample) who indicated that they had in the past 12 months received
counseling or therapy for mental illness or had been prescribed
psychiatric medication. We concentrated on this group to maxi-
mize (a) the sample prevalence of infrequent mental disorders and
(b) participants’ recall for any psychiatric diagnoses assigned by a
mental health professional (see below).
Our treatment-seeking sample was largely (72%) female and
was on average 21.77 (SD ⫽ 2.40) years old. The racial/ethnic
breakdown was 73% White, 13% Asian American, 6% African
American, 1% Native American, and 10% other or mixed race.
Eight percent of the sample identified as Latino/a. The majority of
respondents were pursuing undergraduate degrees (72%), and most
of the remainder was enrolled in graduate (19%) and professional
(3%) programs. The University of Michigan Institutional Review
Board approved the study procedures (HUM00100169).
Measures
Participants were asked whether they had ever been diagnosed
by a health professional (e.g., primary care doctor, psychiatrist,
psychologist) with any of a list of mental health conditions. This
list included various diagnostic clusters: anxiety, depressive, eat-
ing, obsessive– compulsive, posttraumatic stress, substance use,
and attentional disorders (see Table S1 for the full sequence of
diagnostic questions). If participants indicated that they had been
assigned a diagnosis from a particular cluster, they were then
presented with a list of possible diagnoses within that cluster. For
instance, when participants responded that they had been diag-
nosed with a depressive disorder, they then chose from a set of
specific diagnoses that included major depressive disorder, dys-
thymia, bipolar disorder, and cyclothymia. A number of diagnostic
categories were combined or omitted due to base rate issues.
Bipolar disorder and cyclothymia were collapsed due to the low
base rate of the latter. Likewise, we combined panic disorder and
agoraphobia into one condition for all analyses. Acute stress dis-
order, schizophrenia-spectrum and psychotic disorders, and per-
sonality disorders were all disregarded in our analysis because of
low prevalence. Table 1 presents the diagnoses analyzed here.
The HMS survey also assessed correlates of psychopathology.
We examined three here in relation to HiTOP dimensions: suicide
attempts, nonsuicidal self-injury, and binge drinking. We selected
these outcomes because of their (a) clinical relevance and (b)
theorized relations to the internalizing (suicide and self-injury)
versus externalizing (binge drinking) spectra (e.g., Eaton et al.,
2013; Krueger, Markon, Patrick, Benning, & Kramer, 2007). Re-
garding suicidality, the survey prompted students to indicate
whether they had attempted suicide in the past year (0 ⫽ no, 1 ⫽
yes). They also indicated how often over the past year they had
hurt themselves on purpose without intending to die (1 ⫽ once or
twice, 2 ⫽ once a month or less, 3 ⫽ 2 or 3 times a month, 4 ⫽
once or twice a week, 5 ⫽ 3 to 5 days a week, 6 ⫽ nearly every
day or every day). The final two response options were collapsed
due to the infrequency of the last category. Finally, to assess binge
drinking, students reported on how many occasions over the past
2 weeks they had four (females) or five (males) drinks in a row
(1 ⫽ none, 2 ⫽ once, 3 ⫽ twice, 4 ⫽ 3 to 5 times, 5 ⫽ 6 to 9 times,
6 ⫽ 10 or more times).
Data Analyses
To specify a measurement model (or models) that best captured
the comorbidity among diagnoses, we evaluated the fit of a series
of confirmatory factor analyses (CFAs). We first tested a 1-factor
model in which the single dimension reflected risk for all mental
disorders. Next, we evaluated a 2-factor model with correlated
 DIMENSIONAL MODEL OF PSYCHOPATHOLOGY 733

Table 1
Prevalence and Tetrachoric Correlations of Student-Reported Psychiatric Diagnoses

Condition 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1. Major depressive disorder —

2. Dysthymia .420 —
3. Bipolar disorder .476 .345 —
4. Panic disorder .466 .420 .440 —
5. Generalized anxiety disorder .635 .551 .475 .660 —
6. Specific phobia .288 .313 .303 .530 .529 —
7. Social phobia .481 .464 .370 .523 .602 .500 —
8. Obsessive-compulsive disorder .411 .379 .419 .486 .589 .500 .459 —
9. Posttraumatic stress disorder .488 .376 .459 .518 .523 .368 .376 .429 —
10. ADHD .318 .297 .381 .242 .349 .193 .305 .310 .282 —
11. Anorexia nervosa .423 .316 .327 .324 .409 .238 .270 .423 .354 .154 —
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

12. Bulimia nervosa .443 .304 .344 .296 .406 .208 .206 .372 .326 .228 .776 —
This document is copyrighted by the American Psychological Association or one of its allied publishers.

13. Binge eating disorder .407 .295 .346 .307 .354 .268 .358 .400 .354 .211 .669 .782 —
14. Alcohol use disorder .393 .330 .489 .333 .367 .272 .306 .299 .434 .300 .316 .434 .398 —
15. Drug use disorder .386 .338 .481 .296 .341 .319 .334 .282 .391 .316 .300 .332 .327 .729 —
N 3858 2198 1550 1920 7465 319 1133 1544 1357 4049 1095 821 398 699 482
% 15 9 6 8 30 1 5 6 5 16 4 3 2 3 2
Note. N ⫽ number of patients diagnosed with each condition. ADHD ⫽ attention-deficit hyperactivity disorder. All correlations are statistically significant
at the .001 alpha level.

internalizing and externalizing spectra. The internalizing dimen-
sion featured loadings from anxiety, depressive, eating, obsessive–
compulsive, and posttraumatic stress disorders, whereas external-
izing was defined by attentional and substance use disorders. In
response to an outstanding local area of strain in this solution (see
Brown, 2015), we allowed bipolar disorder to cross-load on the
externalizing spectrum in this and subsequent models. We also
tested a 3-factor model in which eating disorders broke away from
the internalizing spectrum and formed a separate factor (Forbush et
al., 2010). In our final correlated-factor model, the internalizing
dimension was divided into distress and fear subfactors to account
for possible clustering of disorders within the internalizing spec-
trum (Watson, 2005). The fear dimension was characterized by
panic disorder, specific phobia, social phobia, and obsessive–
compulsive disorder, whereas the distress dimension featured the
remaining internalizing syndromes. Building on these correlated-
factor models, we tested a set of higher-order factor models in
which 2 (internalizing, externalizing)2, 3 (internalizing, external-
izing, eating pathology), and 4 (fear, distress, externalizing, eating
pathology) first-order factors loaded onto a second-order factor
reflecting the general factor of psychopathology. Finally, we eval-
uated a series of bifactor models in which all disorders loaded onto
a general factor of psychopathology and only one group factor
(Lahey et al., 2012). Specifically, we fit variants of the bifactor
model with 2 (internalizing, externalizing), 3 (internalizing, exter-
nalizing, eating pathology), and 4 (fear, distress, externalizing,
eating pathology) orthogonal group factors.
Bifactor models have been increasingly applied in a variety of
domains within psychology and psychiatry, most notably when
studying psychopathology (e.g., Oltmanns, Smith, Oltmanns, &
Widiger, 2018; see also Reise, 2012). Most often, the bifactor
model is compared with alternative measurement models, such as
unidimensional and correlated-factor models, in order to assess its
applicability. Typically, the bifactor model “wins” these fit con-
tests (Mansolf & Reise, 2017; Murray & Johnson, 2013), possibly
because the bifactor model has a greater propensity than these
other models to fit any set of data (Bonifay, 2015; Preacher, 2006).
Therefore, in addition to assessing model fit for the measurement
models applied herein, we calculated additional indices to assess
the factors identified by these models according to two additional
characteristics: construct reliability, as measured by Hancock’s H
(Hancock & Mueller, 2001; Rodriguez, Reise, & Haviland, 2016);
and explained common variance (ECV; Reise, Moore, & Haviland,
2010; Reise, Scheines, Widaman, & Haviland, 2013), which is
used to assess the relative strengths of the general and group
factors in a bifactor model.
After the measurement models were evaluated, we used SEM to
predict suicide, self-harm, and binge drinking from the factors in
the CFA models. When assessing the higher-order model, we
could not predict the outcomes from all factors (first- and second-
order) simultaneously, and we therefore estimated two SEMs for
each higher-order model: one including only the first-order factors
as predictors, and the other including only the second-order factor
as a predictor. The factor structure (i.e., factor loadings and factor
correlations) for the SEM models was fixed to be identical to the
estimated factor structure in the CFA models. All variables, in-
cluding clinical outcomes, were treated as ordered categorical
variables in structural analyses, meaning that a standardized, nor-
mally distributed random variable was assumed to underlie the
observed categorical responses. Parameter estimates from these
models reflect the relationships between the underlying standard-
ized variables and can therefore be interpreted directly as stan-
dardized coefficients. Model modification indices (Sörbom, 1989)
were used to determine whether allowing residual covariances
between the unique portion (i.e., residual variance term) of any
disorder and any outcome would lead to a significant improvement
in model fit. Any such residual associations would indicate that the
diagnostic categories influenced the outcomes above and beyond
2
The second-order factor loadings were constrained to equality in the
higher-order model with two first-order factors to identify the model.
 734 CONWAY, MANSOLF, AND REISE

the effects of the latent traits (see, e.g., Keyes et al., 2012).
Because the categorical SEM treats all variables as standardized,
the resulting expected parameter change values (EPC; Saris, Sa-
torra, & Sörbom, 1987) can be interpreted on the standardized
metric as residual correlations. All analyses were performed using
the lavaan package in R (Rosseel, 2012). Our analysis code is
available at https://osf.io/fmnkj. The Healthy Minds Study Data
Repository Protocol prohibits public posting of the data sets, but
researchers can access the data by request at http://healthyminds
network.org/research/data-for-researchers.
Results
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
The factor loadings on the general factor for the best-fitting bifac-
tor model (with 3 group factors) were fairly uneven (standardized
factor loading range ⫽ .14 to .53). Also, the loadings on the
internalizing factor indicated that this dimension was most tightly
related to fear disorders (e.g., specific phobia, panic), and much
less so to distress disorders. Further, as shown in Table S3,
Hancock’s H values for the internalizing and externalizing group
factors fell below established benchmarks (Hancock & Mueller,
2001). Together, these observations suggest that interpretability of
the primary factors might be problematic once the general factor is
controlled for (Rodriguez et al., 2016), as is often the case in
clinical psychological applications of bifactor models (e.g., Reise,
2012).

Table 2 presents fit indices for the 10 CFA models. Factor For brevity, for the 10 SEMs estimated we will only discuss the
This document is copyrighted by the American Psychological Association or one of its allied publishers.

loadings and factor correlations are presented in Table S2 in online results for the best-fitting higher-order (with 3 first-order factors),
supplementary material. We judged model fit using standard rules bifactor (with 3 group factors), and correlated-factor (with 3 first-
of thumb (e.g., Hu & Bentler, 1999), where CFI ⬎ .95, TLI ⬎ .95, order factors) models (see Figure 1). The regression coefficients
RMSEA ⬍ .05, SRMR ⬍ .05 indicate adequate fit. The unidimen- from these models, including confidence intervals and z tests, are
sional model and the correlated-factor model with 2 factors fit presented in Table 3. (Structural parameter estimates for all other
poorly, with all indices below typical guidelines. The correlated- factor models are reported in Table S4.) These regression coeffi-
factor model with 3 factors fit better than the previous two models, cients apply to the standardized variables underlying the observed
but failed to satisfy the SRMR criterion, while the correlated-factor categorical variables and the standardized latent factors and can
model with 4 factors satisfied this criterion, albeit with a slight therefore be interpreted on a standardized metric.
increase in RMSEA due to the loss of degrees of freedom. Also, In the correlated-factor model, the effect of internalizing was
the distress and fear group factors were very strongly correlated twice as large as externalizing in predicting suicide potential and
(r ⫽ .93), suggesting they cannot be reasonably distinguished in three times as large in predicting self-harm. For the binge drinking
this dataset. Among the higher-order models, the one with 3 outcome, in contrast, externalizing had a moderate positive effect,
first-order factors demonstrated superior fit (second-order factor whereas internalizing had a more modest inverse influence. The
loadings for internalizing, externalizing, and eating of .70, .59, and eating pathology dimension, across all outcomes, had only a minor
.62, respectively). Adding a fourth first-order dimension worsened influence (mean standardized effect ⫽ .04).
traditional fit indices and led to a standardized general factor In the higher-order model, effects of the first-order factors on
loading greater than 1 (i.e., Heywood case). the three outcomes were identical to those reported for the
The bifactor models with 2 and 3 group factors both satisfied correlated-factor model. This is because, with three group factors,
our fit criteria. Interestingly, adding a fourth group factor distin- the higher-order factor is just identified, and the higher-order
guishing fear from distress disorders worsened model fit compared model with three first-order factors is in fact equivalent to the
to bifactor models with 2 and 3 group factors. This model also correlated-factor model with three factors. The general factor of
displayed distorted parameter estimates compared to the other psychopathology, reflected in the second-order factor, had strong
bifactor models, with the internalizing factor collapsing almost effects on suicide (standardized effect ⫽ .50), self-injury (.45), but
entirely onto the general factor in order to allow the fear factor, not binge drinking (.05).
which was highly correlated with distress in the correlated-factor In the bifactor model, the general factor had strong, positive
model, to be orthogonal to the general factor. Therefore we do not associations with suicide (.73) and self-harm (.46). Unexpectedly,
consider the predictive effects of this measurement model below. the internalizing, eating pathology, and externalizing group factors
were inversely (and highly statistically significantly) related to the
Table 2 both suicide and self-harm after controlling for the effects of the
Confirmatory Factor Analysis Model Fit Statistics general factor, except for the association between the internalizing
group factor and self-harm, which was slightly positive and not
Model df ␹2 CFI TLI RMSEA SRMR statistically significant. This counterintuitive pattern illustrates that
the nature and predictive utility of primary factors (e.g., internal-
Unidimensional 90 3411.47 .813 .781 .038 .114 izing) changes dramatically after partialing out covariation among
Correlated 2-factor 88 2597.26 .858 .831 .034 .100
Correlated 3-factor 86 668.46 .967 .960 .016 .052 all diagnoses with the general factor. Binge drinking, in contrast,
Correlated 4-factor 83 651.17 .968 .959 .017 .049 was most strongly predicted by the externalizing group factor
Higher-order 2-factor 88 2597.16 .858 .831 .034 .100 (.40). There was also a positive effect of the eating pathology
Higher-order 3-factor 86 668.43 .967 .960 .016 .052 group factor (.21) and an inverse effect of the general factor
Higher-order 4-factor 85 730.95 .964 .955 .017 .056
Bifactor: 2 group factors 74 511.57 .975 .965 .015 .042
(⫺.13).
Bifactor: 3 group factors 74 440.54 .979 .971 .014 .039 Finally, to determine whether the disorder-specific components
Bifactor: 4 group factors 74 592.26 .971 .958 .017 .051 of any diagnosis were correlated with clinical outcomes, we in-
Note. CFI ⫽ comparative fit index; TLI ⫽ Tucker–Lewis Index;
spected modification indices for evidence of residual covariance
RMSEA ⫽ root-mean-square error of approximation; SRMR ⫽ standard- between diagnoses and outcomes. Scaled modification indices and
ized root-mean-square residual. associated expected parameter change (EPC) values are presented
 DIMENSIONAL MODEL OF PSYCHOPATHOLOGY 735

A .55
Major depression

Dysthymia
.41

.52 PTSD

.65 Panic disorder

Internalizing .71
GAD

.55
Specific phobia
.70
.57

Social phobia
.56
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

.26 OCD
This document is copyrighted by the American Psychological Association or one of its allied publishers.

General factor
.38 Bipolar disorder
.59

.23 ADHD

.85
Externalizing Alcohol use disorder

.77
.62
Drug use disorder

.80 Anorexia nervosa

.91
Eating pathology Bulimia nervosa

.79
Binge eating disorder

B .55
Major depression

Dysthymia
.41

.52 PTSD

.65 Panic disorder

Internalizing .71
GAD

.55
Specific phobia
.57

Social phobia
.56

.42
.26 OCD

.38 Bipolar disorder

.43

.23 ADHD

.85
Externalizing Alcohol use disorder

.77
Drug use disorder

.36

.80 Anorexia nervosa

.91
Eating pathology Bulimia nervosa

.79
Binge eating disorder

Figure 1. A. Best-fitting higher-order model. GAD ⫽ generalized anxiety disorder; OCD ⫽ obsessive–
compulsive disorder; PTSD ⫽ posttraumatic stress disorder; ADHD ⫽ attention-deficit/hyperactivity disorder.
B. Best-fitting correlated-factor model. GAD ⫽ generalized anxiety disorder; OCD ⫽ obsessive– compulsive
disorder; PTSD ⫽ posttraumatic stress disorder; ADHD ⫽ attention-deficit/hyperactivity disorder.
 736 CONWAY, MANSOLF, AND REISE

Table 3
Structural Regression Coefficients Predicting Clinical Outcomes From Factors in the Best-Fitting Measurement Models

Model Outcome Predictor Estimate 95% CI Lower 95% CI Upper z p

Higher-order (with 3 first-order factors) Suicide General .50 .46 .53 30.38 ⬍.01
Self-harm General .45 .42 .48 30.72 ⬍.01
Binge drinking General .05 .02 .08 3.60 ⬍.01
Bifactor (with 3 group factors) Suicide General .73 .68 .78 28.45 ⬍.01
Internalizing ⫺.17 ⫺.22 ⫺.12 ⫺6.69 ⬍.01
Externalizing ⫺.20 ⫺.26 ⫺.14 ⫺6.24 ⬍.01
Eating ⫺.34 ⫺.40 ⫺.29 ⫺11.53 ⬍.01
Self-harm General .46 .42 .51 19.34 ⬍.01
Internalizing .02 ⫺.03 .07 .85 .40
Externalizing ⫺.07 ⫺.13 ⫺.01 ⫺2.37 .02
Eating ⫺.07 ⫺.12 ⫺.02 ⫺2.61 .01
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Binge drinking General ⫺.13 ⫺.18 ⫺.09 ⫺6.06 ⬍.01

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Internalizing .06 .02 .10 3.01 ⬍.01

Externalizing .40 .35 .44 16.14 ⬍.01
Eating .21 .16 .26 8.74 ⬍.01
Correlated-factor (with 3 first-order factors) Suicide Internalizing .34 .30 .38 17.01 ⬍.01
Externalizing .17 .12 .22 6.76 ⬍.01
Eating ⫺.04 ⫺.09 .01 ⫺1.63 .10
Self-harm Internalizing .27 .24 .31 15.30 ⬍.01
Externalizing .09 .04 .13 3.54 ⬍.01
Eating .08 .04 .12 3.63 ⬍.01
Binge drinking Internalizing ⫺.20 ⫺.23 ⫺.17 ⫺12.74 ⬍.01
Externalizing .29 .25 .33 15.09 ⬍.01
Eating .07 .03 .11 3.28 ⬍.01
Note. The “General” dimension in the higher-order model refers to the general factor of psychopathology, as operationalized as the second-order factor
in that model. The effects for the first-order factors in the higher-order model are identical to those of the first-order factors for the correlated-factor model
(also shown here).

in Table S5. These EPCs can be interpreted on a standard metric
as estimates of the residual correlations between each diagnosis
and each clinical outcome. Due to the very large sample size,
nearly all modification indices were highly significant, and we
therefore focused on EPC effect sizes. In our best-fitting higher-
order model, the median residual correlation across diagnoses
was ⫺.06, ⫺.02, and ⫺.03 for suicide, self-harm, and binge
drinking, respectively. Of these, major depressive disorder had the
strongest residual relationship with suicide (r ⫽ .24) and self-harm
(r ⫽ .11), and attention-deficit/hyperactivity disorder had the
strongest residual relationship with binge drinking frequency (r ⫽
.20). Median residual effects were similar in magnitude (i.e.,
essentially nil) for the best-fitting correlated-factor model.
Discussion
The Hierarchical Taxonomy of Psychopathology is a quantita-
tive structural model of mental illness that is consistently sup-
ported in large-scale studies of the signs and symptoms of psy-
chopathology. Much of the relevant data for this modeling research
derives from structured diagnostic interviews, which are the gold
standard in assessment research. Paradoxically, this convention
creates a disconnect with everyday practice, where most diagnostic
decisions are based on unsystematic clinical observations (e.g.,
Westen, 1997). The HiTOP model’s ecological validity—that is,
its correspondence to patterns of diagnostic judgments in routine
care—therefore remains uncertain. We analyzed patterns of co-
occurrence among diagnoses assigned in ordinary practice to test
the generalizability of the model. Our results indicate that the
structure of diagnoses assigned “in the field” matches, to a large
extent, the structure recovered in controlled research settings.
This alignment was not a foregone conclusion, because diagno-
sis in routine care often follows a completely different script from
diagnosis in research environments. The demands of many health
care systems (e.g., primary care) tend to limit the scope of prac-
titioners’ evaluations, relative to comprehensive assessment pro-
tocols used in virtually all research contexts. A primary care
physician, for instance, who lacks specialized training in mental
health diagnosis might attend primarily to the most problematic or
subjectively distressing features of the symptom profile, leading to
an incomplete clinical impression (Wilk et al., 2006). This bias
might distort the observed clustering of mental illnesses that forms
the basis for the HiTOP model. Indeed, there are some recent
indications that deviation from diagnostic rules codified in DSM—
and adopted in turn by most published clinical assessment proto-
cols— can alter comorbidity patterns, leading to a different view of
the dimensional architecture of mental illness (Conway & Brown,
2018; Kotov, Ruggero, Krueger, Watson, & Zimmerman, 2018).
Nevertheless, the present findings imply that the current HiTOP
structure is a valid representation of the array of diagnoses as-
signed in clinical practice. This generalizability is essential to the
nosological and assessment aims of the HiTOP consortium, be-
cause the HiTOP system ultimately is intended for use in diverse
health care settings.
Comparing Structural Models of Psychopathology
A correlated-factor model with internalizing, externalizing, and
eating pathology dimensions fit our data well. A higher-order
factor model that added a general factor of psychopathology,
which accounted for the correlations among the three first-order
factors, had equivalent fit. Both models also had interpretable
 DIMENSIONAL MODEL OF PSYCHOPATHOLOGY
patterns of criterion validity. In the correlated-factor model, inter-
nalizing had moderate effects on suicide and self-harm, whereas
the externalizing and eating pathology factors had much smaller
associations with these outcomes. In contrast, for binge drinking,
the externalizing factor had the most predictive power, and inter-
nalizing in fact had a modest protective effect. In the higher-order
model, the general factor had moderate-to-strong effects (standard-
ized coefficients ⬃ ⫽ .50) on suicide and self-injury, whereas it
had virtually no influence on likelihood of binge drinking.
We conclude that the data are consistent with both correlated-
factor and higher-order models, and that each merits continued
research attention. In other words, it does not seem necessary or
appropriate to designate a clear “winner” based on our results. We
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
believe this decision is compatible with a basic assumption of the
This document is copyrighted by the American Psychological Association or one of its allied publishers.
HiTOP framework: mental illness can be understood at various
levels of a hierarchy of psychopathological dimensions (e.g., su-
perspectrum, spectrum, symptom component), depending on the
clinical or research objective (see Kotov et al., 2017, Figure 1). For
instance, our results signal that prediction of suicide and self-harm
might be most efficiently accomplished by assessing general psy-
chopathology, whereas accurate prediction of binge drinking re-
quires distinguishing internalizing versus externalizing pathology.
By way of comparison, personality researchers almost universally
agree that the Big Three, Big Five, and their many lower-order
facets are equally valid representations of the personality domain
(e.g., Markon, Krueger, & Watson, 2005). These traits coexist in
one hierarchically oriented, integrated system.
There were several problematic aspects of the bifactor model.
First, the loading pattern from the best-fitting (3 group factor)
model was inconsistent with expectations. The general factor load-
ings were not uniform, as might have been expected if the general
factor affected vulnerability to all forms of mental illness to a
similar degree. Internalizing group factor loadings were similarly
uneven, and indicated that the dimension probably better captured
fear—with strongest loadings for panic and the phobias—than
broad internalizing problems per se. Further, novel metrics for
judging the robustness of bifactor model dimensions were indic-
ative of suboptimal performance for the internalizing and exter-
nalizing group factors (Hancock’s H ⫽ 0.69, 0.63; Rodriguez et
al., 2016). Just as important, the magnitude and direction of
criterion validity coefficients were often counterintuitive. The
group factors generally had inverse effects on the suicide and
self-injury outcomes, contrary to established theory. It appears that
partialing out the common variance among diagnoses (with the
general factor) distorts the fundamental nature of the internalizing
and externalizing dimensions (cf. Lynam, Hoyle, & Newman,
2006). This observation might in fact be meaningful, rather than
merely reflecting an ill-fitting model. That is, the bifactor model
results might be indicating that, at least in this dataset, the majority
of meaningful psychopathology variance is captured by the general
factor, whereas the group factors reflect nuisance (rather than
substantively important) residual variation. In other words, in this
(fairly unique) dataset of students’ clinician-assigned diagnoses,
the p-factor predominates, whereas the primary (group) factors are
not measured well and have little predictive utility.
Although the best-fitting second-order model resulted in more
interpretable parameter estimates, we do not conclude that the
second-order model can be definitively adopted as the most useful
representation of the hierarchical structure of psychopathology
737
either. For instance, the fit of the bifactor model was slightly
superior to that of the second-order model in the present data;
while this difference in fit may be due to the tendency of the
bifactor model to overfit (Reise, Kim, Mansolf, & Widaman,
2016), it may also mean that the bifactor captures meaningful
variance in self-reported diagnoses that the second-order model
does not. Also, the external correlations of the first-order factors in
the higher-order solution might not have been so interpretable had
we focused on first-order factor disturbances (i.e., variance inde-
pendent of the general factor), paralleling the case of the (orthogo-
nalized) group factors in the bifactor model. The second-order
factor loadings (standardized loading range ⫽ .59 to .70) suggest
that much of the first-order factor variation is attributable to the
general factor. Thus, the meaning and amount of reliable variance
in the first-order factors (e.g., internalizing) might be very different
after partialing out the influence of the p-factor.
To illustrate the similarity of higher-order and bifactor model
solutions, we performed 2 ad hoc analyses. First, we calculated
factor scores for the general factor in the bifactor model and the
second-order factor in the higher-order model and found that the
two sets of scores were correlated at r ⫽ .97. Second, we per-
formed a set of predictive analyses for the bifactor model that was
analogous to those for the second-order model, using only the
general factor in the bifactor model to predict suicide, self-harm,
and binge drinking, and found that the structural coefficients were
nearly identical. Taken together, these results indicate that there is
no meaningful difference between the p-factors in the second-
order and bifactor models, either in terms of scoring or predictive
validity. Future research is needed to identify the number, nature,
and structure of latent variables underlying psychiatric diagnoses,
and it seems unlikely that cross-sectional research examining
correlations among diagnoses and/or outcomes can itself deter-
mine which of the two conceptualizations of the p-factor repre-
sents the true data-generating model.
Assessment Implications
Across all structures (including the bifactor model), we came to
the same conclusion about the incremental validity of categorical
diagnoses, compared to higher-order dimensions. That is, categor-
ical entities (e.g., major depression, social phobia) had little pre-
dictive power, net the transdiagnostic dimensions. For instance, in
the criterion validity analyses involving our best-fitting higher-
order model, the median residual correlation between diagnoses
and clinical outcomes was essentially 0. There was no evidence, in
other words, of diagnosis-specific pathways from mental illness to
clinical problems like suicide potential and binge drinking. This
same general result has been documented repeatedly in prior
HiTOP research focused on clinical outcomes (reviewed in Con-
way et al., in press).
This is a clinically important finding because it suggests that
assessment could reasonably focus on cross-cutting dimensions of
mental illness (e.g., internalizing problems), rather than on all
categorical disorders in a given domain (e.g., DSM–5 anxiety and
depressive disorders). This assessment approach arguably has sev-
eral advantages for clinical care. Because it involves rating patients
on continua, not categories, of mental illness, subsyndromal pa-
thology could be taken into account for prognosis and treatment
decisions, and the temporal reliability of clinical ratings would
 738 CONWAY, MANSOLF, AND REISE
almost certainly improve (e.g., Morey et al., 2012). Moreover, a
dimensional system avoids problematic “wastebasket” categories,
such as DSM–5 “other specified/unspecified” conditions that are
used to label cases that do not fit neatly into established categories.
Instead, a dimensional profile can be constructed to characterize
the salient psychopathology processes in any given case. Dimen-
sional ratings ultimately can be transformed into categories when
needed for binary clinical decisions (e.g., whether to hospitalize),
ideally by referring to empirical norms, as is currently the case for
cognitive functioning in the DSM–5 diagnosis of intellectual dis-
ability.
The HiTOP framework also implies a sequential assessment
procedure governed by a hierarchy of mental illness. Clinicians
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
may initially focus on a specific level of the hierarchy based on
This document is copyrighted by the American Psychological Association or one of its allied publishers.
patient preferences and clinical objectives. For instance, it might
be sensible for brief screening measures to address the superordi-
nate components of the hierarchy, such as the general factor of
psychopathology, whereas a more comprehensive workup (e.g.,
for treatment planning) would evaluate the lower-order symptom
components and maladaptive traits for maximum precision (Kotov
et al., 2017). Also, time permitting, these options are not mutually
exclusive. Assessment might start with the spectrum-level con-
structs, and then cascade downward to lower-order dimensions
when an elevation is detected in a certain domain.
Although the acceptability and clinical utility of this assessment
process have not been systematically tested, there are signs that
dimensional ratings might be valuable for clinical communication,
prognosis, and treatment selection. Practitioner survey data indi-
cate that clinicians prefer dimensions to traditional categories
when communicating with patients, at least for personality con-
structs (Morey, Skodol, & Oldham, 2014). Also, research to date
indicates that higher-order dimensions have superior prognostic
value, relative to categorical diagnoses, for the temporal course of
disorder, suicide risk, and medical morbidity (e.g., Kessler et al.,
2011; Sunderland & Slade, 2015). Finally, there are preliminary
data to suggest that treatment decisions tend to be influenced more
by dimensional constructs (e.g., overall illness severity, aggres-
sion) than any categorical condition (Rodriguez-Seijas et al., 2017;
Waszczuk et al., 2017).
Limitations
A number of study limitations are important to consider. First, it
is possible that some participants misremembered diagnoses—
either Type I or type II errors—assigned by health professionals.
We selected a subsample of university students with recent contact
with treatment providers as a way to mitigate such bias, but
memory distortions could have affected observed diagnostic prev-
alence and comorbidity rates. Related, participants reported on
both psychopathology and clinical outcomes, possibly artificially
inflating the degree of association between them. More research is
needed to test the utility of higher-order dimensions using inde-
pendently assessed external variables (e.g., court records,
informant-reports) to minimize shared method variance. Second,
although participants reported on a wide range of mental disorders,
diagnostic coverage was still limited. Omission (due to low prev-
alence) of schizophrenia-spectrum and personality disorders, for
instance, probably precluded extraction of thought disorder and
detachment dimensions—located alongside internalizing and ex-
ternalizing constructs at the spectrum level of the hierarchy—that
have been previously documented in patient samples (e.g., Forbes
et al., 2017). More generally, the HiTOP structure remains a
“working model” that must be repeatedly updated as new evidence
accrues. Studies that include uncommon signs and symptoms of
mental illness will be important in this regard. Third, most uni-
versity students have not passed through the peak period of risk for
certain disorders (e.g., schizophrenia), and different structures may
fit better in other age groups.
Finally, we caution that the validity and interpretation of the
p-factor remains a pressing, unresolved issue for psychopathology
researchers. Researchers have argued that it reflects negative emo-
tionality or general distress that sets the stage for the full spectrum
of mental health problems (e.g., Tackett et al., 2013). Alternately,
it could represent a unitary outcome of mental illness, akin to the
concept of demoralization proposed as a hallmark of all types of
psychopathology (Tellegen et al., 2003). That is, the p-factor could
represent dysfunction that occurs as a common endpoint for var-
ious mental disorders (Widiger & Oltmanns, 2017). More theoret-
ical and empirical work is needed to adjudicate among these
plausible alternatives.
Conclusion
In sum, quantitative models of mental illness provided a good fit
to diagnoses assigned in routine clinical care. Transdiagnostic
factors outperformed categorical diagnoses in tests of criterion
validity, underscoring the potential utility of higher-order dimen-
sions for clinical and research purposes. Together, these observa-
tions offer the first evidence for the ecological validity of the
HiTOP model and suggest that it may be a useful clinical assess-
ment tool in everyday health care settings.
References
Achenbach, T. M. (1966). The classification of children’s psychiatric
symptoms: A factor-analytic study. Psychological Monographs, 80,
1–37. http://dx.doi.org/10.1037/h0093906
Bonifay, W. E. (2015). On the complexity of IRT models [Abstract].
Multivariate Behavioral Research, 50, 128. http://dx.doi.org/10.1080/
00273171.2014.988988
Brown, T. A. (2015). Confirmatory factor analysis for applied research
(2nd ed.). New York, NY: Guilford Press Publications.
Carragher, N., Teesson, M., Sunderland, M., Newton, N. C., Krueger,
R. F., Conrod, P. J., . . . Slade, T. (2016). The structure of adolescent
psychopathology: A symptom-level analysis. Psychological Medicine,
46, 981–994. http://dx.doi.org/10.1017/S0033291715002470
Caspi, A., Houts, R. M., Belsky, D. W., Goldman-Mellor, S. J., Harrington,
H., Israel, S., . . . Moffitt, T. E. (2014). The p factor: One general
psychopathology factor in the structure of psychiatric disorders? Clini-
cal Psychological Science, 2, 119 –137. http://dx.doi.org/10.1177/
2167702613497473
Caspi, A., & Moffitt, T. E. (2018). All for one and one for all: Mental
disorders in one dimension. American Journal of Psychiatry, 175, 831–
844.
Chmielewski, M., Clark, L. A., Bagby, R. M., & Watson, D. (2015).
Method matters: Understanding diagnostic reliability in DSM–IV and.
DSM–5. Journal of Abnormal Psychology, 124, 764 –769. http://dx.doi
.org/10.1037/abn0000069
Clark, L. A. (2007). Assessment and diagnosis of personality disorder:
Perennial issues and an emerging reconceptualization. Annual Review of
 DIMENSIONAL MODEL OF PSYCHOPATHOLOGY
Psychology, 58, 227–257. http://dx.doi.org/10.1146/annurev.psych.57
.102904.190200
Clark, L. A., Cuthbert, B., Lewis-Fernández, R., Narrow, W. E., & Reed,
G. M. (2017). Three Approaches to Understanding and Classifying
Mental Disorder: ICD-11, DSM–5, and the National Institute of
Mental Health’s Research Domain Criteria (RDoC). Psychological
Science in the Public Interest, 18, 72–145. http://dx.doi.org/10.1177/
1529100617727266
Conway, C. C., & Brown, T. A. (2018). Evaluating dimensional models of
psychopathology in outpatients diagnosed with emotional disorders: A
cautionary tale. Depression and Anxiety, 35, 898 –902. http://dx.doi.org/
10.1002/da.22740
Conway, C. C., Forbes, M. K., Forbush, K. T., Fried, E. I., & Eaton, N. R.
(in press). A Hierarchical Taxonomy of Psychopathology can transform
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
mental health research. Perspectives on Psychological Science.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Eaton, N. R., Krueger, R. F., Markon, K. E., Keyes, K. M., Skodol, A. E.,
Wall, M., . . . Grant, B. F. (2013). The structure and predictive validity
of the internalizing disorders. Journal of Abnormal Psychology, 122,
86 –92.
Eaton, N. R., Rodriguez-Seijas, C., Carragher, N., & Krueger, R. F. (2015).
Transdiagnostic factors of psychopathology and substance use disorders:
A review. Social Psychiatry and Psychiatric Epidemiology, 50, 171–
182. http://dx.doi.org/10.1007/s00127-014-1001-2
Eisenberg, D., Hunt, J., & Speer, N. (2013). Mental health in American
colleges and universities: Variation across student subgroups and across
campuses. Journal of Nervous and Mental Disease, 201, 60 – 67. http://
dx.doi.org/10.1097/NMD.0b013e31827ab077
Forbes, M. K., Kotov, R., Ruggero, C. J., Watson, D., Zimmerman, M., &
Krueger, R. F. (2017). Delineating the joint hierarchical structure of
clinical and personality disorders in an outpatient psychiatric sample.
Comprehensive Psychiatry, 79, 19 –30. http://dx.doi.org/10.1016/j
.comppsych.2017.04.006
Forbush, K. T., South, S. C., Krueger, R. F., Iacono, W. G., Clark, L. A.,
Keel, P. K., . . . Watson, D. (2010). Locating eating pathology within an
empirical diagnostic taxonomy: Evidence from a community-based sam-
ple. Journal of Abnormal Psychology, 119, 282–292. http://dx.doi.org/
10.1037/a0019189
Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability
within latent variable systems. Structural equation modeling: Present
and future a Festschrift in honor of Karl Joreskog (pp. 195–216).
Uppsala, Sweden: Scientific Software International.
Harkavy-Friedman, J. M. (2009). Review of Dimensional approaches in
diagnostic classification: Refining the research agenda for DSM-V,
edited by Helzer, J., Kraemer, H. C., Krueger, R. F., Wittchen, H. U.,
Sirovatka, P. J., & Regier, D. A. The American Journal of Psychiatry,
166, 118 –119.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in
covariance structure analysis: Conventional criteria versus new alterna-
tives. Structural Equation Modeling, 6, 1–55. http://dx.doi.org/10.1080/
10705519909540118
Kessler, R. C., Petukhova, M., & Zaslavsky, A. M. (2011). The role of
latent internalizing and externalizing predispositions in accounting for
the development of comorbidity among common mental disorders. Cur-
rent Opinion in Psychiatry, 24, 307–312. http://dx.doi.org/10.1097/YCO
.0b013e3283477b22
Keyes, K. M., Eaton, N. R., Krueger, R. F., McLaughlin, K. A., Wall,
M. M., Grant, B. F., & Hasin, D. S. (2012). Childhood maltreatment and
the structure of common psychiatric disorders. The British Journal of
Psychiatry, 200, 107–115. http://dx.doi.org/10.1192/bjp.bp.111.093062
Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R.,
Bagby, R. M., . . . Zimmerman, M. (2017). The Hierarchical Taxonomy
of Psychopathology (HiTOP): A dimensional alternative to traditional
nosologies. Journal of Abnormal Psychology, 126, 454 – 477. http://dx
.doi.org/10.1037/abn0000258
739
Kotov, R., Perlman, G., Gámez, W., & Watson, D. (2015). The structure
and short-term stability of the emotional disorders: A dimensional ap-
proach. Psychological Medicine, 45, 1687–1698. http://dx.doi.org/10
.1017/S0033291714002815
Kotov, R., Ruggero, C. J., Krueger, R. F., Watson, D., & Zimmerman, M.
(2018). The perils of hierarchical exclusion rules: A further word of
caution. Depression and Anxiety, 35, 903–904. http://dx.doi.org/10
.1002/da.22826
Krueger, R. F. (1999). The structure of common mental disorders. Archives
of General Psychiatry, 56, 921–926. http://dx.doi.org/10.1001/archpsyc
.56.10.921
Krueger, R. F., Kotov, R., Watson, D., Forbes, M. K., Eaton, N. R.,
Ruggero, C. J., . . . Zimmermann, J. (2018). Progress in achieving
quantitative classification of psychopathology. World Psychiatry, 17,
282–293. http://dx.doi.org/10.1002/wps.20566
Krueger, R. F., Markon, K. E., Patrick, C. J., Benning, S. D., & Kramer,
M. D. (2007). Linking antisocial behavior, substance use, and person-
ality: An integrative quantitative model of the adult externalizing spec-
trum. Journal of Abnormal Psychology, 116, 645– 666. http://dx.doi.org/
10.1037/0021-843X.116.4.645
Laceulle, O. M., Vollebergh, W. A. M., & Ormel, J. (2015). The structure
of psychopathology in adolescence: Replication of a general psychopa-
thology factor in the TRAILS Study. Clinical Psychological Science, 3,
1–11. http://dx.doi.org/10.1177/2167702614560750
Lahey, B. B., Applegate, B., Hakes, J. K., Zald, D. H., Hariri, A. R., &
Rathouz, P. J. (2012). Is there a general factor of prevalent psychopa-
thology during adulthood? Journal of Abnormal Psychology, 121, 971–
977. http://dx.doi.org/10.1037/a0028355
Lahey, B. B., Krueger, R. F., Rathouz, P. J., Waldman, I. D., & Zald, D. H.
(2017). A hierarchical causal taxonomy of psychopathology across the
life span. Psychological Bulletin, 143, 142–186. http://dx.doi.org/10
.1037/bul0000069
Lahey, B. B., Rathouz, P. J., Keenan, K., Stepp, S. D., Loeber, R., &
Hipwell, A. E. (2015). Criterion validity of the general factor of psy-
chopathology in a prospective study of girls. Journal of Child Psychol-
ogy and Psychiatry, 56, 415– 422. http://dx.doi.org/10.1111/jcpp.12300
Lynam, D. R., Hoyle, R. H., & Newman, J. P. (2006). The perils of
partialling: Cautionary tales from aggression and psychopathy. Assess-
ment, 13, 328 –341. http://dx.doi.org/10.1177/1073191106290562
Mansolf, M., & Reise, S. P. (2017). When and why the second-order and
bifactor models are distinguishable. Intelligence, 61, 120 –129. http://dx
.doi.org/10.1016/j.intell.2017.01.012
Markon, K. E., Krueger, R. F., & Watson, D. (2005). Delineating the
structure of normal and abnormal personality: An integrative hierarchi-
cal approach. Journal of Personality and Social Psychology, 88, 139 –
157. http://dx.doi.org/10.1037/0022-3514.88.1.139
Morey, L. C., Hopwood, C. J., Markowitz, J. C., Gunderson, J. G., Grilo,
C. M., McGlashan, T. H., . . . Skodol, A. E. (2012). Comparison of
alternative models for personality disorders, II: 6-, 8- and 10-year
follow-up. Psychological Medicine, 42, 1705–1713. http://dx.doi.org/10
.1017/S0033291711002601
Morey, L. C., Skodol, A. E., & Oldham, J. M. (2014). Clinician judgments
of clinical utility: A comparison of DSM–IV–TR personality disorders
and the alternative model for DSM–5 personality disorders. Journal of
Abnormal Psychology, 123, 398 – 405. http://dx.doi.org/10.1037/
a0036481
Murray, A. L., & Johnson, W. (2013). The limitations of model fit in
comparing the bi-factor versus higher-order models of human cognitive
ability structure. Intelligence, 41, 407– 422. http://dx.doi.org/10.1016/j
.intell.2013.06.004
Oltmanns, J. R., Smith, G. T., Oltmanns, T. F., & Widiger, T. A. (2018).
General factors of psychopathology, personality, and personality disor-
der: Across domain comparisons. Clinical Psychological Science, 6,
581–589. http://dx.doi.org/10.1177/2167702617750150
 740 CONWAY, MANSOLF, AND REISE
Patalay, P., Fonagy, P., Deighton, J., Belsky, J., Vostanis, P., & Wolpert,
M. (2015). A general psychopathology factor in early adolescence. The
British Journal of Psychiatry, 207, 15–22. http://dx.doi.org/10.1192/bjp
.bp.114.149591
Preacher, K. J. (2006). Quantifying parsimony in structural equation mod-
eling. Multivariate Behavioral Research, 41, 227–259. http://dx.doi.org/
10.1207/s15327906mbr4103_1
Regier, D. A., Narrow, W. E., Clarke, D. E., Kraemer, H. C., Kuramoto,
S. J., Kuhl, E. A., & Kupfer, D. J. (2013). DSM–5 field trials in the
United States and Canada, Part II: Test-retest reliability of selected
categorical diagnoses. The American Journal of Psychiatry, 170, 59 –70.
http://dx.doi.org/10.1176/appi.ajp.2012.12070999
Reise, S. P. (2012). The rediscovery of bifactor measurement models.
Multivariate Behavioral Research, 47, 667– 696. http://dx.doi.org/10
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
.1080/00273171.2012.715555
Reise, S. P., Kim, D. S., Mansolf, M., & Widaman, K. F. (2016). Is the
This document is copyrighted by the American Psychological Association or one of its allied publishers.
bifactor model a better model or is it just better at modeling implausible
responses? Application of iteratively reweighted least squares to the
Rosenberg Self-Esteem Scale. Multivariate Behavioral Research, 51,
818 – 838. http://dx.doi.org/10.1080/00273171.2016.1243461
Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and
rotations: Exploring the extent to which multidimensional data yield
univocal scale scores. Journal of Personality Assessment, 92, 544 –559.
http://dx.doi.org/10.1080/00223891.2010.496477
Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2013).
Multidimensionality and structural coefficient bias in structural equation
modeling: A bifactor perspective. Educational and Psychological Mea-
surement, 73, 5–26. http://dx.doi.org/10.1177/0013164412449831
Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor
models: Calculating and interpreting statistical indices. Psychological
Methods, 21, 137–150. http://dx.doi.org/10.1037/met0000045
Rodriguez-Seijas, C., Eaton, N. R., Stohl, M., Mauro, P. M., & Hasin, D. S.
(2017). Mental disorder comorbidity and treatment utilization. Compre-
hensive Psychiatry, 79, 89 –97. http://dx.doi.org/10.1016/j.comppsych
.2017.02.003
Rosseel, Y. (2012). Lavaan: An R package for structural equation model-
ing and more (Version 0.5–12 (BETA)). Belgium: Ghent University.
Saris, W. E., Satorra, A., & Sörbom, D. (1987). The detection and correc-
tion of specification errors in structural equation models. Sociological
Methodology, 17, 105–129. http://dx.doi.org/10.2307/271030
Snyder, H. R., Young, J. F., & Hankin, B. L. (2017). Strong homotypic
continuity in common psychopathology-, internalizing-, and
externalizing-specific factors over time in adolescents. Clinical
Psychological Science, 5, 98 –110. http://dx.doi.org/10.1177/
2167702616651076
Sörbom, D. (1989). Model modification. Psychometrika, 54, 371–384.
http://dx.doi.org/10.1007/BF02294623
Sunderland, M., & Slade, T. (2015). The relationship between internalizing
psychopathology and suicidality, treatment seeking, and disability in the
Australian population. Journal of Affective Disorders, 171, 6 –12. http://
dx.doi.org/10.1016/j.jad.2014.09.012
Tackett, J. L., Lahey, B. B., van Hulle, C., Waldman, I., Krueger, R. F., &
Rathouz, P. J. (2013). Common genetic influences on negative emotion-
ality and a general psychopathology factor in childhood and adoles-
cence. Journal of Abnormal Psychology, 122, 1142–1153. http://dx.doi
.org/10.1037/a0034151
Tellegen, A., Ben-Porath, Y. S., McNulty, J. L., Arbisi, P. A., Graham,
J. R., & Kaemmer, B. (2003). MMPI–2 Restructured Clinical Scales:
Development, validation, and interpretation. Minneapolis: University of
Minnesota Press.
Verheul, R., & Widiger, T. A. (2004). A meta-analysis of the prevalence
and usage of the personality disorder not otherwise specified (PDNOS)
diagnosis. Journal of Personality Disorders, 18, 309 –319. http://dx.doi
.org/10.1521/pedi.2004.18.4.309
Waszczuk, M. A., Zimmerman, M., Ruggero, C., Li, K., MacNamara, A.,
Weinberg, A., . . . Kotov, R. (2017). What do clinicians treat: Diagnoses
or symptoms? The incremental validity of a symptom-based, dimen-
sional characterization of emotional disorders in predicting medication
prescription patterns. Comprehensive Psychiatry, 79, 80 – 88. http://dx
.doi.org/10.1016/j.comppsych.2017.04.004
Watson, D. (2005). Rethinking the mood and anxiety disorders: A quan-
titative hierarchical model for DSM-V. Journal of Abnormal Psychol-
ogy, 114, 522–536. http://dx.doi.org/10.1037/0021-843X.114.4.522
Westen, D. (1997). Divergences between clinical and research methods for
assessing personality disorders: Implications for research and the evo-
lution of axis II. The American Journal of Psychiatry, 154, 895–903.
http://dx.doi.org/10.1176/ajp.154.7.895
Widiger, T. A., & Oltmanns, J. R. (2017). The general factor of psycho-
pathology and personality. Clinical Psychological Science, 5, 182–183.
http://dx.doi.org/10.1177/2167702616657042
Wilk, J. E., West, J. C., Narrow, W. E., Marcus, S., Rubio-Stipec, M., Rae,
D. S., . . . Regier, D. A. (2006). Comorbidity patterns in routine
psychiatric practice: Is there evidence of underdetection and underdiag-
nosis? Comprehensive Psychiatry, 47, 258 –264. http://dx.doi.org/10
.1016/j.comppsych.2005.08.007
Zimmerman, M., & Mattia, J. I. (1999). Psychiatric diagnosis in clinical
practice: Is comorbidity being missed? Comprehensive Psychiatry, 40,
182–191. http://dx.doi.org/10.1016/S0010-440X(99)90001-9
Received March 9, 2018
Revision received November 20, 2018
Accepted December 10, 2018 䡲