Version of Record: https://www.sciencedirect.

com/science/article/pii/S0005796721001716
Manuscript_280d5d41535349c8a30f2f7cc5d78ba4

The Effect of Depression and Anxiety Symptom Severity on Clinical Outcomes and App Use in
Digital Mental Health Treatments: Meta-Regression of Three Trials

David C. Mohr, PhD,1 Mary J. Kwasny, ScD,1 Jonah Meyerhoff, PhD,1 Andrea K. Graham,
PhD,1 Emily G. Lattie, PhD.1

1. Center for Behavioral Intervention Technologies, Northwestern University, Chicago, IL

Corresponding Author
David C. Mohr, PhD
Center for Behavioral Intervention Technologies
Northwestern University
750 N. Lakeshore Drive, 10th Floor
Chicago, IL 60611
Email: d-mohr@northwestern.edu

© 2021 published by Elsevier. This manuscript is made available under the Elsevier user license
https://www.elsevier.com/open-access/userlicense/1.0/
ABSTRACT
A large number of trials have consistently shown that guided digital mental health treatments
(DMHTs) are effective for depression and anxiety. As DMHTs are adopted by healthcare
organizations, payers, and employers, they are often considered most appropriate for people with
mild-to-moderate levels of symptom severity. Thus, the aim of this study was to examine the
effects of symptom severity on depression and anxiety outcomes and app use across three trials
of a guided DMHT, IntelliCare. Participants were categorized into mild, moderate, moderately
severe, and severe symptom severity groups on depression and anxiety. All symptom severity
groups showed significant reductions in depression and anxiety in a clear ordinal pattern, with
the mild symptom severity group showing the smallest changes and the severe symptom group
showing the largest improvements. Those with the lowest levels of educational attainment
showed the largest symptom improvement. Baseline symptom severity was not significantly
related to app use. App use was significantly related to depression and anxiety outcomes. These
findings suggest that depression and anxiety symptom severity is not useful in determining who
should be referred to a guided DMHT.

Keywords
Depression; Anxiety; eHealth; mHealth
INTRODUCTION
Depression and anxiety are common, affecting 7% and 19% respectively of Americans in
any given year (Substance Abuse and Mental Health Services Administration (SAMHSA),
2017), and take an enormous toll in terms of cost, morbidity, suffering, and mortality (Wells, et
al., 2002; Whooley & Simon, 2000; Whooley, Stone, & Soghikian, 2000). There is a shortage of
mental healthcare specialists, which will not likely improve (Kazdin & Blase, 2011).
Digital mental health treatments (DMHTs), which teach and support self-management
skills via commonly available devices such as smartphones, tablets, and computers, could
overcome both access and provider shortage problems (Mohr, et al., 2021; National Advisory
Mental Health Council, et al., 2017). A large number of randomized controlled trials (RCTs)
have consistently demonstrated that DMHTs are effective for depression and anxiety (Andrews,
et al., 2018; Linardon, Cuijpers, Carlbring, Messer, & Fuller-Tyszkiewicz, 2019; Weisel, et al.,
2019). While self-guided DMHTs only produce modest benefits (Karyotaki, et al., 2017), effect
sizes are similar to those seen for psychotherapy when the DMHT is supported by a human
coach or care manager (Andersson, Topooco, Havik, & Nordgreen, 2016; Nicholas, et al., 2021).
Typically, this support aims to maintain patient adherence to the app and monitor progress
through periodic symptom assessment but may also include assisting patients in understanding
concepts or skills training, and triaging patients who do not respond to the DMHT to a higher
level of care (Mohr, Cuijpers, & Lehman, 2011; Schueller, Tomasino, & Mohr, 2016).
Given the strong evidence supporting DMHTs for common mental health problems, there
are a growing number of companies offering products in a variety of settings, including
healthcare organizations, payers, self-insured employers, and direct-to-consumer. There is
frequently an assumption by healthcare organizations and payers that DMHTs are most
appropriate for people with mild to moderate levels of severity. In addition, there has been an
assumption that DMHT usage by patients must be consistently strong to see strong outcomes, a
concern that has arisen in part due to the poor engagement often seen in real-world deployments
(Gilbody, et al., 2017; Gilbody, et al., 2015).
To address the question of whether DMHTs are effective for people with more severe
depression and anxiety symptom severity, we use all available data for one DMHT, the
IntelliCare platform. IntelliCare, supported by low-intensity coaching, has produced strong
patient engagement and substantial reductions in depression and anxiety symptom severity
across three trials (Graham, et al., 2020; Mohr, Schueller, et al., 2019; Mohr, et al., 2017),
however, the impact of symptom severity on outcomes has not been rigorously investigated. In
this combined data analysis, we examine four questions: 1) Are there are significant differences
in the effectiveness of IntelliCare for the treatment of depression and anxiety at mild, moderate,
moderate-severe, and severe symptom severity levels; 2) Does co-morbidity of anxiety and
depression impact outcomes for IntelliCare; 3) Do patient characteristics, including age, race,
and ethnicity on depressive and anxiety severity outcomes moderate outcomes; and 4) Does
baseline symptom severity and patient characteristics affect app use?
METHODS
This is a secondary cumulative trial analysis using all participants who received
IntelliCare with coaching across three trials. While details of each trial are available in the
primary outcome papers (Graham, et al., 2020; Mohr, Schueller, et al., 2019; Mohr, et al., 2017),
we provide a summary here.
Settings
 Study 1 recruited participants from both a large midwestern healthcare system and
through social media (Mohr, et al., 2017). Study 2 recruited participants through social media,
research registries, print advertising, and commercial research panels (Mohr, Schueller, et al.,
2019). Study 3 recruited participants through a southern healthcare system (Graham, et al.,
2020).
Participants
In all three studies, participants were included if they met criteria for depression (PHQ-
9≥10; (Kroenke, Spitzer, & Williams, 2001)) or anxiety (GAD-7≥8; (Spitzer, Kroenke,
Williams, & Lowe, 2006)); were 18 years or older (aged 19 years if in Nebraska, given age of
consent); resided in the United States; and could speak and read English. Participants were
excluded if they had visual, voice, motor, or hearing impairments that would prevent
participation; met diagnostic criteria for a severe psychiatric disorder such as psychotic or
bipolar disorders for which study treatments would be inappropriate; imminent suicidality that
included both a plan and intent; or had initiated or modified antidepressant pharmacotherapy in
the previous 14 days.
Intervention
Treatment length was 8 weeks across all studies. All participants received access to the
IntelliCare platform apps. In Studies 1 and 2, the platform included 12 clinical apps, each of
which was designed to target a specific behavioral or psychological treatment strategy (e.g., goal
setting, behavioral activation, social support, living one’s values, cognitive restructuring,
emotion regulation, positive self-affirmations, coping, exercise for mood, sleep hygiene,
relaxation, and psychoeducation with reminders) and improve symptoms of depression and
anxiety through efficacious treatment strategies. Most apps were designed to require less than 30
seconds to use. Apps included automated reminders to encourage both app use and
implementation of the psychological or behavioral strategies. These apps were developed in
Android only for these first phases. Study 3 consisted of a rebuilt platform that was delivered on
both Android and iOS, but only included a subset of 5 of the 12 original apps. Some apps were
dropped due to low user engagement in previous work, and others were not included due to
software development delays. In all studies, the user’s experience with the clinical apps was
coordinated through a Hub app that provided psychoeducational materials, administered weekly
assessments, and facilitated downloading and use of the apps. The Hub app launched weekly
reminders to complete assessments and weekly recommendations to try new apps. The clinical
apps produced daily notifications either reminding the individual to use the app or a reminder of
how to implement the skills being taught.
The coaching protocol was consistent across all studies. Coaching was guided by the
IntelliCare coaching manual (Noth, Bardsley, Lattie, & Mohr, 2018), which is based on the
supportive accountability model (Mohr, et al., 2011) and the efficiency model (Schueller, et al.,
2016), and was aimed primarily at encouraging participants to try each of the apps, answering
questions about how to use the tools represented in the apps and the rationale behind the skills
taught by the apps, encouraging application of the skills in daily life, and providing some
technical support as needed. Coaching began with an initial 30-45 minute engagement phone call
to explain the program, understand the participant’s goals for mood and anxiety management, set
expectations for the coach-participant relationship, build rapport, and ensure the Hub app was
properly installed on the participant’s phone. Coaches were consistently able to accomplish the
onboarding within this timeframe. After the initial engagement call, participants received 2 to 3
text messages per week from their coach to provide support in using apps, offer encouragement,
reinforce app use, and check-in on progress or challenges. A bank of message templates was
available to coaches. Coaches also responded to all participant-initiated text messages within 1
business day. Coaches offered but did not require an additional 10-minute call around mid-
treatment to support engagement. Fewer than half the patients in each trial requested a mid-
treatment call; when these occurred, they lasted 13 minutes. The coaches had an online
dashboard that provided information about the IntelliCare apps on each participant’s phone,
including which apps were installed, when they were downloaded, each time an app was used,
and which apps were selected as primary in the Hub app. The dashboard also included a text
messaging tool, a section for brief non-participant-facing notes, and an alert indicating when no
IntelliCare app had been used for 3 days or when a participant sent a text message indicating
they might be at risk for self-harm, which resulted in an automated safety response and prompted
coaches to reach out to participants. Coaches had at least a bachelor’s degree in psychology or a
related field and were trained and supervised by a clinical psychologist. Training consisted of
reading the coaching manual (Noth, et al., 2018), using the IntelliCare apps, 2-4 hours of didactic
training, and practicing with at least 3 mock patients. Fidelity was monitored by the supervisor
through review of randomly selected audio recorded onboarding calls and text logs.
Trial Designs
The trials varied in design. Study 1 was a single arm trial (Mohr, et al., 2017). Study 2
was a factorial design, randomizing participants to receive a coach vs. self-guided and a weekly
recommendations to use a specific app vs. no recommendations (only the coached arms were
used in this analysis). An independent statistician provided a sequentially masked randomization
scheme, using blocks of 4, prior to start of the trial (Mohr, Schueller, et al., 2019). Study 3 was a
randomized controlled wait-list controlled trial conducted in a primary care setting. An
independent statistician provided a sequentially masked randomization scheme, using permutated
blocks of 4 to 6, prior to the start of the trial (Graham, et al., 2020). We note that data from
participants who received IntelliCare during as part of the waitlist control were included in this
analysis, using the outcome measurements for start of treatment, mid-treatment, and end of
treatment.
Outcome Assessment
Depression was measured using the PHQ-9 (Kroenke, et al., 2001), and anxiety was
measured using the GAD-7 (Spitzer, et al., 2006). The PHQ-9 is a self-rating of the symptoms of
major depressive disorder, while the GAD-7 is self-rating of the symptom of generalized anxiety
disorder. These measures were administered online through REDCap (Harris, et al., 2009) at
baseline, and week 8 (end of treatment).
PHQ-9 and GAD-7 scores were classified into mild (<10), moderate (10-14), moderately-
severe (15-19), and severe (≥20) based on widely used criteria that have been broadly
implemented in healthcare (Kroenke, Spitzer, Williams, & Lowe, 2010).
App use was defined using two commonly used metrics (Pham, et al., 2019): number of
app sessions, and days of app use. The number of app sessions was defined as the total number
of “sessions” across the study period. A session was operationalized as a sequence of user-
initiated actions or events on the app, separated by less than 5 minutes between events. A new
app launch (or session) was defined as a new activity after 5 minutes of no activity. Days of app
use was the number of days during treatment that an app was used.
Statistical Analyses
Patient demographics are presented using means and standard deviations, medians and
interquartile ranges, or counts and percentages and compared across the three studies using
ANOVA, Kruskal Wallis, or Chi-squared tests, respectively. In general, comparisons across
severity, and predictive models for use metrics or depression and anxiety outcomes were
modeled using generalized linear mixed models, which allowed heterogeneity across study in the
effect size (i.e., random intercept for each study) and separate residual variances for each study.
Table 2 presents otherwise unadjusted least square means of PHQ-9 and GAD-7 at baseline,
week 8, and the mean change. We modeled the change in outcome avoid overfitting, which
might occur by estimating additional subject-specific variance parameters when looking at
measures over time. A secondary analysis was run, further adjusting for covariates that impacted
the outcome or differed by study. Least square means are showed in Figures 1 and 2.
To assess the impact of demographics on change in outcomes, separate models were fit
for each potential predictor. Multiple predictors were not assessed together due to limited sample
sizes and the need to account for the effects of study. As the app use metrics varied across
studies, and were bimodal and skewed, quartiles of use were created within each study and then
modeled using a cumulative logit link. Models were fit to assess associations between use and
participant characteristics. Further models were examined to assess if moderation existed
between demographics and use metrics on change in outcome.
RESULTS
Participants
Table 1 displays the participant characteristics by study. There were a few significant
differences in characteristics across the studies. Participants in Study 3 (conducted in primary
care) were significantly older, less educated, had less income, and were less likely to be in
psychotherapy than participants in the other studies. A greater proportion of Study 3 participants
were Black or African-American and fewer were White. Participants in Study 2 had higher
baseline anxiety, were more likely to be in psychotherapy, but less likely to be taking
antidepressant medication relative to the other two studies. Study 1 participants were more likely
to have comorbid depression and anxiety.
Outcomes by Severity
Table 2 displays the PHQ-9 and GAD-7 outcomes by severity. There were significant
reductions in both PHQ-9 and GAD-7 at every severity level (ps<0.001), with the largest drops
in symptom severity appearing in the highest severity categories. Models were further adjusted
for education, therapy, comorbid depression/anxiety, race, sex as a biological variable, and age.
Those showed little difference in inference (means and 95% confidence intervals are presented in
Figures 1 and 2).
Relationship between Participant Characteristics and Outcome.
Table 3 displays the relationship between participant characteristics and changes in PHQ-
9 and GAD-7. Participants who had not completed a college degree were more likely to improve
on both PHQ-9 and GAD-7, relative those with college or advanced degrees. Higher baseline
PHQ-9 scores were associated with greater improvement in depression but were not associated
with improvement on GAD-7, while higher baseline GAD-7 scores were associated with greater
reductions in anxiety but not in depression. Participants with comorbid depression and anxiety
showed greater improvement on both depression and anxiety, relative to those with depression or
anxiety only. Concurrent psychotherapy was associated with a greater reduction in PHQ-9
scores, however there was no significant effect for GAD-7. There was a trend towards a
relationship of race and GAD-7, with Black or African-American participants showing a greater
reduction in anxiety than White participants. There were no significant effects for age, gender,
ethnicity, marital status, employment, or income.
App Use
There were significant and substantial differences in all app use metrics across the studies
(see Table 1). Nevertheless, even in the Study 3, which had the lowest level of use, adherence
was strong, with a median of 95 app sessions, on 27 out of 56 days (8 weeks). Table 3 displays
the relationship between app use metrics and outcomes.
Table 4 displays the relationship between both metrics of app use and participant
characteristics. Generally, being older, female, identifying as White, and current medication
were associated with higher app use statistics.
Relationship of App Use and Week 8 PHQ-9
Number of app sessions significantly predicted week 8 PHQ-9 after adjusting for baseline
PHQ-9 (β=-0.005 (SE= 0.001), p<0.001). This persisted after adjusting for education and
psychotherapy (app session regression parameter β=-0.005 (SE=0.001), p<0.001). There was no
evidence of moderation by education (F(2, 358)=0.33, p=0.716), nor current psychotherapy (F(1,
359)=0.72, p=0.40). Additionally, after adjusting for number of app sessions, the effect of
psychotherapy was no longer significant (F(1, 360)=0.60, P=0.44), and was removed from the
model. After further adjustment for comorbid depression and anxiety, medication, age, sex, and
race, number app sessions used continued to be a significant predictor of week 8 PHQ-9 (β=-
0.006 (SE=0.001), p<0.001). There was no indication of moderation by medication (F(1,
352)=0.92, p=0.38), age (F(1, 352)=0.00, p=0.98), gender (F(1, 352)=1.44, p=0.23), or race (F(2,
351)=0.26, p=0.77). There was, however, evidence of moderation by comorbid depression and
anxiety (F(1, 351)=3.95, p=0.048) such that the relationship between number of app sessions and
PHQ-9 was stronger among those with comorbidity than those without.
Number of days used significantly predicted week 8 PHQ-9 after adjusting for baseline
PHQ-9 (β=-0.06 (SE=0.02), p=0.001). This persisted after adjusting for education and
psychotherapy (number of days regression parameter β=-0.06 (SE=0.02), p=0.001). There was
no evidence of moderation by education (F(2, 358)=0.80, p=0.45), nor current psychotherapy
(F(1, 359)=0.31, p=0.58). Additionally, after adjusting for number of days used, the effect of
psychotherapy was no longer significant (F(1, 360)=0.25, p=0.62), and was removed from the
model. After further adjustment for medication, comorbidity, age, sex, and race, number of days
used continued to be a significant predictor of week 8 PHQ-9 (β=-0.08 (SE=0.02), p<0.001).
There was no indication of moderation by medication (F(1, 352)=0.44, p=0.51), age (F(1,
352)=0.11, p=0.74), gender (F(1, 352)=2.37, p=0.13), or race (F(2, 351)=0.79, p=0.46). There
was, however, evidence of moderation by comorbidity (F(1, 351)=4.90, p=0.03), such that the
relationship between number of days of app use and improvement was greater among those with
comorbid anxiety than those without.
Relationship of App Use and Week 8 GAD-7
Number of app sessions significantly predicted week 8 GAD-7 after adjusting for
baseline GAD-7 (β=-0.004 (SE=0.001), p=0.002). This persisted after adjusting for education
and psychotherapy (number of app sessions regression parameter (β=-0.004 (SE=0.001),
p=0.002). There was no evidence of moderation by education (F(2, 359)=0.24, p=0.784), nor
current psychotherapy (F(1, 360)=0.03, p=0.85). Additionally, after adjusting for number of app
sessions, the effects of education and therapy were no longer significant (F(2, 361)=1.39, p=0.25,
F(1, 361)=0.03, p=0.86, respectively), and were removed from the model. After further
adjustment for comorbid depression and anxiety, medication, age, sex, and race, number of app
session continued to be a significant predictor of week 8 GAD-7 (β=-0.005 (SE=0.001),
P<0.001). There was no indication of moderation by comorbidity (F(1, 354)=2.52, p=0.11),
medication (F(1, 355)=0.08, P=0.77), age (F(1, 355)=1.52, p=0.22), gender (F(1, 355)=0.45,
P=0.50), or race (F(2, 354)=0.05, P=0.95).
Number of days used significantly predicted week 8 GAD-7 after adjusting for baseline
GAD-7 (β=-0.04 (SE=0.02), p=0.022). This persisted after adjusting for education and
psychotherapy (number of days regression parameter β=-0.04 (SE=0.02), p=0.019). There was
no evidence of moderation by education (F(2, 359)=1.29, p=0.28), nor current psychotherapy
(F(1, 360)=0.21, p=0.65). Additionally, after adjusting for number of days used, the effects of
education and psychotherapy were no longer significant (F(2, 361)=1.49, p=0.23, F(1,
361)=0.00, p=0.98, respectively), and were removed from the model. After adjustment for
comorbid depression/anxiety, medication, age, sex, and race, number of days used continued to
be a significant predictor of week 8 GAD-7 (β=-0.06 (SE=0.02),p=0.001). There was no
indication of moderation by comorbid depression and anxiety (F(1, 354)=2.96, p=0.09),
medication (F(1, 355)=0.14, p=0.70), age (F(1, 355)=2.70, p=0.10), gender (F(1, 355)=0.57,
p=0.45), or race(F(2, 354)=0.28, p=0.76).
DISCUSSION
Overall, we found that participants showed substantial and clinically meaningful
reductions in depression and anxiety across all levels of symptom severity. This is consistent
with other findings in the literature, which have consistently found that people with high levels
of depression and anxiety symptom severity can benefit from DMHTs (Weisel, et al., 2018).
Indeed, we see that the magnitude of symptom reduction by severity category was clearly
ordinal. People who began treatment in the severe range of depression and anxiety showed the
greatest reductions in symptom severity, while those in the mild range showed the smallest
changes. There are a number of potential reasons for the greater reduction in symptom scores
with greater levels of baseline severity. There is simply more room for improvement with higher
symptom scores. In addition, artifacts such as regression toward the mean may contribute to
these differences. However, people with comorbid depression and anxiety, another marker of
severity, also showed larger reductions in symptoms relative to those with depression or anxiety
only. Thus, there is no evidence that depression and anxiety symptom severity alone should be
used to determine who should or should not receive a DMHT.
There was no evidence that baseline symptom severity or comorbidity affected app use.
This is generally consistent with the literature, although there are some isolated reports that
people with greater levels of symptom severity are both less adherent (Calear, Christensen,
Mackinnon, & Griffiths, 2013; Farrer, Griffiths, Christensen, Mackinnon, & Batterham, 2014) as
well as more adherent to digital interventions (Fuhr, et al., 2018). While greater app use was
significantly related to greater improvement for both depression and anxiety outcomes, the
magnitude of these effects was small. This is a fairly consistent finding in the research literature
(Alfonsson, Olsson, & Hursti, 2016; Calear, et al., 2013; Enrique, Palacios, Ryan, & Richards,
2019; Fuhr, et al., 2018) and likely reflects the complex relationship between use and outcome.
While engagement with the digital intervention does increase the likelihood of improvement, it
may also be that as people improve, the digital tools have less value to the individual and their
use drops (Mohr, et al., 2013). It is also likely that app use metrics alone do not adequately
capture a user’s engagement with an DMHT tool. For example, an individual who is cognitively
and emotionally engaged while using a tool may obtain more benefit, compared to someone who
uses the tool in a rote and more disengaged fashion (Doherty & Doherty, 2018). Thus, simple
correlations of use with outcome likely miss the complexity and nuance of the relationship
between engagement with DMHTs and the impact on clinical symptoms.
 We also investigated the possibility that patient characteristics might influence outcomes
and app usage. We found a non-linear relationship between education and outcome, such that
people with less education (had not completed a bachelor’s degree) showed the greatest
reductions in symptoms, followed by those with advanced degrees, and people with a bachelor’s
degree showing the least improvement. While there have been isolated studies showing a
relationship between education and outcomes for digital interventions, meta-analyses have found
no consistent findings (Karyotaki, et al., 2018). With respect to app use, being older, female,
White, and taking an antidepressant medication were associated with higher use. While
significant, these effect sizes were small. The general literature has not shown consistent
relationships between DMHT use and either demographics or medication use (Beatty & Binnion,
2016). We note that a majority of participants were receiving pharmacotherapy and a sizable
portion were receiving psychotherapy. These treatments were not related outcomes. Receipt of
psychotherapy was unrelated to app usage, however receipt of pharmacotherapy was associated
with increased number of app sessions and days of app use. This suggests that receipt of
pharmacotherapy could be related to an increased capacity to engage in a DMHT. Nonetheless,
the inconsistency of our findings relating to participant characteristics and DMHT use with the
literature and the likelihood of chance findings resulting from the number of analyses, suggest
that these findings should be viewed with caution.
As DMHTs begin to be implemented in healthcare systems, clinical pathways must be
defined that determine where DMHTs fit into mental healthcare and who should be referred to a
DMHT, as opposed to other psychological treatment options. Mental health screening using
symptoms severity measures such as the PHQ-9 has become a standard of care in primary care
(Siu, et al., 2016). However, these findings suggest that symptom severity is not useful in
determining who should be referred to a DMHT. This supports the recent Banbury consensus
statement that all people with common mental health conditions such as depression and anxiety
should be offered DMHT as a treatment option (Mohr, et al., 2021). It is nonetheless possible
that there are factors that can be used to determine which patients should be referred to DMHTs
and which should be referred directly to other psychological treatments. Whether or not an
individual has connectivity and access to the device through which it is delivered clearly is such
a factor. Further research is needed to determine whether there are other factors that may be
used in triaging patients to DMHT and psychological services, such as history of symptoms,
diagnoses, and treatments, or preference. Research is also needed to understand how DMHTs can
be integrated and implemented in clinical settings. For example, several studies have suggested
that stepped care models, which offer DMHTs as a first line treatment, stepping up to
psychotherapy for those who do not respond, can result in outcomes that are equivalent to
psychotherapy but at a much lower cost (Haug, et al., 2015; Mohr, Lattie, et al., 2019;
Nordgreen, et al., 2016).
There are a number of limitations that should be considered in interpreting our results.
First, while we have combined the findings from three trials conducted in different contexts, they
all used the same DMHT. While these findings are consistent with the findings from trials using
other DMHTs (Weisel, et al., 2018), generalizing these findings should be done cautiously,
carefully considering the features of the DMHT, population, and setting. While our findings
showed no significant effects of baseline symptom severity on outcomes, and significant
reductions in both depression and anxiety were seen across all symptom severity categories, it is
possible that other markers of severity, such as history of illness or history of being refractory to
treatment, could impact effectiveness. We emphasize that these analyses only examined coached
interventions; thus, these findings should not be generalized to fully automated, unguided
interventions. Finally, our findings are based entirely on self-report. It is possible that outcomes
measured in other ways, such as through standardized interviews, could lead to different results.
In summary, this analysis of coached IntelliCare across three trials found no evidence that
depression and anxiety symptom severity affected either improvement in symptoms of
depression and anxiety, or adherence to the apps. The significant improvement in symptoms was
seen at all levels of baseline symptoms severity, including mild, moderate, moderately severe,
and severe symptom levels. This suggests that symptom severity alone should not be a
consideration in determining who should be referred to DMHTs.
FUNDING
This work was funded by National Institutes of Health grants P50 MH119029; R44
MH114725; K01 DK116925; K08 MH112878
REFERENCES
Alfonsson, S., Olsson, E., & Hursti, T. (2016). Motivation and Treatment Credibility Predicts
Dropout, Treatment Adherence, and Clinical Outcomes in an Internet-Based Cognitive
Behavioral Relaxation Program: A Randomized Controlled Trial. J Med Internet Res, 18,
e52.
Andersson, G., Topooco, N., Havik, O., & Nordgreen, T. (2016). Internet-supported versus face-
to-face cognitive behavior therapy for depression. Expert Rev Neurother, 16, 55-60.
Andrews, G., Basu, A., Cuijpers, P., Craske, M. G., McEvoy, P., English, C. L., & Newby, J. M.
(2018). Computer therapy for the anxiety and depression disorders is effective,
acceptable and practical health care: An updated meta-analysis. Journal of Anxiety
Disorders, 55, 70-78.
Beatty, L., & Binnion, C. (2016). A Systematic Review of Predictors of, and Reasons for,
Adherence to Online Psychological Interventions. Int J Behav Med, 23, 776-794.
Calear, A. L., Christensen, H., Mackinnon, A., & Griffiths, K. M. (2013). Adherence to the
MoodGYM program: outcomes and predictors for an adolescent school-based population.
Journal of Affective Disorders, 147, 338-344.
Doherty, K., & Doherty, G. (2018). Engagement in HCI: Conception, Theory and Measurement.
ACM Computing Surveys, 51, Article 99.
Enrique, A., Palacios, J. E., Ryan, H., & Richards, D. (2019). Exploring the Relationship
Between Usage and Outcomes of an Internet-Based Intervention for Individuals With
Depressive Symptoms: Secondary Analysis of Data From a Randomized Controlled
Trial. J Med Internet Res, 21, e12775.
Farrer, L. M., Griffiths, K. M., Christensen, H., Mackinnon, A. J., & Batterham, P. J. (2014).
Predictors of adherence and outcome in Internet-based cogntive behavior therapy
delivered in a telephone counseling setting. Cognitive Therapy and Research, 38, 358-
367.
Fuhr, K., Schroder, J., Berger, T., Moritz, S., Meyer, B., Lutz, W., Hohagen, F., Hautzinger, M.,
& Klein, J. P. (2018). The association between adherence and outcome in an Internet
intervention for depression. Journal of Affective Disorders, 229, 443-449.
Gilbody, S., Brabyn, S., Lovell, K., Kessler, D., Devlin, T., Smith, L., Araya, R., Barkham, M.,
Bower, P., Cooper, C., Knowles, S., Littlewood, E., Richards, D. A., Tallon, D., White,
D., Worthy, G., & collaborative, R. (2017). Telephone-supported computerised
cognitive-behavioural therapy: REEACT-2 large-scale pragmatic randomised controlled
trial. British Journal of Psychiatry, 210, 362-367.
Gilbody, S., Littlewood, E., Hewitt, C., Brierley, G., Tharmanathan, P., Araya, R., Barkham, M.,
Bower, P., Cooper, C., Gask, L., Kessler, D., Lester, H., Lovell, K., Parry, G., Richards,
D. A., Andersen, P., Brabyn, S., Knowles, S., Shepherd, C., Tallon, D., White, D., &
Team, R. (2015). Computerised cognitive behaviour therapy (cCBT) as treatment for
depression in primary care (REEACT trial): large scale pragmatic randomised controlled
trial. BMJ, 351, h5627.
Graham, A. K., Greene, C. J., Kwasny, M. J., Kaiser, S. M., Lieponis, P., Powell, T., & Mohr, D.
C. (2020). Coached Mobile App Platform for the Treatment of Depression and Anxiety
Among Primary Care Patients: A Randomized Clinical Trial. JAMA Psychiatry, 77, 906-
914.
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research
electronic data capture (REDCap)--a metadata-driven methodology and workflow
 process for providing translational research informatics support. J Biomed Inform, 42,
377-381.
Haug, T., Nordgreen, T., Ost, L. G., Kvale, G., Tangen, T., Andersson, G., Carlbring, P.,
Heiervang, E. R., & Havik, O. E. (2015). Stepped care versus face-to-face cognitive
behavior therapy for panic disorder and social anxiety disorder: Predictors and
moderators of outcome. Behav Res Ther, 71, 76-89.
Karyotaki, E., Ebert, D. D., Donkin, L., Riper, H., Twisk, J., Burger, S., Rozental, A., Lange, A.,
Williams, A. D., Zarski, A. C., Geraedts, A., van Straten, A., Kleiboer, A., Meyer, B.,
Unlu Ince, B. B., Buntrock, C., Lehr, D., Snoek, F. J., Andrews, G., Andersson, G., Choi,
I., Ruwaard, J., Klein, J. P., Newby, J. M., Schroder, J., Laferton, J. A. C., Van Bastelaar,
K., Imamura, K., Vernmark, K., Boss, L., Sheeber, L. B., Kivi, M., Berking, M., Titov,
N., Carlbring, P., Johansson, R., Kenter, R., Perini, S., Moritz, S., Nobis, S., Berger, T.,
Kaldo, V., Forsell, Y., Lindefors, N., Kraepelien, M., Bjorkelund, C., Kawakami, N., &
Cuijpers, P. (2018). Do guided internet-based interventions result in clinically relevant
changes for patients with depression? An individual participant data meta-analysis. Clin
Psychol Rev, 63, 80-92.
Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., Mackinnon, A.,
Meyer, B., Botella, C., Littlewood, E., Andersson, G., Christensen, H., Klein, J. P.,
Schroder, J., Breton-Lopez, J., Scheider, J., Griffiths, K., Farrer, L., Huibers, M. J.,
Phillips, R., Gilbody, S., Moritz, S., Berger, T., Pop, V., Spek, V., & Cuijpers, P. (2017).
Efficacy of Self-guided Internet-Based Cognitive Behavioral Therapy in the Treatment of
Depressive Symptoms: A Meta-analysis of Individual Participant Data. JAMA
Psychiatry, 74, 351-359.
Kazdin, A. E., & Blase, S. L. (2011). Rebooting Psychotherapy Research and Practice to Reduce
the Burden of Mental Illness. Perspectives on Psychological Science, 6, 21-37.
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: validity of a brief depression
severity measure. Journal of General Internal Medicine, 16, 606-613.
Kroenke, K., Spitzer, R. L., Williams, J. B., & Lowe, B. (2010). The Patient Health
Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review.
General Hospital Psychiatry, 32, 345-359.
Linardon, J., Cuijpers, P., Carlbring, P., Messer, M., & Fuller-Tyszkiewicz, M. (2019). The
efficacy of app-supported smartphone interventions for mental health problems: a meta-
analysis of randomized controlled trials. World Psychiatry, 18, 325-336.
Mohr, D. C., Azocar, F., Bertagnolli, A., Choudhury, T., Chrisp, P., Frank, R., Harbin, H.,
Histon, T., Kaysen, D., Nebeker, C., Richards, D., Schueller, S. M., Titov, N., Torous, J.,
Arean, P. A., & Banbury Forum on Digital Mental, H. (2021). Banbury Forum Consensus
Statement on the Path Forward for Digital Mental Health Treatment. Psychiatr Serv,
appips202000561.
Mohr, D. C., Cuijpers, P., & Lehman, K. (2011). Supportive Accountability: A Model for
Providing Human Support to Enhance Adherence to eHealth Interventions. J Med
Internet Res, 13, e30.
Mohr, D. C., Duffecy, J., Ho, J., Kwasny, M., Cai, X., Burns, M. N., & Begale, M. (2013). A
Randomized Controlled Trial Evaluating a Manualized TeleCoaching Protocol for
Improving Adherence to a Web-Based Intervention for the Treatment of Depression.
PLoS One, 8, e70086.
Mohr, D. C., Lattie, E. G., Tomasino, K. N., Kwasny, M. J., Kaiser, S. M., Gray, E. L., Alam, N.,
Jordan, N., & Schueller, S. M. (2019). A randomized noninferiority trial evaluating
remotely-delivered stepped care for depression using internet cognitive behavioral
therapy (CBT) and telephone CBT. Behav Res Ther, 123, 103485.
Mohr, D. C., Schueller, S. M., Tomasino, K. N., Kaiser, S. M., Alam, N., Karr, C., Vergara, J. L.,
Gray, E. L., Kwasny, M. J., & Lattie, E. G. (2019). Comparison of the Effects of
Coaching and Receipt of App Recommendations on Depression, Anxiety, and
Engagement in the IntelliCare Platform: Factorial Randomized Controlled Trial. J Med
Internet Res, 21, e13609.
Mohr, D. C., Tomasino, K. N., Lattie, E. G., Palac, H. L., Kwasny, M. J., Weingardt, K., Karr, C.
J., Kaiser, S. M., Rossom, R. C., Bardsley, L. R., Caccamo, L., Stiles-Shields, C., &
Schueller, S. M. (2017). IntelliCare: An Eclectic, Skills-Based App Suite for the
Treatment of Depression and Anxiety. J Med Internet Res, 19, e10.
National Advisory Mental Health Council, Arean, P. A., Leshner, A., Barch, D. M., Ben-Zeev,
D., Brent, D. A., Carpenter, R. L., Casey, B. J., Davison, J. W., Druss, B. J., Hogan, M.
F., Linehan, M. M., Mohr, D. C., Nebeker, C., Picard, R., Poldrack, R., Robinson, G. E.,
Robinson Beale, R. J., Rotheram, M. J., & Singh, K. (2017). Opportunities and
Challenges of Developing Information Technologies on Behavioral and Social Science
Clinical Research. In. National Institute of Mental Health: National Institute of Mental
Health.
Nicholas, J., Knapp, A. A., Vergara, J. L., Graham, A. K., Gray, E. L., Lattie, E. G., Kwasny, M.
J., & Mohr, D. C. (2021). An Exploratory Brief Head-To-Head Non-Inferiority
Comparison of an Internet-Based and a Telephone-Delivered CBT Intervention for
Adults with Depression. J Affect Disord, 281, 673-677.
Nordgreen, T., Haug, T., Ost, L. G., Andersson, G., Carlbring, P., Kvale, G., Tangen, T.,
Heiervang, E., & Havik, O. E. (2016). Stepped Care Versus Direct Face-to-Face
Cognitive Behavior Therapy for Social Anxiety Disorder and Panic Disorder: A
Randomized Effectiveness Trial. Behav Ther, 47, 166-183.
Noth, K. N., Bardsley, L., Lattie, E. G., & Mohr, D. C. (2018). IntelliCare Study Coaching
Manual. In. Chicago, IL: Northwestern University.
Pham, Q., Graham, G., Carrion, C., Morita, P. P., Seto, E., Stinson, J. N., & Cafazzo, J. A.
(2019). A Library of Analytic Indicators to Evaluate Effective Engagement with
Consumer mHealth Apps for Chronic Conditions: Scoping Review. JMIR Mhealth
Uhealth, 7, e11941.
Schueller, S. M., Tomasino, K. N., & Mohr, D. C. (2016). Integrating Human Support into
Behavioral Intervention Technologies: The Efficiency Model of Support. Clinical
Psychology: Science and Practice, 27-45.
Siu, A. L., Force, U. S. P. S. T., Bibbins-Domingo, K., Grossman, D. C., Baumann, L. C.,
Davidson, K. W., Ebell, M., Garcia, F. A., Gillman, M., Herzstein, J., Kemper, A. R.,
Krist, A. H., Kurth, A. E., Owens, D. K., Phillips, W. R., Phipps, M. G., & Pignone, M.
P. (2016). Screening for Depression in Adults: US Preventive Services Task Force
Recommendation Statement. JAMA, 315, 380-387.
Spitzer, R. L., Kroenke, K., Williams, J. B., & Lowe, B. (2006). A brief measure for assessing
generalized anxiety disorder: the GAD-7. Archives of Internal Medicine, 166, 1092-1097.
Substance Abuse and Mental Health Services Administration (SAMHSA). (2017). Results from
the 2017 National Survey on Drug Use and Health. In (Vol. 2019). Washington, DC:
Substance Abuse and Mental Health Services Administration (SAMHSA).
Weisel, K. K., Fuhrmann, L. M., Berking, M., Baumeister, H., Cuijpers, P., & Ebert, D. D.
(2019). Standalone smartphone apps for mental health-a systematic review and meta-
analysis. NPJ Digit Med, 2, 118.
Weisel, K. K., Lehr, D., Heber, E., Zarski, A. C., Berking, M., Riper, H., & Ebert, D. D. (2018).
Severely Burdened Individuals Do Not Need to Be Excluded From Internet-Based and
Mobile-Based Stress Management: Effect Modifiers of Treatment Outcomes From Three
Randomized Controlled Trials. J Med Internet Res, 20, e211.
Wells, K., Miranda, J., Bauer, M., Bruce, M., Durham, M., Escobar, J., Ford, D., Gonzalez, J.,
Hoagwood, K., Horwitz, S., Lawson, W., Lewis, L., McGuire, T., Pincus, H., Scheffler,
R., Smith, W., & Unutzer, J. (2002). Overcoming barriers to reducing the burden of
affective disorders. Biological Psychiatry, 52, 655.
Whooley, M. A., & Simon, G. E. (2000). Managing depression in medical outpatients. New
England Journal of Medicine, 343, 1942-1950.
Whooley, M. A., Stone, B., & Soghikian, K. (2000). Randomized trial of case-finding for
depression in elderly primary care patients. Journal of General Internal Medicine, 15,
293-300.
Table 1: Participant Characteristics and App Use Across the Studies
Characteristic Study 1 Study 2 Study 3 p-value
N 105 150 146
Age 38.9 ± 37.3 ± 12.2 42.3 ± 0.005
14.1 13.8
Gender (male) 25 (24%) 29 (19%) 25 (17%) 0.45
Race <0.001
White 88 (84%) 114 (76%) 89 (61%)
Black or African- 6 (6%) 8 (5%) 47 (32%)
American 11 (10%) 28 (19%) 10 (7%)
Other
Ethnicity (Latinx) 5 (5%) 17 (12%) 10 (7%) 0.12
Marital status 0.08
Single 35 (33%) 61 (41%) 42 (29%)
Married, Partnered 53 (50%) 72 (48%) 72 (49%)
Separated, Widowed 17 (16%) 17 (11%) 32 (22%)
Education 0.046
Less than Bachelors 41 (39%) 48 (32%) 72 (49%)
Bachelors 37 (35%) 63 (42%) 43 (29%)
Advanced degree 27 (26%) 39 (26%) 31 (21%)
Employed 76 (72%) 102 (68%) 107 (73%) 0.57
Household income $63,000 $60,000 $42,000 0.02
median (25th, 75th) (30,000, (35,000, (29,000,
100,000) 98,000) 80,000)
Depression (PHQ-9≥10) 78 (81%) 118 (79%) 122 (84%) 0.56
Anxiety (GAD-7≥8) 77 (80%) 129 (86%) 131 (90%) 0.11
Baseline PHQ-9 12.5 ± 4.3 12.9 ±4.6 11.7 ± 6.1 0.13
Baseline GAD-7 10.9 ± 4.5 12.1 ± 4.3 10.2 ± 5.2 0.003
Comorbidity (PHQ-9≥10 107 (73%) 59 (56%) 97 (65%) 0.018
& GAD≥8)
Current psychotherapy 23 (22%) 62 (41%) 17 (12%) <0.001
Current medication 66 (64%) 72 (48%) 82 (56%) 0.04
Number of app sessions 257 (144, 122 (69, 95 (51, <0.001
385) 204) 143)
Days of app use 48 (33, 38 (29, 45) 27 (17, <0.001
53) 35)
Table 2: Least square means and standard errors of Baseline and Week 8 PHQ-9 and GAD-7 by
depression or anxiety status at baseline (all changes p<0.001)
PHQ-9 GAD-7
Severity N Baseline Week 8 Change N Baseline Week 8 Change
Mild (0-9) 120 6.7 (0.2) 4.1 (0.5) -2.3 (0.5) 165 6.4 (0.1) 4.4 (0.3) -2.1 (0.4)
Moderate 163 11.9 (0.1) 6.6 (0.4) -5.2 (0.4) 136 11.9 (0.1) 6.1 (0.4) -5.8 (0.4)
(10-14)
Moderatel 76 16.7 (0.2) 8.6 (0.5) -7.9 (0.6) 83 16.6 (0.2) 9.3 (0.5) -7.3 (0.5)
y Severe
(15-19)
Severe 42 22.1 (0.3) 10.7 (0.7) -11.1 (0.8) 17 20.7 (0.4) 10.4 (1.0) -10.3
(20+) (1.1)

Table 3: General Linear mixed models associations between participant characteristics and app
use, and outcome [regression slopes (Standard errors)]
Participant Characteristics Change in p-value Change in p-value
PHQ-9 GAD-7
Age 0.01 (0.02) 0.730 0.02 (0.02) 0.24
Gender (female) 0.14 (0.71) 0.846 0.17 (0.63) 0.79
Race 0.10 0.06
White -- --
Black or African American -1.86 (0.87) -1.54 (0.73)
Other -0.17 (0.82) 0.63 (0.75)
Ethnicity (Latinx) 0.05 (1.0) 0.958 1.22 (0.89) 0.17
Marital status 0.249 0.97
Single -- --
Married, Partnered 0.38 (0.62) 0.13 (0.56)
Separated, Widowed -0.95 (0.83) 0.07 (0.74)
Education <0.001 0.04
No or some college (no degree) -- --
Bachelor’s degree 2.44 (0.64) 1.49 (0.58)
Advanced degree 1.31 (0.70) 0.83 (0.64)
Employed 0.11 (0.62) 0.86 -0.35 (0.55) 0.53
Household income ($1,000) 0.006 (0.004) 0.153 0.003 (0.004) 0.48
median (25th, 75th)
Baseline depression (PHQ-9) -3.87 (0.66) <0.001 0.07 (0.63) 0.91
Baseline anxiety (GAD-7) 0.15 (0.79) 0.850 -3.54 (0.70) <0.001
Comorbidity (PHQ-9≥10 & -2.69 (0.57) <0.001 -1.81 (0.52) 0.001
GAD≥8)
Current psychotherapy -1.61 (0.62) 0.010 -0.85 (0.58) 0.15
Current medication 0.24 (0.56) 0.665 0.34 (0.50) 0.503
App Use
Number of app sessions -0.01 (0.002) 0.002 -0.005 (0.002) 0.005
Days used app -0.07 (0.02) <0.001 -0.05 (0.02) 0.004
Table 4. Multinomial regression parameters and standard errors for baseline participant
characteristics adjusted for study (predicting higher quartile of days used or higher quartile of
total launches)
Characteristic Number of p- Days used p-
App Sessions value value
Age 0.02 (0.01) 0.01 0.01 (0.01) 0.19
Gender (male) -0.64 (0.23) 0.005 -0.53 (0.23) 0.02
Race 0.01 0.04
White -- --
Black or African American -0.50 (0.25) -0.46 (0.25)
Other -0.72 (0.28) -0.58 (0.28)
Ethnicity (Latinx) 0.03 (0.33) 0.93 -0.12 (0.33) 0.21
Marital status 0.81 0.75
Single -- --
Married, Partnered -0.07 (0.20) 0.02 (0.20)
Separated, Widowed 0.10 (0.27) 0.19 (0.27)
Education 0.65 0.93
Less than Bachelors -- --
Bachelors 0.18 (0.21) -0.08 (0.21)
Advanced degree 0.01 (0.23) -0.05 (0.23)
Employed -0.07 (0.20) 0.71 -0.11 (0.20) 0.60
Household income ($1,000) -0.002 (0.001) 0.29 -0.002 0.12
median (25th, 7th) (0.001)
Baseline PHQ-9 -0.01 (0.02) 0.44 0.01 (0.02) 0.73
Baseline GAD-7 -0.01 (0.02) 0.56 0.01 (0.02) 0.56
Depression -0.02 (0.23) 0.94 0.17 (0.23) 0.45
Anxiety 0.01 (0.26) 0.98 0.05 (0.26) 0.83
Comorbidity (PHQ-9≥10 & 0.29 (0.19) 0.12 0.14 (0.19) 0.47
GAD≥8)
Current therapy 0.36 (0.21) 0.08 0.31 (0.21) 0.13
Current medication 0.46 (0.18) 0.01 0.56 (0.18) 0.002
Figure 1.
Figure 2.