Behaviour Research and Therapy 59 (2014) 1e11

Contents lists available at ScienceDirect

Behaviour Research and Therapy

journal homepage: www.elsevier.com/locate/brat

Effectiveness and cost-effectiveness of individually tailored

Internet-delivered cognitive behavior therapy for anxiety disorders
in a primary care population: A randomized controlled trial
Lise Bergman Nordgren a, *, Erik Hedman b, c, Julie Etienne a, Jessica Bodin a,
Åsa Kadowaki d, Stina Eriksson e, Emelie Lindkvist e, Gerhard Andersson a, b,
Per Carlbring f
a
Department of Behavioural Sciences and Learning, Linko €ping University, SE-581 83 Linko €ping, Sweden
b
Department of Clinical Neuroscience, Psychiatry Section, Karolinska Institutet, 171 77 Stockholm, Sweden
c
Department of Clinical Neuroscience, Osher Center for Integrative Medicine, Karolinska Institutet, 171 77 Stockholm, Sweden
d €
County Council of Osterg €tland, 581 91 Linko
o €ping, Sweden
e
Department of Psychology, Umeå University, 901 87 Umeå, Sweden
f
Department of Psychology, Stockholm University, 106 91 Stockholm, Sweden

a r t i c l e i n f o a b s t r a c t

Article history: A significant proportion of the general population suffers from anxiety disorders, often with comorbid
Received 25 June 2013 psychiatric conditions. Internet-delivered cognitive behavior therapy (ICBT) has been found to be a
Received in revised form potent treatment for patients with specific psychiatric conditions. The aim of this trial was to investigate
19 April 2014
the effectiveness and cost-effectiveness of ICBT when tailoring the treatment to address comorbidities
Accepted 21 May 2014
Available online 2 June 2014
and preferences for primary-care patients with a principal anxiety disorder. One hundred participants
were recruited through their primary-care contact and randomized to either treatment or an active
control group. The treatment consisted of 7e10 weekly individually assigned modules guided by online
Keywords:
Primary care
therapists. At post-treatment, 46% of the treatment group had achieved clinically significant improve-
Anxiety ment on the primary outcome measure (CORE-OM) and between-group effect sizes ranged from d ¼ 0.20
Depression to 0.86, with a mean effect of d ¼ 0.59. At one-year follow-up, within-group effect sizes varied between
Comorbidity d ¼ 0.53 to 1.00. Cost analysis showed significant reduction of total costs for the ICBT group, the results
Internet-delivered cognitive behavior were maintained at one-year follow-up and the incremental cost-effectiveness ratio favored ICBT
therapy compared to control group. Individually tailored ICBT is an effective and cost-effective treatment for
Cognitive behavior therapy primary-care patients with anxiety disorders with or without comorbidities.
Cost-effectiveness
Trial Registration: Clinicaltrials.gov: NCT01390168.
© 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

Background
In Europe, anxiety disorders have an estimated 12-month
prevalence of nearly 14 percent (Alonso et al., 2004b; Wittchen
et al., 2011). Comorbidity across different anxiety and mood dis-
orders is high in the general population (Alonso et al., 2004a) and
the occurrence of an additional Axis-I diagnosis is even higher in
individuals seeking treatment for their anxiety in primary care
centers for treatment of stress, anxiety and related disorders
(Brown, Campbell, Lehman, Grisham, & Mancill, 2001). The most
* Corresponding author. Tel.: þ46 13 28 20 37.
E-mail address: lise.bergman.nordgren@liu.se (L.B. Nordgren).
common comorbid disorder for individuals with one anxiety dis-
order is a mood disorder or another anxiety disorder. All of these
conditions are associated with reduced quality of life (Mendlowicz
& Stein, 2000; Olatunji, Cisler, & Tolin, 2007), an increased risk of
developing somatic disorders (Denollet, Maas, Knottnerus, Keyzer,
& Pop, 2009), and high societal costs (Smit, Cuijpers, et al., 2006).
Despite these high numbers of individuals suffering from anx-
iety disorders, only a small proportion receive adequate treatment
(Weisberg, Dyck, Culpepper, & Keller, 2007; Wittchen & Jacobi,
2005). Primary care facilities often lack resources to meet the
need for psychological treatments in general and evidence-based
treatments in particular. One way of increasing access to
evidence-based psychological treatments in primary care could be
to use guided Internet-delivered cognitive behavior therapy (ICBT)

http://dx.doi.org/10.1016/j.brat.2014.05.007
0005-7967/© 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).
2 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11
(Andersson, 2009; Carlbring & Andersson, 2006). Guided ICBT has
been shown to be effective for a variety of anxiety and mood dis-
orders (Andrews, Cuijpers, Craske, McEvoy, & Titov, 2010; Spek
et al., 2007), and has also been found to be cost-effective when
compared to no treatment or to conventional cognitive behavior
therapy (Hedman, Andersson, Ljo  tsson, Andersson, Rück, Mo
€rtberg,
et al., 2011; Hedman, Andersson, Ljo  tsson, Andersson, Rück, &
Lindefors, 2011; Hedman, Ljo  tsson, & Lindefors, 2012). There is
some evidence for the long-term effects of ICBT for anxiety disor-
ders (Carlbring et al., 2011; Carlbring, Nordgren, Furmark, &
Andersson, 2009; Hedman, Furmark, et al., 2011; Paxling et al.,
2011), and the evidence base for the clinical effectiveness of ICBT
is growing, showing promising results (Bell, Colhoun, Carter, &
Frampton, 2012; Hedman, Ljo  tsson, et al., 2013; Hoifodt, Strom,
Kolstrup, Eisemann, & Waterloo, 2011; Mewton, Wong, &
Andrews, 2012; Roy-Byrne et al., 2010; Ruwaard, Lange,
Schrieken, Dolan, & Emmelkamp, 2012). One potential problem
when disseminating guided ICBT in primary care is that many
randomized controlled trials (RCTs) on guided ICBT make use of
strict single diagnosis protocols and suffer from high exclusion
rates. One example is a study by Carlbring et al. (2005), where over
80% of the individuals applying for the study were excluded, the
lion's share because panic disorder was not their primary problem.
The impact of existing co-morbidity on these protocols is largely
unknown. In the field of guided ICBT, there is some evidence that
comorbidity is reduced following ICBT for social anxiety disorder
(Titov, Gibson, Andrews, & McEvoy, 2009). Overall, however, it is
reasonable to assume that patients with comorbidities present
with more deficits in their functioning and that treatments that
focus on only one aspect of the problem may run the risk of going
against patient preferences, which potentially can affect outcome.
In CBT, different approaches have been developed to deal with
comorbidity and high exclusion rates. One such approach is the
unified transdiagnostic treatment protocol (Barlow, Allen, &
Choate, 2004), in which common shared characteristics across
treatments for specific conditions are assumed to serve as active
ingredients across disorders. But still there is not much evidence
that a unified treatment approach is more effective than conven-
tional CBT (Craske, 2012).
There have been some promising results from the ICBT format
for the delivery of transdiagnostic treatment (Dear et al., 2011;
Titov, Andrews, Johnston, Robinson, & Spence, 2010; Titov et al.,
2011). Another way of dealing with comorbidities is by address-
ing them directly, using individually tailored treatments. When
tailoring the treatment both the patient and the therapist are
allowed to influence the protocol.
A similar tailored program used in this study has been shown to
be effective previously in a heterogeneous self-recruited sample,
both directly at post-treatment and at a long-term follow-up
(Carlbring et al., 2011). In another study, the same protocol
approach was tested for young adults and adults with panic
symptoms (Silfvernagel et al., 2012). Moreover, tailored ICBT was
found to be more effective for more severely depressed patients
compared to moderately depressed patients in a controlled trial
comparing standard ICBT versus tailored ICBT for depression
(Johansson et al., 2012). All of these previous studies were efficacy
trials and tailored ICBT has not been tested in any published
effectiveness study. The current study, designed to be clinically
representative (Shadish, Matt, Navarro, & Phillips, 2000), aims to
investigate whether this approach could be beneficial for a sample
recruited through their primary-care contact and to explore if it is a
cost-effective alternative. We expected, on the basis of previous
efficacy trials, that the treatment would be moderately effective,
both at post-treatment and at a one-year follow-up. We also ex-
pected that the treatment would be cost-effective.
Method
Ethics statement
The study was approved by the regional Ethical Board and has
been registered in clinicaltrials.gov (NCT01390168). Written
informed consent was collected from all participants by surface
post.
Study design
The present study was an effectiveness study examining the
effects of ten weeks of ICBT with scheduled e-mail guidance
compared against an active waiting list control group who had
access to health care and can be viewed as equivalent to care-as-
usual when comparing pre- to post-treatment and pre-to-one-
year follow-up results. For the full study design, we refer to the
published study protocol (Nordgren, Andersson, Kadowaki, &
Carlbring, 2012). We made two changes to the protocol in the
present study by not using ANCOVA for data analysis, but instead
using a linear mixed model because of the large amount of missing
data in the follow-up measurements (Gueorguieva & Krystal,
2004). Moreover, we managed to include only 100 participants,
instead of the 128 outlined in the protocol. This change was made
because of time limits of the project and funding. Inclusion took
place in 18 different primary care settings, in ten different cities,
between January 2010 and April 2011. A new power calculation
based on 100 participants revealed a power of 84%, given an alpha-
level of .05 and an effect size of d ¼ 0.6.
A total of 100 patients were recruited through their primary care
contact (e.g., physician) and were randomly assigned by an online
true random-number service independent of the investigators and
therapists to either immediate treatment or control. The control
group received treatment after ten weeks, leaving no control group
for the follow-up measurements.
Inclusion and exclusion criteria
To be included in the study, participants had to fulfill the DSM-
IV (American Psychiatric Association, 2000) criteria for any anxiety
disorder as a primary diagnosis. Hence, all participants had a clin-
ically significant impairment because of at least one anxiety dis-
order with or without comorbid problems. Participants had to be at
least 18 years old, have Internet access, not be participating in any
ongoing psychological treatment, and if taking any medication for
anxiety or depression, this had to be at an unchanged dosage for at
least 12 weeks pre-treatment. Participants with a suicidal risk,
defined as scoring >3 on item 9 of the Montgomery Åsberg
Depression Rating Scale (MADRS-S; Svanborg & Åsberg, 1994),
were further interviewed using the SAD PERSONS scale (Patterson,
Dohn, Bird, & Patterson, 1983) during a telephone-administered
diagnostic interview and were excluded if their risk of suicide
was confirmed (they then were referred back to their original pri-
mary care setting). Scoring above 30 on the MADRS-S or below 9 on
the Beck Anxiety Inventory (BAI; Beck, Epstein, Brown, & Steer,
1988) also served as reasons for exclusion.
Recruitment and participants
Participants were recruited from a clinical population, typically
by their general practitioner or nurse, when seeking help at their
primary care setting. Prescribing clinicians were informed about
the intervention and the inclusion criteria in order to decide for
whom this might be adequate, regardless if the patient presented
themselves with anxiety as their primary complaint. Based on the
 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 3

Fig. 1. Flowchart of study participants, point of random assignment, and dropouts at each stage of a study of individually-tailored, internet-delivered cognitive behavior therapy for
anxiety disorders.

clinical judgment of the clinician eligible participants were given an
information folder containing brief information about the trial and
a link to the study website. On the website they found further in-
formation about the treatment, an informed consent form, a link to
the questionnaires serving as screening, and the application form
for participation.
Initial selection took place through computerized screening
consisting of the Clinical Outcomes in Routine Eval-
uationdOutcome Measure (CORE-OM; Barkham et al., 2001), the
self-rated version of the Montgomery Åsberg Depression Rating
Scale (MADRS-S; Svanborg & Åsberg, 1994), the Beck Anxiety In-
ventory (BAI; Beck et al., 1988), the Quality of Life Inventory (QOLI;
Frisch, Cornell, Villanueva, & Retzlaff, 1992), and ten additional
questions regarding demographics and current and past treat-
ments. These results were used as the pre-treatment assessment.
The outcome measures all show good psychometric properties
even when administered online (Carlbring et al., 2007; Hedman
et al., 2010).
Participants who fulfilled the initial inclusion criteria according
to the computerized screening were contacted to take part in a
4 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11

telephone interview where they were interviewed using the Table 1

Structured Clinical Interview for DSM-IVdAxis I Disorders (SCID-I; Demographic characteristics of participants at baseline.

First, Gibbon, Spitzer, & Williams, 1997). Seven MSc clinical psy- Treatment Control
chology students in the latter part of their clinical training con- (n ¼ 50) (n ¼ 50)
ducted the interviews. All received training, including role-playing Gender Female 33 30
and coding of videotapes, and supervision in using the interview. Male 17 20
Telephone administration of the SCID-I interview has been found to Age Mean (SD) 35 (13) 36 (12)
Min-max 20e68 19e60
be equally valid as a face-to-face administration of the SCID-I
Marital status Married/living together 25 38
(Rohde, Lewinsohn, & Seeley, 1997). A psychiatrist and a licensed In a relationship 3 3
clinical psychologist together with the interviewers assessed the Single 20 9
SCID protocols, and together they tailored the treatment according Other 2 0
Highest Secondary school 13 10
to each participant's unique set of problems and, to some extent,
education
preferences. Both the psychiatrist and clinical psychologist had level Vocational school (compl) 4 5
experience of working in primary care facilities, and the assess- College/university 13 14
ments and judgments were aimed to mirror how this procedure (not compl.)
could be made in usual care. In summary, participants were first College/university (compl) 20 21
Employment Employed full- or part-time 21 26
judged by their general practitioner to suffer from anxiety and after
status
online screening were diagnosed thoroughly by the research team Registered sick 4 8
using SCID-I. Unemployed 6 1
Of the 152 individuals who initially applied to participate in the Student 16 15
study, 100 were included after a telephone-administrated diag- Other 3 0
Psychotherapy None 17 (34%) 19 (38%)
nostic interview and randomized by an online true random- Earlier 33 (66%) 31 (62%)
number service independent of the investigators and therapists. Medication None 21 (42%) 22 (44%)
The reasons for exclusion are specified in the CONSORT flowchart Earlier 15 (30%) 16 (32%)
(Fig. 1). Ongoing 14 (28%) 12 (24%)
Primary Generalized 4 (8%) 6 (12%)
The mean age was 35.4 years, and 63 percent were female (63/
diagnosis anxiety disorder
100). Thirty six percent (36/100) reported their marital status as Social phobia 16 (32%) 16 (32%)
married or living together, with children in the household. Sixty Panic Disorder þ/ 16 (32%) 15 (30%)
four per cent had a history of previous psychological treatment, and agoraphobia 4 (8%) 4 (8%)
26 per cent (26/100) were using antianxiety or antidepressant Agoraphobia
Anxiety NOS 10 (20%) 9 (18%)
medication, while 31 percent reported a history of medical treat- Comorbidity Any comorbidity 30 (60%) 28 (56%)
ment for their condition. Demographic data on the included par- Mood disorder 28 (56%) 15 (30%)
ticipants are presented in detail in Table 1. Generalized anxiety 2 9
All included participants had an anxiety disorder as their prin- disorder
Social Phobia 6 7
cipal diagnosis. A majority (58/100) had one (n ¼ 39) or more co-
Panic disorder þ/ agoraphobia 1
morbid Axis-I disorder. For 43 participants, a comorbid mood Agoraphobia 2 2
disorder (major depressive disorder or dysthymia) was present. Specific phobia 1
There were no difference between the treatment and control group Obsessive compulsive disorder 1
regarding patients presenting comorbidities (n ¼ 30 in the treat- Hypochondriasis 1

ment condition and n ¼ 28 in the control condition). Note: NOS, Not otherwise specified. Participants could have more than one co-
morbid condition.

Outcome measures
Primary outcome measure
The CORE-OM (Barkham et al., 2001) served as the primary
outcome measure. This is a 34-item scale covering four subscales of
generic psychological well-being (Barkham et al., 2001). The sub-
scales are subjective well-being, symptoms (anxiety, depression,
physical problems, trauma), functioning (general functioning, close
relationships, social relationships), and risk (to self and others).
Items are scored from 0 to 4 and relate to the preceding week.
CORE-OM clinical scores can range from 0 to 40, with higher scores
indicating greater severity. Internal consistency has been reported
as a ¼ .94 (Barkham et al., 2001), with a cut-off score between a
clinical and non-clinical population of 12 for men and 12.6 for
women on the composite scale (Elfstrom et al., 2012).
high (Beck et al., 1988), with Cronbach's alpha ranging from .90 to
.94, and test-retest reliability at r ¼ .75.
The MADRS-S (Svanborg & Åsberg, 1994) measures depressive
symptoms over the past three days and consists of nine items rated
on a seven-grade scale. Total scores range between 0 and 54, with
scores above 35 (Snaith, Harrop, Newby, & Teale, 1986) considered
severe. Concurrent validity is shown to be good with a test-retest
reliability of r ¼ .83 (Mattila-Evenden, Svanborg, Gustavsson, &
Åsberg, 1996).
The QOLI (Frisch et al., 1992) assesses the subjective degree of
satisfaction in 16 different areas of life (e.g., health and work). The
scale shows high internal consistency (a ¼ .77e.89), and a test-
retest reliability from r ¼ .80 to .91 (Frisch et al., 1992; Lindner,
€ & Carlbring, in press).
Andersson, Ost,

Secondary outcome measures Cost assessment

As secondary outcome measures, the BAI (Beck et al., 1988), the
MADRS-S (Svanborg & Åsberg, 1994), and the QOLI (Frisch et al.,
1992) were used.
BAI measures anxiety symptoms and contains 21 short ques-
tions. Total scores range between 0 and 63, with a score of 30 or
higher indicating severe anxiety symptoms. Internal consistency is
We used the Trimbos and Institute of Medical Technology
Assessment Cost Questionnaire for Psychiatry (TIC-P; Hakkaart-Van
Roijen, Van Straten, & Donker, 2002) to obtain health economic cost
data. The TIC-P measures monthly direct medical costs, that is,
health care consumption (e.g., primary care visits) as well as
 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11
nondirect medical costs, that is, costs of other health-related ser-
vices not directly associated with health care (e.g., time spent in
self-help groups). Nonmedical costs, which are costs pertaining to
work and domestic productivity loss, were also measured with the
TIC-P.
The costs of ICBT and the control condition was represented
mainly by the costs of therapists, and we used the tariff of visits to
licensed clinical psychologists when estimating costs for both
conditions. This tariff was multiplied with time spent treating
patients.
Health related quality of life
The EuroQol (EQ-5D) was used to assess quality of life from a
health perspective (EuroQol-Group, 1990). The EQ-5D measures
five health domains related to quality of life: mobility, self-care,
usual activities, pain/discomfort, and anxiety/depression (Rabin &
Charro, 2001). The measure has demonstrated good psychomet-
rics properties, including acceptable test-retest reliability (intra-
class coefficient ¼ .82e.83) and acceptable convergent validity
(Ravens-Sieberer et al., 2010).
Intervention
The ICBT treatment were a form of text-based guided self-help
treatment with therapist support via the internet that consisted
of seven-to-ten treatment modules, or chapters, per participant
covering a period of ten weeks. The modules have been shown to be
effective previously when used on specific diagnosed samples (i.e.,
panic disorder; Carlbring et al., 2006), social phobia (Andersson
et al., 2006), generalized anxiety disorder (Paxling et al., 2011),
and depression (Vernmark et al., 2010) and were adapted to be able
to be presented together in the tailored format by being slightly
rewritten (e.g., words relating to specific conditions were
removed). Tailoring included both the order of treatment modules,
the amount of text presented to the participant and how many
modules to include in the ten week treatment protocol. The same
seven MSc students who conducted the SCID-interviews served as
Internet therapists. The therapists had access to supervision by a
licensed clinical psychologist both regarding the format and client-
specific questions throughout the study period. All communication
between participants and therapists was made through the
internet, using a messenger system within the treatment platform
similar to e-mail, and the main nature of the feedback was to
answer any questions regarding the module and homework
assignments.
Each module consisted of text and illustrations (9e39 A4 pages)
presenting a specific symptom and exercises and were to be
completed by three-to-eight essay questions to be worked through
during a period of one week. Some of the modules (i.e., relaxation
and mindfulness) had audio files attached to them for the partici-
pants to listen to. The homework questions were intended to
encourage learning and to help the Internet therapist assess
whether the participants had assimilated the material or not. Par-
ticipants were asked to answer the questions and provide work-
sheets and report on outcomes of different exercises to their
therapist once a week. Following submission of the report, they
were given individual feedback, most often within 24 h. When the
therapist received a homework assignment showing that the
participant has assimilated the material, the next module was
made accessible through an encrypted message exchange system.
The therapists were instructed not to spend more than 15 min per
participant per week in reading and communicating feedback.
Prescribed modules were available for download in PDF format,
and participants were advised to print out or to download the self-
help material to have the material readily available.
5
The first module (introduction) and the last (relapse prevention)
were fixed, which gave the possibility to tailor the treatment by
adding any of the following: cognitive restructuring (2 modules),
social anxiety (2 modules), generalized anxiety (3 modules), panic
disorder (2 modules), agoraphobia, behavioral activation (2 mod-
ules), applied relaxation, mindfulness, assertiveness, problem
solving, stress management, and sleep. All modules, except for
cognitive restructuring, applied relaxation, assertiveness, problem
solving and stress management, were disorder specific in that they
targeted core symptoms of each of the diagnoses.
Attention control group
Participants in the control condition were asked questions about
their well-being on a weekly basis by their therapist. Each partici-
pant e-mailed his or her answers to the therapist once a week in
order to track possible deterioration, but was generally not given
any specific feedback on the answers unless the therapist judged
the answer to warrant further action.
Statistical analysis
Treatment effects
In order to handle missing data and be able to use the full
sample, we used intention-to-treat (ITT) analysis. This method ac-
counts for missing data without assuming that the last measure-
ment was stable (as is assumed when using the last observation
carried forward; Salim, Mackinnon, Christensen, & Griffiths, 2008)
and is appropriate for designs with few time points. For pre- to
post- changes, the sample size was all 100 participants; for the
calculation of pre- to one-year follow-up effects, sample size was
set to 92 participants, excluding the patients withdrawing their
application during their time on the waiting list (n ¼ 8). For both
treatment effects and cost-effectiveness, SPSS 20.0 (SPSS Inc., Chi-
cago) was used, and for further calculations of costs and cost-
effectiveness, STATA IC/11.0 (Stata Corp) was used.
Treatment effects were calculated using a linear mixed model
with restricted maximum-likelihood estimation (REML) and an
unstructured (UN) covariance structure. Analyses of post-treatment
effects were made using the treatmentdcontrol group design. And
for the analyses for effects at one-year follow-up, we used the
treatment and control group as one. Measure points were indi-
vidual, using each participant's baseline as time one and then
assessing post and twelve month follow-up measures using this
time as starting point. This resulted in that data were collected from
first inclusions in January 2010 until the spring of 2013 for the last
follow-up measures. Effect sizes (Cohen's d) were calculated by
dividing the differences between means by the pooled standard
deviation. Standard deviations were calculated from the standard
errors by using the formula SD ¼ SE*√N. An effect size of 0.20 was
considered to be small, of 0.50 to be moderate, and 0.80 and above
to be large (Cohen, 1988).
To define responders on the main outcome measure, we used
the criteria for clinically significant improvement proposed by
Jacobson and Truax (1991). The reliable change index was calcu-
lated using the test-retest reliability coefficient of 0.85 and then
using the cut-off scores for the nonclinical sample (Elfstrom et al.,
2012) to decide whether or not the changes were clinically
significant.
Costs and cost-effectiveness. Costs were assessed at baseline, post-
treatment, and at 12-month follow-up. Because the control con-
dition was crossed over to treatment after post-treatment, be-
tween-group comparisons were made only for the post-treatment
data and not at 12-month follow-up. In this analysis, missing data
from baseline to post-treatment were imputed using the last
6 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11

observation carried forward (LOCF), which has been shown to
produce the same cost-effectiveness outcomes as imputation based
on linear methods when applied on cost data assessed with the TIC-
P (Andersson et al., 2011, Hedman, Andersson, Ljo  tsson, Andersson,
Rück, Mo €rtberg, et al., 2011; Hedman, Andersson, Ljo tsson,
Andersson, Rück, & Lindefors, 2011), but does not rely on
assumption of linearity of cost data. However, considering the long
period between post-treatment and 1-year follow-up LOCF was not
used at the latter assessment point. Instead, 1-year follow-up es-
timates are based only on participants who provided data. The
statistical analysis was conducted in four steps.
Incremental cost effectiveness ratio (ICER), which was the pri-
mary health economic outcome, was estimated using the following
formula: (DC1  DC2)/(DE1  DE2),where C1eC2 is the difference in
costs between ICBT and the control condition, and E1eE2 refers to
the difference of the average effectiveness of the two conditions
(Drummond, Sculpher, Torrence, O'Brien, & Stoddart, 2005). The
costs, from all cost domains, of the participants in the ICBT condi-
tion were subtracted from the costs of the participants in the
control condition. This difference was then divided by the sub-
tracted effects, i.e. improvement of the CORE-OM scores. This
procedure was bootstrapped 5000 times, generating an estimated
figure of the treatment groups' incremental costs in relation to their
incremental health benefit. Bootstrap analysis was used because it
is considered to generate reliable cost distribution estimates (Efron
& Tibshirani, 1993).
A cost-utility analysis was also conducted, which is identical to
the cost-effectiveness analysis with the exception that the cost of
an additional quality-adjusted life year (QALY; Bravo Vergel &
Sculpher, 2008) is calculated instead of an additional unit of the
CORE-OM. The analysis was performed applying the population-
based index weights proposed by the EuroQol Group (Dolan,
1997). A QALY of 1 is equivalent to one year of full health and
score of 0 equals death, which means that five years lived with a
quality of life of 0.20 yields a QALY of 1 (Bravo Vergel & Sculpher,
2008).
Two sensitivity analyses were performed to test the robustness
of the results. The main objective in these analyses was to inves-
tigate how sensitive the ICERs would be to changes of cost esti-
mates within a reasonable uncertainty range. In a first step, we
repeated the analysis but increased the estimated intervention cost
of ICBT with $200. In a second step, we performed an analysis
where $600 was added to the cost of ICBT, corresponding to the cost
of ICBT during the first year of providing the service and thus
including developmental costs and costs of establishing the treat-
ment unit. These costs were expected to decrease rapidly after
establishing the treatment unit as they are one-time costs. In
addition to analysis of ICERs, repeated measures ANOVA and t-tests
were used to test cost changes for each cost domain.
treatment period. In the control group, eight persons withdrew
their application before starting the treatment. During the treat-
ment of the control group, three participants dropped out, and two
were excluded.
Treatment effects
Pre to post changes
Effect sizes varied from d ¼ 0.20 to 0.86 with a mean effect size
of d ¼ 0.59. In terms of treatment response, 46% (23/50) of the
participants in the treatment group showed a clinically significant
improvement on the CORE-OM, compared to 12% (6/50) in the
control condition. Detailed data on the pre to post results are
presented in Table 2.
When taking a comorbid mood disorder into consideration, no
interaction effects of time and group and mood disorder were
found on either the primary measure (F (1,90.026) ¼ 1.666,
p ¼ .200) or on secondary measures (BAI; F (1,93.632) ¼ 0.266,
p ¼ .608; MADRS-S; F (1,89.482) ¼ 0.280, p ¼ .598; QOLI;
F(1,89.318) ¼ 0.832, p ¼ .364).
Taken together, the results indicate that participants in the
treatment group reported an improvement on all outcomes and the
mean between-group effect was moderate. Comorbid depression
did not moderate the outcome.
Pre to follow-up
Results from the follow up measures (see Table 3) revealed that
results from post-treatment were sustained at a one-year follow-up
with within-group effect-sizes ranging from d ¼ 1.00e0.53, with a
mean of d ¼ 0.71, indicating that the treatment had sustained ef-
fects one year after completion. On the primary measure (CORE-
OM), 33.3% (25/75) showed a clinical significant improvement one
year after completion. If non responding persons would be regar-
ded as treatment failures, these figures would have changed to
27.2% (25/92).
Further exploration of interaction effects of comorbid mood dis-
order revealed no moderating effect on the CORE-OM (F
(1,77.974) ¼ 2.353, p ¼ .102) or on the BAI (F (1,75.120) ¼ 1.301,
p ¼ .278) or the MADRS-S (F (1,71.172) ¼ 2.492, p ¼ .090). There was a
significant interaction effect of time and mood disorder on the QOLI,
however (F (1,67.021) ¼ 3.969, p ¼ .023). This indicates that patients
with a comorbid mood disorder had a larger gain in their quality-of-
life assessments from pretreatment to one year follow-up.
Table 2
Estimated means, standard error (SE), on all outcome measures at pre-treatment
and post-treatment, effects, and between-group Cohen’s effect sizes (d) at post-
treatment.
Measure Time Treatment Control Effects F Between-group d
(n ¼ 50) (n ¼ 50) (1,89e97)a

CORE-OM Pre 18.44(0.63) 17.65(0.63) T: 95.51***

3. Results
Post 10.97(0.75) 17.65(0.63) G: 5.097* 0.86
I: 28.82***
Adherence and attrition BAI Pre 21.18(1.37) 21.32(1.37) T: 68.334***
Post 11.81(1.10) 16.30(1.10) G: 2,305ns 0.58
Adherence was measured by using the percentage of completed I: 6.23*
MADRS-S Pre 19.62(0.96) 17.84(0.96) T: 59.734***
prescribed modules for the treatment group. Completion was Post 10.84(1.04) 15.94(1.04) G: 1.825ns 0.70
defined as when the participant had sent in his or her homework I: 24.77***
assignment to the therapist for the prescribed module. All pre- QOLI Pre 0.446(0.23) 0.865(0.23) T: 12.424***
scribed modules were completed by 32 per cent (16/50) of the Post 1.320(0.21) 1.033(0.21) G: 0.064ns 0.20
I:5.70*
treatment group. One participant (2%) did not complete any of the
prescribed modules, while the mean completion rate was 53.5 Abbreviations: CORE-OM Outcomes in Routine Evaluation e Outcome Measure; BAI,
percent of the prescribed modules. Beck Anxiety Inventory; MADRS-S, Montgomery Åsberg Depression Rating Scale-
Self rated; QOLI, Quality of Life Inventory; T, Time; G, Group; I, Interaction time*-
In the treatment group three persons dropped out, and two group.
were excluded (for details, see Fig. 1), leaving 45 participants ns
non significant * ¼ p < .05 ** ¼ p < .01; *** ¼ p < .001.
a
remaining in the treatment group at the end of the ten-week denominator degrees of freedom varied between 89.336 and 96.972.
 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11 7

Table 3
Estimated means, standard error (SE), standard deviation (SD) on all outcome measures at pre-treatment, post-treatment and one-year follow up, and within-group Cohen’s
effect sizes (d) at one-year follow-up.

Measure Time Estimated means (SE) SD p (time) Significant pairwise comparisons Within-group d

CORE-OM Pre 15.634 (.494) 4.74 p < .000 Pre > Post ¼ 1 yr 0.75
Post 11.314 (.579) 5.55
1Fup 11.487 (.643) 6.17
BAI Pre 16.559 (.920) 8.82 p < .000 Pre > Post ¼ 1 yr 0.53
Post 12.312 (.746) 7.16
1Fup 11.941 (.914) 8.77
MADRS-S Pre 18.098 (.722) 6.93 p < .000 Pre > Post ¼ 1 yr 1.00
Post 10.603 (.837) 8.03
1Fup 10.550 (.851) 8.16
QOLI Pre .703 (.145) 1.39 p < .000 Pre > Post ¼ 1 yr 0.57
Post 1.491 (.155) 1.49
1Fup 1.597 (.182) 1.75

Abbrevations: CORE-OM Outcomes in Routine Evaluation e Outcome Measure; BAI, Beck Anxiety Inventory; MADRS-S, Montgomery Åsberg Depression Rating Scale-Self
rated; QOLI, Quality of Life Inventory; SE, standard error; SD, Standard deviation.

Psychological treatment and medication
At the one-year follow-up, based on observed data (n ¼ 75), 36%
(27/75) reported that they had started another psychological
treatment for their condition, and 16% (12/75) reported increasing
or starting anxiolytic or antidepressant medication. A majority
90.7% (68/75) reported no changes in medication, and 12% (9/75)
reported lowering their dosage. In these data the persons not using
anxiolytic or antidepressant medication at baseline were reported
as “no changes in medication” if this had not changed. Divided into
the original groups 31% (13/42) of the treatment group started an
additional psychological treatment, 16.3% (7/42) increased or star-
ted medication, 9.5% (4/42) decreased or stopped and 69.1% (29/42)
reported unchanged medication. In the control group 39.4% (13/33)
started an additional psychological treatment, 3% (1/33) increased
or started medication, 6.1% (2/33) decreased and 51.5% (17/33) re-
ported no changes in medication. No differences were found on
altering medication (c2(2) ¼ .952, p ¼ .621) or starting psycho-
logical treatment (c2(1) ¼ .319, p ¼ .572) between the two groups.
Costs
Table 4 presents the per capita costs at baseline, post-treatment,
and at a 12-month follow-up. Analysis of change in gross total costs
from baseline to post-treatment showed a significant interaction
effect of time and group, indicating that the ICBT group made larger
cost reductions than the control condition (F (1,95) ¼ 7.80;
p ¼ .006). Analysis of subtype of the cost domain revealed that this
effect was primarily driven by larger indirect cost reductions in the
ICBT group compared to the control condition (F (1,95) ¼ 7.54;
p ¼ .007). There were no significant interaction effects of group and
time in the other cost domains ((F (1,95) ¼ 0.41e1.42); p ¼ .24e53).
Within-group tests showed that participants who received ICBT
made significant gross total cost reductions from baseline to post-
treatment (t(47) ¼ 3.23; p ¼ .002), and from baseline to the 12-
month follow-up (t(40) ¼ 4.02; p < .001), indicating stability of
cost reductions. Participants in the control condition did not
significantly reduce their gross total at post-treatment compared to
baseline (t(47) ¼ 0.94; p ¼ .35). The intervention costs per
participant were estimated to be $507 for ICBT and $68 for the
control condition.
Cost-effectiveness
At post-treatment, the incremental cost-effectiveness ratio
(ICER) was 616/0.34 ¼ $1824, favoring ICBT over the control
condition. This meant that each incremental improvement on
CORE-OM for participants in ICBT relative to the control condition
generated a societal earning of $1824. This gain was because the
total net costs were lower in the ICBT condition compared to the
control condition and because clinically significant improvements
were more likely to occur in the ICBT condition. The scatter of
simulated ICERs across the four quadrants of the ICER plane indi-
cated the degree of uncertainty around the ICER (see Fig. 2 in the
Supplementary Material online). From a cost-effectiveness
perspective, the most favorable outcome is a concentration of
scatter in the southeast quadrant, indicating superior treatment

Table 4
Per capita costs in US Dollars at pre-treatment, post-treatment and 1-year follow-up. Cost data at each assessment point.

Cost Pre-treatment Post-treatment 1-Year follow-up

ICBT CC ICBT CC ICBT CC

Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD

Direct medical 333 357 257 305 234 397 215 399 250 543 229 309
Health care visits 322 354 250 303 224 395 210 397 246 543 225 308
Medication 11 27 7 12 9 24 6 12 5 7 4 7
Direct non-medical costs 73 190 42 69 83 158 86 172 73 204 22 51
Indirect costs 2142 1691 1505 1593 1443 1668 1776 1787 1100 1499 995 1450
Unemployment 1674 1784 1073 1644 1177 1697 1376 1767 827 1533 666 1412
Sick leave 214 608 193 626 124 559 199 893 70 190 251 704
Work cutback 104 311 160 388 77 198 120 341 127 321 45 127
Domestic 150 231 78 98 65 111 74 107 77 148 33 52
Gross total costs 2548 1812 1803 1694 1757 1870 2078 1868 1423 1806 1247 1501
Intervention costs 507 68 507 575
Net total costs 2264 2146 1930 1822

Abbreviations: ICBT, Internet-based cognitive behavior therapy; CC, control condition. Note: at 1-year follow-up, participants in the control condition had received ICBT.
8 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11
effects and lower costs of the experimental treatment (ICBT)
compared to the control condition. A concentration of ICERs in the
southwest quadrant would indicate lowered treatment effects and
lowered costs of ICBT compared to the control condition. If a ma-
jority of the simulated ICERs appeared in the northwest quadrant,
ICBT would be associated with higher costs and lowered effec-
tiveness compared to the control condition, thus making it unac-
ceptable from a cost-effectiveness perspective. A majority of the
simulated ICERs are located in the southeast quadrant (95.3%),
indicating that ICBT is highly cost-effective compared to the control
condition.
The same data are used to plot the acceptability curve (see Fig. 3
in the Supplementary Material online for a graphic vision of the
curve). The curve indicates that ICBT has a 95% probability of be-
ing cost-effective if society was willing to pay $0 for one additional
improved patient. If society was willing to pay $1000 for one case of
improvement, the probability of ICBT being cost-effective would
increase to 99%.
Cost-utility analysis
At post-treatment, the cost-utility ICER was 474/
0.063 ¼ $7523. This meant that one additional QALY generated a
societal earning of $7523 when comparing ICBT to the control
condition.
A majority of the cost-utility ICERs are located in the southeast
quadrant of the ICER-plane (77%), suggesting that the most prob-
able outcome is that ICBT leads to additional QALYs while pro-
ducing societal cost reductions. Of the remaining cost-utility ICERs,
13.7% are located in the southwest quadrant, 7.5% in the northeast
quadrant, and 2.1% in the northwest quadrant (see Fig. 4 in the
Supplementary Material online).
The same data are used for the acceptability curve (see Fig. 5 in
the Supplementary Material online). The curve indicates that ICBT
has a 90% probability of being cost-effective if the society would
pay $0 for one gained QALY. If the society were willing to pay $3000
for one additional QALY, the probability of ICBT being cost-effective
would increase to 95%.
Sensitivity analyses
The acceptability curves assuming additional costs of ICBT ($200
and $600) correspond to (a) a scenario of low productivity and (b) a
scenario assuming the cost of ICBT during the first year of deliv-
ering the service. The latter analyses included all one-time costs of
development of ICBT and establishing the treatment in a health-
care context. ICBT would remain the most cost-effective treat-
ment if $200 were added to the costs of ICBT even if willingness to
pay for an additional case of improvement were $0. If $600 were
added to the intervention cost, ICBT would be most likely to be
cost-effective even if the willingness to pay was $0 if using clinically
significant improvement as ICER outcome (Fig. 3). If adding $600 to
the costs of ICBT and using QALY as outcome, the treatment would
have a 37% probability of being cost-effective in comparison to no
treatment if willingness to pay for an additional QALY was $0. This
probability would increase to 60% if society were willing to pay
$3500 for one additional QALY.
Discussion
The aim of this study was to investigate if ten weeks of tailored
ICBT could be an effective treatment for a heterogeneous sample of
patients, referred by their primary care contact, and if treatment
effects were sustained at a one-year follow-up. We also intended to
answer questions regarding whether treatment could lower direct
and indirect societal costs in this population. Findings show that
tailored ICBT was effective in our sample of primary care patients
who had comorbidities and approximately one-third of whom
were on medication for their condition. There were small-to-large
between-group effect sizes at post-treatment on the primary and
secondary outcome measures, and a clinically significant
improvement was obtained by 46 percent of the patients on the
primary outcome measure (CORE-OM). At the one-year follow-up,
results were sustained on both primary and secondary measures.
About one-third of the sample sought additional treatment. Finally,
health economic analyses suggested that the treatment can be cost-
effective. Our results thus suggest that tailored ICBT could be a
feasible approach in treating patients with anxiety disorders,
regardless of comorbidity, in primary care settings, and that it could
be a cost-effective alternative.
ICBT has been found to be effective for a range of disorders when
tested as manualized treatment targeting specific disorders in ef-
ficacy trials (Andrews et al., 2010; Cuijpers et al., 2009; Reger &
Gahm, 2009; Spek et al., 2007). There is less evidence for effec-
tiveness, even if studies suggest that ICBT can work in more regular
clinical settings (Bergstro€ m et al., 2010; Hedman, Ljo tsson, et al.,
2013; Ruwaard et al., 2012). Although there are a few previous ef-
ficacy studies on tailored ICBT (Carlbring et al., 2011), this study is
the first to test this new form of ICBT with more regular clinic pa-
tients. From a clinical perspective, where resources for conducting
extensive diagnostic test procedures may be limited and hetero-
geneity in symptom presentation is common, tailored ICBT may be
a feasible approach to use because it considers patient preferences
as well as diagnostic information and allows more patients to be
offered treatment even if they do not present with specific condi-
tions. Moreover, by using tailored ICBT, clinicians could make use of
the different manuals already existing, without having to learn a
new one for each condition diagnosed, and this might facilitate
implementation in regular health care. The tailored approach is
different than the of for example a unified protocol (Barlow et al.,
2004). The differences lie in that the patients’ specific comorbid-
ities are being addressed explicitly through what are believed to be
core symptoms of each separate diagnosis and that patient pref-
erences guide the course of treatment together with clinical char-
acteristics. In a unified protocol one single protocol is being
followed regardless of comorbidities and patient preferences.
For the time being, limited data are accessible for the cost-
effectiveness for ICBT (Hedman et al., 2012); this study provides
some evidence of cost savings without compromising clinical
effectiveness. We found that the incremental improvement
generated by ICBT was associated with a societal cost reduction,
which is a promising finding as it suggests that ICBT could poten-
tially be administered at no extra cost from a societal perspective
compared to no treatment. Incremental cost-effectiveness esti-
mates in this study were similar to those of previously reported
studies investigating cost-effectiveness of ICBT for depression
(Gerhards et al., 2010; Warmerdam, Smit, van Straten, Riper, &
Cuijpers, 2010), social anxiety disorder (Hedman, Andersson,
Ljotsson, Andersson, Rück, Mo € rtberg, et al., 2011; Hedman,
Andersson, Ljo tsson, Andersson, Rück, & Lindefors, 2011), and se-
vere health anxiety (Hedman, Andersson, Ljo  tsson, Andersson,
Rück, Mo € rtberg, et al., 2011; Hedman, Andersson, Ljo tsson,
Andersson, Rück, & Lindefors, 2011). Given that evidence-based
psychological treatments often are lacking in primary care (Stein
et al., 2004), it is in the interest of both the patients and society
to provide as effective treatment as possible, within the limits of
budget and competences. As results are emerging that guided ICBT
can be as effective as face-to-face delivered CBT (Carlbring et al.,
2005; Hedman, Andersson, Ljo tsson, Andersson, Rück, Mo € rtberg,
et al., 2011; Hedman, Andersson, Ljo tsson, Andersson, Rück, &
Lindefors, 2011; Kiropoulos et al., 2008), a crucial factor to
 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11
consider for future implementation is the role of moderators and
mediators of outcomedin other words, for which patients is ICBT
suitable and how can good outcomes be obtained (Andersson,
Carlbring, Berger, Almlo €v, & Cuijpers, 2009)? The possibility is
that moderators may reveal subgroups with different effects of
treatment, even if these relations are complex it might be of help in
decision making upon which treatment to offer (Kraemer, Wilson,
Fairburn, & Agras, 2002). Mediators on the other hand, which
answer the question how treatment works, would possibly help us
understand not only the way in which specific parts of a treatment
work, but also add to the knowledge of the nature of clinical psy-
chological distress (Kraemer et al., 2002). The present study adds to
the literature by investigating whether comorbid depression
moderates outcomes. We did not find this to be the case, and
indeed there are more studies that suggest that comorbid condi-
tions may even improve following ICBT (Titov et al., 2009). How-
ever, given the small sample size in this study, larger clinical series
of patients will be needed to investigate moderators of outcome
with adequate power.
This study has limitations. First, we did not compare guided
tailored ICBT against another active treatment or treatment as
usual, even if we did not prevent control group participants from
seeking treatment or continuing with medication (which is the
most common treatment option in GP settings in Sweden). There is
a possibility that adding an extra three or six months to the waiting
period would have provided reliable information about the effec-
tiveness of ICBT compared to passage of time or usual care. This was
not done because of ethical considerations of having patients
waiting for treatment. Also, it would have been interesting to
compare tailored ICBT against either disorder-specific treatment
(i.e., for one disorder) or against a unified treatment protocol such
as the one by Titov and colleagues (Titov et al., 2010). However, we
made use of an active control group as a control condition, which is
probably more effective than only waiting because at least some of
the nonspecific factors in therapy (e.g., being monitored and having
a contact person to send messages to) are controlled for. Second,
the follow-up measurements suffered from a significant amount of
missing data. Only 75 of the original 100 completed the online
measurements at the one-year follow-up, forcing us to estimate
means even though we were missing data. While this dropout rate
is not worse than in other studies on ICBT, it would have been
preferable to have more completers of the measures. Third, we did
not administer the SCID-interview at post treatment or at follow-
up, giving us no possibility to answer questions regarding remis-
sion or recovery from the initial diagnoses. We rely on self-report
measures as describing the outcome, a method commonly used
but possibly affecting the reliability (e.g. Eaton, Neufeld, Chen, &
Cai, 2000). Further the SCID-interviews were not recorded, so no
inter-rater reliability of the diagnostic interviews could be calcu-
lated. As a fourth limitation, it needs to be stated that this study was
only partly an effectiveness study if one considers the recommen-
dations by Shadish et al. (1997), because we did not used experi-
enced, professional therapists with regular caseloads who worked
in the settings our patients were referred from (criterion c in
Shadish et al., 1997). However, because the role of the therapist in
guided ICBT is probably less important than in regular CBT (Almlo €v
et al., 2011), it is not unlikely that the way we administered ICBT in
this study will be the way it is delivered in future health care (e.g.,
with an external therapist providing the guidance). Fifth, using the
primary care contact as gatekeeper for this treatment, in order to
mirror usual procedure might have biased the sample in that we
might have included less (or more) impaired patients in the study.
However, in order to address this we invited several primary care
facilities to participate, and GPs were allowed to refer patients to
other treatments if they wished for this. We provided all
9
participating settings with thorough information about the treat-
ment and the project. All caregivers were also advised to contact
the team with any upcoming questions. A sixth limitation might be
that the tailoring of the treatment was based on clinical judgment,
and not on empirical evidence. Since there is limited data on how
these decisions are to be done we decided to rely on the clinical
experience of the psychiatrist and the licensed clinical psychologist.
A seventh limitation was that between-group comparisons could
only be made at post-treatment and not at long-term follow-up.
Finally, an eight limitation is the fact that we did not collect any
data on how many participants actually sought other treatments
during the trial giving us no possibility to chart or analyze the
reasons or timing for this.
In spite of these limitations, we conclude that tailored, guided
ICBT appears to work with patients referred from general practice
and that tailoring can be a feasible approach to handle the issue
with comorbidity across anxiety and mood disorders. We also
conclude that tailored ICBT can be cost-effective and that the
additional costs of providing treatment clearly are compensated for
by the societal gains. Future studies should investigate the effects of
tailored ICBT in larger clinical samples, in comparison to another
active treatment, and the cost-effectiveness over longer follow-up
periods. It might also be interesting to further explore the fre-
quency and exact amount of time for the participants use of their
therapists.
Acknowledgments
We thank the health-care providers in the primary care settings
who were willing to recommend this treatment to their patients.
We also like to thank the following students; Sara Edstro€m, Therese
€m, Ika Wahlgren, Linnea Jarl, Jesper Arvidsson and Ellinor
Wågstro
Bånkestad, for their contribution. This study is funded by Forte (FAS
2008-1145), the Swedish Research Council for Health, Working Life
and Welfare.
Appendix A. Supplementary data
Supplementary data related to this article can be found at http://
dx.doi.org/10.1016/j.brat.2014.05.007.
References
€ v, J., Carlbring, P., K€
Almlo allqvist, K., Paxling, B., Cuijpers, P., & Andersson, G. (2011).
Therapist effects in guided internet-delivered CBT for anxiety disorders.
Behavioural and Cognitive Psychotherapy, 39(03), 311e322. http://dx.doi.org/
10.1017/S135246581000069X.
Alonso, J., Angermeyer, M. C., Bernert, S., Bruffaerts, R., Brugha, T. S., Bryson, H., &,
EsemeD/Mhedea Investigators, E. S. o. t. E. o. M. D. P. (2004a). 12-Month co-
morbidity patterns and associated factors in Europe: results from the European
Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psy-
chiatrica Scandinavica. Supplementum, 109(420), 28e37. http://dx.doi.org/
10.1111/j.1600-0047.2004.00328.x.
Alonso, J., Angermeyer, M. C., Bernert, S., Bruffaerts, R., Brugha, T. S., Bryson, H., &,
EsemeD/Mhedea Investigators, E. S. o. t. E. o. M. D. P. (2004b). Prevalence of
mental disorders in Europe: results from the European Study of the Epidemi-
ology of Mental Disorders (ESEMeD) project. Acta Psychiatrica Scandinavica.
Supplementum, 109(420), 21e27. http://dx.doi.org/10.1111/j.1600-
0047.2004.00327.x.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental
disorders (4th ed., text revision ed.). Washington, DC: American Psychiatric
Press.
Andersson, G. (2009). Using the Internet to provide cognitive behaviour therapy.
Behaviour Research and Therapy, 47(3), 175e180. http://dx.doi.org/10.1016/
j.brat.2009.01.010.
Andersson, E., Ljo tsson, B., Smit, F., Paxling, B., Hedman, E., Lindefors, N., et al.
(2011). Cost-effectiveness of internet-based cognitive behavior therapy for ir-
ritable bowel syndrome: results from a randomized controlled trial. BMC Public
Health, 11, 215.
Andersson, G., Carlbring, P., Berger, T., Almlo €v, J., & Cuijpers, P. (2009). What makes
internet therapy work? Cognitive Behaviour Therapy, 38(SUPPL.1), 55e60.
10 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11
Andersson, G., Carlbring, P., Holmstrom, A., Sparthan, E., Furmark, T., Nilsson-
Ihrfelt, E., et al. (2006). Internet-based self-help with therapist feedback and
in vivo group exposure for social phobia: a randomized controlled trial. Journal
of Consulting and Clinical Psychology, 74(4), 677e686. http://dx.doi.org/10.1037/
0022-006X.74.4.677.
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010). Computer
therapy for the anxiety and depressive disorders is effective, acceptable and
practical health care: a meta-analysis. PLoS ONE, 5(10), e13196. http://
dx.doi.org/10.1371/journal.pone.0013196.
Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., et al.
(2001). Service profiling and outcomes benchmarking using the CORE-OM:
toward practice-based evidence in the psychological therapies. Journal of
Consulting and Clinical Psychology, 69(2), 184e196.
Barlow, D. H., Allen, L. B., & Choate, M. L. (2004). Toward a unified treatment for
emotional disorders. Behavior Therapy, 35(2), 205e230.
Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring
clinical anxiety: psychometric properties. Journal of Consulting and Clinical
Psychology, 56(6), 893e897. http://dx.doi.org/10.1037/0022-006X.56.6.893.
Bell, C. J., Colhoun, H. C., Carter, F. A., & Frampton, C. M. (2012). Effectiveness of
computerised cognitive behaviour therapy for anxiety disorders in secondary
care. Australian and New Zealand Journal of Psychiatry, 46(7), 630e640. http://
dx.doi.org/10.1177/0004867412437345.
Bergstro €m, J., Andersson, G., Ljo
tsson, B., Rück, C., Andreewitch, S., Karlsson, A., et al.
(2010). Internet-versus group-administered cognitive behaviour therapy for
panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry, 10,
54. http://dx.doi.org/10.1186/1471-244X-10-54.
Bravo Vergel, Y., & Sculpher, M. (2008). Quality-adjusted life years. Practical
Neurology, 8(3), 175e182. http://dx.doi.org/10.1136/pn.2007.140186.
Brown, T. A., Campbell, L. A., Lehman, C. L., Grisham, J. R., & Mancill, R. B. (2001).
Current and lifetime comorbidity of the DSM-IV anxiety and mood disorders in
a large clinical sample. Journal of Abnormal Psychology, 110(4), 585e599. http://
dx.doi.org/10.1037/0021-843x.110.4.585.
Carlbring, P., & Andersson, G. (2006). Internet and psychological treatment. How
well can they be combined? Computers in Human Behavior, 22, 545e553.
Carlbring, P., Bohman, S., Brunt, S., Buhrman, M., Westling, B. E., Ekselius, L., et al.
(2006). Remote treatment of panic disorder: a randomized trial of internet-
based cognitive behavior therapy supplemented with telephone calls. Amer-
ican Journal of Psychiatry, 163(12), 2119e2125. http://dx.doi.org/10.1176/
appi.ajp.163.12.2119.
Carlbring, P., Brunt, S., Bohman, S., Austin, D., Richards, J., Ost, L. G., et al. (2007).
Internet vs. paper and pencil administration of questionnaires commonly used
in panic/agoraphobia research. Computers in Human Behavior, 23(3), 1421e1434.
http://dx.doi.org/10.1016/j.chb.2005.05.002.
Carlbring, P., Maurin, L., To€rngren, C., Linna, E., Eriksson, T., Sparthan, E., et al. (2011).
Individually-tailored, Internet-based treatment for anxiety disorders: a ran-
domized controlled trial. Behaviour Research and Therapy, 49(1), 18e24. http://
dx.doi.org/10.1016/j.brat.2010.10.002.
Carlbring, P., Nilsson-Ihrfelt, E., Waara, J., Kollenstam, C., Buhrman, M., Kaldo, V.,
et al. (2005). Treatment of panic disorder: live therapy vs. self-help via the
Internet. Behaviour Research and Therapy, 43(10), 1321e1333. http://dx.doi.org/
10.1016/j.brat.2004.10.002.
Carlbring, P., Nordgren, L. B., Furmark, T., & Andersson, G. (2009). Long-term
outcome of Internet-delivered cognitive-behavioural therapy for social phobia:
a 30-month follow-up. Behaviour Research and Therapy, 47(10), 848e850. http://
dx.doi.org/10.1016/j.brat.2009.06.012.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).
Hillsdale, NJ, USA: Lawrence Erlbaum Associates.
Craske, M. G. (2012). Transdiagnostic treatment for anxiety and depression.
Depression and Anxiety, 29(9), 749e753.
Cuijpers, P., Marks, I. M., van Straten, A., Cavanagh, K., Gega, L., & Andersson, G.
(2009). Computer-aided psychotherapy for anxiety disorders: a meta-analytic
review. Cognitive Behaviour Therapy, 38(2), 66e82. http://dx.doi.org/10.1080/
16506070802694776.
Dear, B. F., Titov, N., Schwencke, G., Andrews, G., Johnston, L., Craske, M. G., et al.
(2011). An open trial of a brief transdiagnostic internet treatment for anxiety
and depression. Behaviour Research and Therapy, 49(12), 830e837. http://
dx.doi.org/10.1016/j.brat.2011.09.007.
Denollet, J., Maas, K., Knottnerus, A., Keyzer, J. J., & Pop, V. J. (2009). Anxiety pre-
dicted premature all-cause and cardiovascular death in a 10-year follow-up of
middle-aged women. [Research Support, Non-U.S. Gov't]. Journal of Clinical
Epidemiology, 62(4), 452e456.
Drummond, M. F., Sculpher, M. J., Torrence, G. W., O'Brien, B. J., & Stoddart, G. L.
(2005). Methods for the evaluation of health care programmes. Oxford: Oxford
University Press.
Dolan, P. (1997). Modeling valuations for EuroQol health states. Medical Care, 35(11),
1095e1108.
Eaton, W. W., Neufeld, K., Chen, L.-S., & Cai, G. (2000). A comparison of self-report
and clinical diagnostic interviews for depression. Archives of General Psychiatry,
57, 217e222.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York:
Chapman & Hall.
Elfstrom, M. L., Evans, C., Lundgren, J., Johansson, B., Hakeberg, M., & Carlsson, S. G.
(2012). Validation of the Swedish version of the Clinical Outcomes in Routine
Evaluation Outcome Measure (CORE-OM). Clinical Psychology & Psychotheraphy.
http://dx.doi.org/10.1002/cpp.1788.
EuroQol-Group. (1990). EuroQol e a new facility for the measurement of health-
related quality of life. Health Policy, 16(3), 199e208.
First, M. B., Gibbon, M., Spitzer, R. L., & Williams, J. B. W. (1997). Structured clinical
interview for DSM-IV Axis I Disorders (SCID-I). Washington, D.C.: American Psy-
chiatric Press.
Frisch, M. B., Cornell, J., Villanueva, M., & Retzlaff, P. J. (1992). Clinical validation of
the Quality of Life Inventory. A measure of life satisfaction for use in treatment
planning and outcome assessment. Psychological Assessment, 4(1), 92e101.
Gerhards, S. A., de Graaf, L. E., Jacobs, L. E., Severens, J. L., Huibers, M. J., Arntz, A.,
et al. (2010). Economic evaluation of online computerised cognitive-
behavioural therapy without support for depression in primary care: rando-
mised trial. Brittish Journal of Psychiatry, 196, 310e318.
Gueorguieva, R., & Krystal, J. H. (2004). Move over ANOVA: progress in analyzing
repeated-measures data and its reflection in papers published in the Archives of
General Psychiatry. Archives of General Psychiatry, 61(3), 310e317. http://
dx.doi.org/10.1001/archpsyc.61.3.310.
Hakkaart-Van Roijen, L., Van Straten, A., & Donker, M. (2002). Manual:Trimbos/iMTA
Questionnaire for costs associated with psychiatric illness. Rotterdam: Erasmus
University.
Hedman, E., Andersson, E., Ljo tsson, B., Andersson, G., Rück, C., & Lindefors, N.
(2011). Cost-effectiveness of Internet-based cognitive behavior therapy vs.
cognitive behavioral group therapy for social anxiety disorder: results from a
randomized controlled trial. Behaviour Research and Therapy, 49(11),
729e736.
Hedman, E., Andersson, G., Ljo  tsson, B., Andersson, E., Rück, C., Mo€ rtberg, E., et al.
(2011). Internet-based cognitive behavior therapy vs. Cognitive behavioral
group therapy for social anxiety disorder: a randomized controlled non-
inferiority trial. PLoS ONE, 6(3), e18001. http://dx.doi.org/10.1371/
journal.pone.0018001.
Hedman, E., Furmark, T., Carlbring, P., Ljo tsson, B., Rück, C., Lindefors, N., et al.
(2011). A 5-Year follow-up of internet-based cognitive behavior therapy for
social anxiety disorder. Journal of Medical Internet Research, 13(2), e39. http://
dx.doi.org/10.2196/jmir.1776.
Hedman, E., Ljo tsson, B., & Lindefors, N. (2012). Cognitive behavior therapy via the
Internet: a systematic review of applications, clinical efficacy and cost-effec-
tiveness. Expert Review of Pharmacoeconomics and Outcomes Research, 12(6),
745e764.
Hedman, E., Ljo tsson, B., Rück, C., Bergstrom, J., Andersson, G., Kaldo, V., et al.
(2013). Effectiveness of Internet-based cognitive behaviour therapy for panic
disorder in routine psychiatric care. Acta Psychiatrica Scandinavica. http://
dx.doi.org/10.1111/acps.12079.
Hedman, E., Ljo  tsson, B., Rück, C., Furmark, T., Carlbring, P., Lindefors, N., et al.
(2010). Internet administration of self-report measures commonly used in
research on social anxiety disorder: a psychometric evaluation. Computers in
Human Behavior, 26(4), 736e740. http://dx.doi.org/10.1016/j.chb.2010.01.010.
Hoifodt, R. S., Strom, C., Kolstrup, N., Eisemann, M., & Waterloo, K. (2011). Effec-
tiveness of cognitive behavioural therapy in primary health care: a review.
Family Practice, 28(5), 489e504. http://dx.doi.org/10.1093/fampra/cmr017.
Jacobson, N. S., & Truax, P. (1991). Clinical significance: a statistical approach to
defining meaningful change in psychotherapy research. Journal of Consulting
and Clinical Psychology, 59(1), 12e19.
Johansson, R., Sjoberg, E., Sjogren, M., Johnsson, E., Carlbring, P., Andersson, T., et al.
(2012). Tailored vs. standardized internet-based cognitive behavior therapy for
depression and comorbid symptoms: a randomized controlled trial. PLoS ONE,
7(5), e36905.
Kiropoulos, L. A., Klein, B., Austin, D. W., Gilson, K., Pier, C., Mitchell, J., et al. (2008).
Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-
face CBT? Journal of Anxiety Disorders, 22(8), 1273e1284.
Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and
moderators of treatment effects in randomized clinical trials. Archives of General
Psychiatry, 59, 877e883.
Lindner, P., Andersson, G., Ost, € L.-G., & Carlbring, P. (2013). Validation of the
Internet-administered Quality of Life Inventory (QOLI) in different psychiatric
conditions. Cognitive Behaviour Therapy, 42, 315e327.
Mattila-Evenden, M., Svanborg, P., Gustavsson, P., & Åsberg, M. (1996). De-
terminants of self-rating and expert rating concordance in psychiatric out-
patients, using the affective subscales of the CPRS. Acta Psychiatrica Scandi-
navica, 94(6), 386e396.
Mendlowicz, M. V., & Stein, M. B. (2000). Quality of life in individuals with anxiety
disorders. American Journal of Psychiatry, 157(5), 669e682.
Mewton, L., Wong, N., & Andrews, G. (2012). The effectiveness of internet cognitive
behavioural therapy for generalized anxiety disorder in clinical practice.
Depression and Anxiety. http://dx.doi.org/10.1002/da.21995.
Nordgren, L. B., Andersson, G., Kadowaki, A., & Carlbring, P. (2012). Tailored
internet-administered treatment of anxiety disorders for primary care patients:
study protocol for a randomised controlled trial. Trials, 13, 16. http://dx.doi.org/
10.1186/1745-6215-13-16.
Olatunji, B. O., Cisler, J. M., & Tolin, D. F. (2007). Quality of life in the anxiety dis-
orders: a meta-analytic review. Clinical Psychology Review, 27(5), 572e581.
http://dx.doi.org/10.1016/j.cpr.2007.01.015.
Patterson, W. M., Dohn, H. H., Bird, J., & Patterson, G. A. (1983). Evaluation of suicidal
patients: the SAD PERSONS scale. Psychosomatics, 24(4). http://dx.doi.org/
10.1016/S0033-3182(83)73213-5, 343e345, 348e349.
Paxling, B., Almlov, J., Dahlin, M., Carlbring, P., Breitholtz, E., Eriksson, T., et al.
(2011). Guided internet-delivered cognitive behavior therapy for generalized
 L.B. Nordgren et al. / Behaviour Research and Therapy 59 (2014) 1e11
anxiety disorder: a randomized controlled trial. Cognitive Behaviour Therapy,
40(3), 159e173. http://dx.doi.org/10.1080/16506073.2011.576699.
Rabin, R., & Charro, F. d (2001). EQ-SD: a measure of health status from the EuroQol
Group. Annals of Medicine, 33(5), 337e343. http://dx.doi.org/10.3109/
07853890109002087.
Ravens-Sieberer, U., Wille, N., Badia, X., Bonsel, G., Burstro €m, K., Cavrini, G., et al.
(2010). Feasibility, reliability, and validity of the EQ-5D-Y: results from a
multinational study. Quality of Life Research, 19(6), 887e897. http://dx.doi.org/
10.1007/s11136-010-9649-x.
Reger, M. A., & Gahm, G. A. (2009). A meta-analysis of the effects of internet- and
computer-based cognitive-behavioral treatments for anxiety. Journal of Clinical
Psychology, 65(1), 53e75. http://dx.doi.org/10.1002/jclp.20536.
Rohde, P., Lewinsohn, P. M., & Seeley, J. R. (1997). Comparability of telephone and
face-to-face interviews in assessing axis I and II disorders. American Journal of
Psychiatry, 154(11), 1593e1598.
Roy-Byrne, P., Craske, M. G., Sullivan, G., Rose, R. D., Edlund, M. J., Lang, A. J., et al.
(2010). Delivery of evidence-based treatment for multiple anxiety disorders in
primary care: a randomized controlled trial. JAMA, 303(19), 1921e1928. http://
dx.doi.org/10.1001/jama.2010.608.
Ruwaard, J., Lange, A., Schrieken, B., Dolan, C. V., & Emmelkamp, P. (2012). The
effectiveness of online cognitive behavioral treatment in routine clinical prac-
tice. PLoS ONE, 7(7), e40089. http://dx.doi.org/10.1371/journal.pone.0040089.
Salim, A., Mackinnon, A., Christensen, H., & Griffiths, K. (2008). Comparison of data
analysis strategies for intent-to-treat analysis in pre-test-post-test designs with
substantial dropout rates. Psychiatry Research, 160(3), 335e345.
Shadish, W. R., Matt, G. E., Navarro, A. M., & Phillips, G. (2000). The effects of
psychological therapies under clinically representative conditions: a meta-
analysis. Psychological Bulletin, 126(4), 512e529.
Shadish, W. R., Navarro, A. M., Crits-Christoph, P., Jorm, A. F., Nietzel, M. T.,
Robinson, L., et al. (1997). Evidence that therapy works in clinically represen-
tative conditions. Journal of Consulting and Clinical Psychology, 65(3), 355e365.
Silfvernagel, K., Carlbring, P., Kabo, J., Edstrom, S., Eriksson, J., Manson, L., et al.
(2012). Individually tailored internet-based treatment for young adults and
adults with panic attacks: randomized controlled trial. Journal of Medical
Internet Research, 14(3), e65.
Smit, F., Cuijpers, P., Oostenbrink, J., Batelaan, N., de Graaf, R., & Beekman, A. (2006).
Costs of nine common mental disorders: implications for curative and pre-
ventive psychiatry. Journal of Mental Health Policy and Economics, 9(4), 193e200.
Snaith, R. P., Harrop, F. M., Newby, D. A., & Teale, C. (1986). Grade scores of the
Montgomery-Asberg depression and the clinical anxiety scales. British Journal of
Psychiatry, 148(MAY), 599e601.
11
Spek, V., Cuijpers, P., Nyklicek, I., Riper, H., Keyzer, J., & Pop, V. (2007). Internet-
based cognitive behaviour therapy for symptoms of depression and anxiety: a
meta-analysis. Psychological Medicine, 37(3), 319e328. http://dx.doi.org/
10.1017/S0033291706008944.
Stein, M. B., Sherbourne, C. D., Craske, M. G., Means-Christensen, A., Bystritsky, A.,
Katon, W., et al. (2004). Quality of care for primary care patients with anxiety
disorders. American Journal of Psychiatry, 161(12), 2230e2237. http://dx.doi.org/
10.1176/appi.ajp.161.12.2230.
Svanborg, P., & Åsberg, M. (1994). A new self-rating scale for depression and anxiety
states based on the Comprehensive Psychopathological Rating Scale. Acta Psy-
chiatrica Scandinavica, 89(1), 21e28.
Titov, N., Andrews, G., Johnston, L., Robinson, E., & Spence, J. (2010). Transdiagnostic
Internet treatment for anxiety disorders: a randomized controlled trial.
Behaviour Research and Therapy, 48(9), 890e899.
Titov, N., Dear, B. F., Schwencke, G., Andrews, G., Johnston, L., Craske, M. G.,
et al. (2011). Transdiagnostic internet treatment for anxiety and depres-
sion: a randomised controlled trial. Behaviour Research and Therapy, 49(8),
441e452.
Titov, N., Gibson, M., Andrews, G., & McEvoy, P. (2009). Internet treatment for social
phobia reduces comorbidity. Australian and New Zealand Journal of Psychiatry,
43(8), 754e759. http://dx.doi.org/10.1080/00048670903001992.
Vernmark, K., Lenndin, J., Bjarehed, J., Carlsson, M., Karlsson, J., Oberg, J., et al. (2010).
Internet administered guided self-help versus individualized e-mail therapy: a
randomized trial of two versions of CBT for major depression. Behaviour Research
and Therapy, 48(5), 368e376. http://dx.doi.org/10.1016/j.brat.2010.01.005.
Warmerdam, L., Smit, F., van Straten, A., Riper, H., & Cuijpers, P. (2010). Cost-utility
and cost-effectiveness of internet-based treatment for adults with depressive
symptoms: randomized trial. Journal of Medical and Internet Research, 12, e53.
Weisberg, R. B., Dyck, I., Culpepper, L., & Keller, M. B. (2007). Psychiatric treatment
in primary care patients with anxiety disorders: a comparison of care received
from primary care providers and psychiatrists. American Journal of Psychiatry,
164(2), 276e282. http://dx.doi.org/10.1176/appi.ajp.164.2.276.
Wittchen, H. U., & Jacobi, F. (2005). Size and burden of mental disorders in
Europeea critical review and appraisal of 27 studies. European Neuro-
psychopharmacology, 15(4), 357e376. http://dx.doi.org/10.1016/
j.euroneuro.2005.04.012.
Wittchen, H. U., Jacobi, F., Rehm, J., Gustavsson, A., Svensson, M., Jonsson, B., et al.
(2011). The size and burden of mental disorders and other disorders of the brain
in Europe 2010. European Neuropsychopharmacology, 21(9), 655e679. http://
dx.doi.org/10.1016/j.euroneuro.2011.07.018.