DOI: 10.1002/jclp.

22823

RESEARCH ARTICLE

The Working Alliance Inventory for guided

Internet interventions (WAI‐I)

Juan Martín Gómez Penedo1 | Thomas Berger1 |

Martin grosse Holtforth1 | Tobias Krieger1 | Johanna Schröder2 |
Fritz Hohagen3 | Björn Meyer4,5 | Steffen Moritz6 |
3
Jan Philipp Klein

1
Department of Clinical Psychology and
Psychotherapy, University of Bern, Abstract
Switzerland Objective: This study analyses the psychometric properties
2
Department of Psychiatry and
of the Working Alliance Inventory adapted for guided
Psychotherapy, Institute for Sex Research and
Forensic Psychiatry, University Medical Internet interventions (WAI‐I).
Center Hamburg‐Eppendorf, Hamburg,
Methods: We drew on the data set from a multicenter trial
Germany
3
Department of Psychiatry and that examined a guided Internet intervention (deprexis) for
Psychotherapy, Lübeck University, Lübeck, patients with mild to moderate depression. Two hundred
Germany
4
twenty‐three patients completed the WAI‐I and the Patient
GAIA AG, Hamburg, Germany
5
Department of Psychology, University of
Satisfaction Questionnaire (ZUF‐8) at posttreatment, and
London, London, United Kingdom the Attitudes toward Psychological Online‐Interventions
6
Department of Psychiatry and Questionnaire (APOI) at baseline. We ran confirmatory
Psychotherapy, University Medical Center
Hamburg‐Eppendorf, Hamburg, Germany factor analyses (CFA) testing two‐ and three‐factor solutions
and calculated Cronbach’s α, item‐total correlations, and
Correspondence
Juan Martín Gómez Penedo, Department of correlations of the WAI‐I with APOI and ZUF‐8.
Clinical Psychology and Psychotherapy, Results: The results suggested a two‐factor solution, with a
University of Bern, Fabrikstrasse 8, 3012
Bern, Switzerland. very good model fit and evidence of factor independency,
Email: jmgomezpenedo@gmail.com adequate internal consistency, and external validity for the
complete scale and the sub‐scales.
Conclusions: The WAI‐I showed as a reliable and valid
instrument to capture alliance in guided Internet interven-
tions, which might facilitate process‐outcome research and
treatment development efforts.

KEYWORDS
Internet‐based interventions, online interventions, therapeutic
alliance, working alliance, Working Alliance Inventory

J. Clin. Psychol. 2019;1–14. wileyonlinelibrary.com/journal/jclp © 2019 Wiley Periodicals, Inc. | 1

2 | PENEDO ET AL.

Internet interventions are effective for a variety of diagnoses, and particularly clinician‐guided interventions have
shown to have effects that equal the ones of face‐to‐face (i.e., in‐person) psychotherapy (Andrews et al., 2018;
Carlbring, Andersson, Cuijpers, Riper, & Hedman‐Lagerlöf, 2018). While unguided Internet interventions provide
information to the participants without any contact with a clinician during the treatment, in guided Internet
interventions, patients work independently through several modules, and clinicians support them via telephone or
secure e‐mail systems mostly once a week (Berger & Andersson, 2009). A qualitative study by Paxling et al. (2013)
showed that the supporting messages in guided Internet interventions consist mainly of psychoeducation, task
prompting and reinforcement, empathic expressions, enhancement of participants’ self‐efficacy, alliance fostering
comments, and deadlines flexibility. Even though the efficacy of certain guided Internet interventions has been
shown in well‐designed studies and meta‐analyses, it is also clear that these interventions are not all equally
effective for all participants. Hence, many questions remain with respect to how these effects are produced, for
whom, and under what circumstances the interventions are particularly effective (Andersson, Carlbring, Berger,
Almlöv, & Cuijpers, 2009; Dallery, Jarvis, Marsch, & Xie, 2015). The question of how a treatment effect is achieved
is usually answered by the identification of processes of change, which are events that transpire over the course of
treatment and explain outcome variance (Crits‐Christoph, Gibbons, & Mukherjee, 2013; Doss, 2004). In research on
face‐to‐face psychotherapy, the working alliance has been the most thoroughly studied process of change (Crits‐
Christoph, Gibbons, & Mukherjee, 2013) and is regarded as a robust predictor of treatment outcome according to
several meta‐analyses (see e.g., Flückiger, Del Re, Wampold, & Horvath, 2018). With regard to Internet
interventions, several recent studies have examined the working alliance (e.g., Andersson et al., 2012; Berger,
Boettcher, & Caspar, 2014; Bergman Nordgren, Carlbring, Linna, & Andersson, 2013; Jasper et al., 2014; Kiropoulos
et al., 2008, B. Klein et al., 2009, 2010; Lindner et al., 2014; Meyer et al., 2015). A narrative review (Berger, 2017)
and a recent meta‐analysis (Flückiger, Del Re, Wampold, & Horvath, 2018) suggest that average client‐rated
alliance scores are equivalent to alliance ratings in face‐to‐face therapy, and that the correlation of the alliance and
treatment outcome approximately equivalent to that found in face‐to‐face therapy research. However, there are
some notable challenges associated with research on the working alliance in Internet interventions (Berger, 2017).
One of the main challenges is a conceptual one. The working alliance is usually defined as the relationship of
collaboration between patient and therapist to achieve treatment goals (Doran, 2016; Horvath & Bedi, 2002). Three
dimensions characterize the alliance, according to Bordin’s (1979) classic conceptualization. The bond dimension
defines the emotional component of the relationship, the Tasks dimension represents patient–therapist agreement
regarding the activities conducted in therapy, and the Goals dimension describes patient–therapist agreement with
regard to the aims of treatment. This definition of the working alliance is grounded in the specific characteristics of
face‐to‐face therapy and may not be similarly applicable to other treatment formats, such as unguided and guided
self‐help and digital treatments (Berger et al., 2014; Jasper et al., 2014). For example, in unguided self‐help
interventions, which involve no contact between user and clinician during treatment, it is not possible for
intervention users to establish an interpersonal bond with a therapist. Nevertheless, some studies also suggest that
patients may develop a sense of emotional attachment toward a self‐help program, and this type of alliance with
the program appears to be related to treatment outcome (e.g., Heim, Rötger, Lorenz, & Maercker, 2018; Meyer
et al., 2015). Furthermore, in guided Internet interventions, which involve regular online or telephone contact with
a therapist, the tasks and goals of the treatment are typically not negotiated directly with the therapist like in
traditional psychotherapy (e.g., Castonguay, Constantino, & Holtforth, 2006; Safran & Muran, 2006). Instead, the
goals and the tasks to achieve these goals are typically predetermined by the interventions, although some
programs may offer a range of options and, therefore, some degree of flexibility. Thus, patient–therapist agreement
on tasks and goals may be inherently more limited in guided Internet interventions compared to face‐to‐face
therapy. To ameliorate such limitations, pretreatment interviews could be used to ensure that the intervention
seems suitable and acceptable to the client.
Another challenge in examining the alliance in Internet interventions is related to the previous one but
concerns the instruments used to measure the alliance. Almost all the studies that have analyzed the alliance in
PENEDO ET AL. | 3

Internet interventions have used the same instruments used in traditional psychotherapy (for a review see Berger,
2017). This appears problematic because the construct of the working alliance is conceptually not fully
transferrable from an interpersonal (human to human) to an Internet intervention (human‐software/computer).
Furthermore, the psychometric characteristics of these instruments may differ when they are applied to the
context of Internet interventions.
For traditional psychotherapy, the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989) is the most
commonly used instrument to measure the therapeutic alliance (Doran, 2016). In its original version, the WAI
contains 36 items to capture the three dimensions postulated by Bordin (1979), each of which is rated on a 7‐point
Likert scale by the patient, the therapist or independent raters. Tracey and Kokotovic (1989) developed a 12‐item
version of the WAI, which contains four items per subscale, and Hatcher and Gillaspy (2006) created a
revised version of the WAI‐S (i.e., WAI‐SR), which retains only positively worded items that are rated on 5‐point
Likert scales.
Recently, Kiluk, Serafini, Frankforter, Nich, and Carroll (2016) have presented an adaptation of the WAI
(Horvath & Greenberg, 1989) for technology‐based interventions (WAI‐Tech), based on the 36‐item original
version. In the WAI‐Tech most of the items corresponding to the task and goals subscales of the WAI remained
unmodified (i.e., the only change in these items was to replace the name of the therapist by the online program).
However, the items of the bond subscale were mostly adjusted, replacing the wording to refer to the attachment to
the online program. Although the study by Kiluk et al. (2014) represented an advance toward improving
the measuring of working alliance for Internet interventions, the small sample size of the study (n = 34) limited the
possibility of draw sound inferences regarding the psychometric properties of the measure. Additionally, the
instrument was only focused on exploring the alliance with an unguided computerized program without taking into
account the relational aspects involved in guided Internet interventions. Thus, to date we are not aware of any
measure with demonstrated reliability and validity that has been developed or adapted specifically for the
assessment of the working alliance in treatments using Internet interventions overall, and more specifically guided
Internet interventions.
The aims of this study are (a) to present a modified version of the WAI that is specially adapted for the context
of guided Internet interventions (i.e., the Working Alliance Inventory for Internet Interventions [WAI‐I]), and (b) to
explore its psychometric properties in a sample of participants with elevated depressive symptoms who used a
guided Internet intervention. Although this version of the WAI has been used previously in trials evaluating
Internet interventions (e.g., Berger et al., 2014), the instrument has not yet been published, nor have its
psychometric properties been explored. We anticipate that such a measure could facilitate process‐outcome
research, reveal insights on how these treatments work, and ultimately provide information to improve the
interventions and thus optimize outcomes.

1 | METHODS

1.1 | Participants
We drew on a data set from a large multicenter randomized controlled trial, the EVIDENT study (J. P. Klein et al.,
2013, 2016), which examined the effects of the Internet intervention deprexis (Meyer et al., 2009) in adults with
mild to moderate depression symptoms. Inclusion criteria were: age between 18 and 65 years, sufficient command
of the German language, willingness to participate in online as well as telephone diagnostic assessments, a score
between 5 and 14 on the Patient Health Questionnaire (PHQ‐9; Kroenke, Spitzer, Williams, & Lowe, 2010), and
electronic informed consent. Exclusion criteria were acute suicidality or a lifetime diagnosis of schizophrenia or
bipolar disorder (as determined by a telephone diagnostic interview). On the randomized control trial (RCT), a
sample of 1,013 were randomly assigned either to care‐as‐usual (n = 504) or to the Internet intervention deprexis
(n = 509). Whereas participants with mild depressive symptoms (PHQ‐9 score 5–9) received a self‐guided version of
4 | PENEDO ET AL.

the intervention (n = 192), those with moderate symptoms (PHQ score 10–14) received additional e‐mail‐support
by trained clinicians (n = 317). The sample of this study included only those 223 patients with moderate depression
symptoms who received weekly e‐mail support during the 12‐week online intervention and completed the WAI‐I at
the end of treatment. This represents a 70.34% of the patients that received e‐mail support, 43.81% of the sample
that was treated with deprexis, and 22.01% of the total sample of the study. Patients in the sample of the study
were all German speaking (n = 223), mostly female (70.4%), high school educated (48%), and in a romantic
relationship when the study began (67.7%). Patients were on average 44.48 years old (SD = 10.68). When
comparing participants that completed or not the WAI‐I (within the ones that were assigned to deprexis) using
Bayesian statistics (independent‐sample test and log‐linear regression models) we found evidence in favor of lack of
differences (i.e., supporting the null hypothesis) regarding gender, having a romantic relationship, being
unemployed, degree of use of internet, previous depressive episodes, and depression chronicity. However, there
was a significant difference on age between patients that completed or not the WAI‐I, t(507) = 2.95, Bayesian 95%
credible intervals (BCI95) [0.97, 4.81], p = .003. The Bayes factor (0.20) showed also moderate evidence in favor of a
difference in age, with patients that completed WAI‐I being older than the ones that did not completed WAI‐I.
Furthermore, there was not a difference on participants’ attitudes toward Internet interventions at baseline, as
measured by the Attitudes towards Psychological Online‐Interventions Questionnaire (APOI; Schröder et al., 2015)
between participants that completed or not the WAI‐I, t(502) = 1.62, BCI95 [−0.21, 2.17], p = .11. The Bayes factor
(BF = 3.87) provided moderate evidence in favor of the null hypothesis in this case (i.e., lack of a difference between
groups in APOI). When comparing the subsample of the study with the total sample of the RCT, there was also only
a significant difference regarding age, with people completing WAI‐I being older than the rest of the sample,
t(1,011) = 2.43, BCI95 [0.41, 3.63], p = .02, BF = 0.90.
E‐mail support consisted of feedback by trained e‐mail supporters on participants’ use of deprexis over the
previous week. The staff in charge of the e‐mail support were psychotherapists in training and master students in
the last phase of their graduate studies in clinical psychology. All the supporters received a 4‐hr training on
the online intervention and on how to provide weekly feedback to the participants, based on case materials. The
supporters were supervised continuously by an expert on Internet interventions that routinely checked the
messages to the participants, providing feedback to the supporters. Participants could respond to these messages
or contact the e‐mail supporters by initiating messages. Messages were sent through a secure e‐mail system
embedded within the Internet intervention. However, e‐mail supporters were permitted some degree of flexibility
with regard to the length and content of their messages, and the provision of these supportive messages was
accompanied by regular supervision sessions with clinical psychologists experienced in both online and offline
psychotherapy. The primary goal of the e‐mail support was to motivate participants to engage with the program,
and instructions given to e‐mail supporters were similar to those used in a previous trial (Berger, Hämmerli, Gubser,
Andersson, & Caspar, 2011). The participants of the study (i.e., that completed the WAI‐I) received an average of
12.11 messages (range: 2–23; SD = 3.23) and read on average 75.50% of the messages received. Furthermore,
they sent on average 1.99 messages to the supporters (range: 0–23; SD = 2.84), with the 54.70% of the
patients sending at least one message. Further details and main results of the EVIDENT study are reported in
J. P. Klein et al. (2013, 2016).

1.2 | Internet intervention

The Internet intervention deprexis had 10 modules (plus extra introductory and summary modules). These modules
were consistent with cognitive‐behavioral manuals, although they were not circumscribed to them. The content of
the program was provided on a simulated dialogue, where the online program explained different concepts and
techniques, illustrating them, and asking the participants to complete exercises. The program requested also
feedback to the participants that was used to tailor the further content received by the user. The participants could
do the different modules in their own pace without a fixed time to complete one. Furthermore, concerning engage
PENEDO ET AL. | 5

in treatment, the subsample of patients that completed the WAI‐I have a median of modules completed of 11
(mean = 9.74, SD = 0.30). While all the participants have done at least one module, 97.20% completed a minimum of
two modules. On average each participant used the Internet intervention for 520.34 min (SD = 21.03 min), with
98.7% of the participants using the intervention for at least 60 min. Further details on deprexis are presented in
Meyer et al. (2009) and J. P. Klein et al. (2013, 2016).

1.3 | Measures
1.3.1 | WAI‐I
The items of the WAI‐I were derived from the WAI‐SR (Munder, Wilmers, Leonhart, Linster, & Barth, 2010)
and adapted to Internet intervention programs with therapist support (Berger et al., 2014). The four items of
the bond subscale remained similar to the original scale but were rephrased slightly to refer to the
acceptance and trust between the patient and his or her e‐mail supporter. For instance, the original item
from the bond subscale of the WAI‐SR “The therapist and I respect each other” was adapted as “The
psychologist who help me in the online program and I respect each other.” The four items of the goals
subscale and the four items of the tasks subscale were rephrased more substantially, to refer to the program
instead of the therapist. As some studies suggest that the patient’s agreement with the online program’s
goals and tasks may be more important than the bond with the supporting therapist (see Berger, 2017), the
items of these subscales focused explicitly on patients’ agreement with the program's goals and tasks. For
example, the item of the goals subscale from the original WAI‐SR “The therapist and I are working towards
mutually agreed upon goals” (Hatcher & Gillaspy, 2006, p. 24) was adapted as “The goals of the online
program were in line with my goals.” As well, the item of the tasks subscale from the original WAI‐SR “What I
am doing in therapy gives me new ways of looking at my problem” (Hatcher & Gillaspy, 2006, p. 24) was
adapted into “What I did in the online program has given me new ways of looking at my problem.” Items had
to be rated on a 5‐point Likert scale ranging from 1 (never) to 5 (always) and are presented in Table 1.

T A B L E 1 Mean and standard deviations of WAI‐I items

Mean SD
WAI‐I items (dimension)
1. With the OP, it has become clearer to me how I can change (T&G) 2.96 1.05
2. What I am doing with the OP gives me new ways of looking at my problems (T&G) 2.94 1.07
3. I believe the psychologist who supports me in the OP likes me (bond) 3.02 1.34
4. I knew what to expect as a result of using the OP (T&G) 2.84 1.16
5. The psychologist who supports me in the OP and I respect each other (bond) 3.97 1.26
6. The goals of the OP are in line with my goals (T&G) 3.31 1.03
7. I feel that the psychologist who supports me in the OP appreciates me (bond) 3.64 1.36
8. The goals of the OP are important goals for me (T&G) 3.38 1.10
9. The psychologist who supports me in the OP is really interested in my well‐being (bond) 3.59 1.28
10. I feel that what I am doing in the OP will help me to accomplish the changes that I want (T&G) 3.14 1.17
11. Working with the OP helps to establish a good understanding of the kind of changes that would be 3.39 1.09
good for me (T&G)
12. I believe the way the OP is working with my problem is correct (T&G) 3.47 1.17
Abbreviations: Bond, bond with therapist dimension from WAI‐I; OP, online program; T&G, Task and goal agreement with
program dimension from WAI‐I; WAI‐I, Working Alliance Inventory for Internet Interventions.
6 | PENEDO ET AL.

1.3.2 | Patient satisfaction questionnaire (ZUF‐8)

The ZUF‐8 is a self‐report measure that explores patients’ overall satisfaction with the treatment (Schmidt,
Lamprecht, & Wittmann, 1989). It contains eight items that are rated on a 4‐point Likert scale from 1 (low
satisfaction) to 4 (high satisfaction). For this study, the instrument was adapted to explore patients’ satisfaction with
the particular Internet intervention studied in this trial (J. P. Klein et al., 2013). The German version of the ZUF‐8
showed adequate psychometric properties with good internal consistency, construct validity, and concurrent
validity (Schmidt et al., 1989). In the current sample, the ZUF‐8 showed an adequate internal consistency with a
Cronbach’s α of .92.

1.3.3 | Attitudes towards Psychological Online‐Interventions Questionnaire

The APOI is a self‐reported questionnaire that explores attitudes to psychological Internet interventions. It
contains 16 items that are rated on 5‐point Likert scales ranging from 1 (totally agree) to 5 (totally disagree; Schröder
et al., 2015). The items can be summarized in four subscales, named Scepticism and Perception of Risks (SCE),
Confidence in Effectiveness (CON), Technologization Threat (TET), and Anonymity Benefits (ABE). The APOI total
score captures the degree of positive attitudes towards online interventions. The instrument showed adequate
internal consistency and construct validity based on confirmatory factor analysis (Schröder et al., 2015). In the
current sample, the APOI had good internal consistency for the total score (α = .76). However, the Cronbach’s αs for
the subscales were lower (SCE: α = .56; TET: α = .61; ABE: α = .55), with the exception of the CON subscale (α = .80).

1.4 | Procedure
Patients completed the APOI at baseline, and the WAI‐I and ZUF‐8 after completion of the intervention (3 months).
The reason for asking patients to complete the WAI‐I at the end of treatment was that in previous studies,
participants described difficulties completing the WAI early in treatment due to minimum contact with the
supporting therapist (Jasper et al., 2014). Thus, we measured WAI‐I at posttreatment to provide sufficient time to
allow patients to develop a relationship with their supporting therapist.
Participation in the study was voluntary, and informed consent was obtained before the baseline assessments.
The procedures of the trial were approved by the Ethics Committee of the German Psychological Association
(DGPs, reference number SM 04_2012) and were conducted in compliance with the Declaration of Helsinki (World
Medical Association, 2000).

1.5 | Analytic strategies

We ran all the statistical analysis using R (R Development Core Team, 2018). To establish construct validity, we
carried out a confirmatory factor analysis with a maximum likelihood estimator, using the lavaan package (Rosseel,
2012). As all items were rated on 5‐point scales, we based our estimations on polychoric correlations, which are
suggested when studying ordinal variables (Freiberg Hoffmann, Stover, de la Iglesia, & Fernández Liporace, 2013).
We first tried a two‐factor solution with the bond dimension as one factor and tasks and goals items as a separate
joint factor (Andrade‐González & Fernández‐Liria, 2015). Although the three‐dimensional structure of the alliance
is still the most common in clinical psychology (Doran, 2016), the high correlations between goals and tasks
dimensions reported in the literature (e.g., r = .82–0.93; Falkenström, Hatcher, & Holmqvist, 2015) suggest that
these two dimensions might be construed as a single factor (usually called the technical dimension of alliance;
Andrade‐González & Fernández‐Liria, 2015). Additionally, in previous studies both exploratory and confirmatory
factor analyses yielded evidence in favor of a two‐factor solution (merging tasks and goals subscale into one)
rather than a three‐factor model (Falkenström et al., 2015; Hatcher & Barends, 1996). To evaluate the models’
PENEDO ET AL. | 7

goodness‐of‐fit, we explored recommended indices, such us χ2, χ2/degrees of freedom ratio, Comparative Fit Index
(CFI), Tucker‐Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean
Square Residual (SRMR). For comparison purposes, we also ran a three‐factor model separating tasks and goals into
two individual factors and using the χ2 test to evaluate if the three‐factor structure significantly improved the
model’s fit. To establish the factors’ convergent and discriminant validity, we analyzed intercorrelations among the
dimensions (of both the two‐ and three‐factor solutions). Between‐factors correlations are recommended to be
within the range from 0.50–0.85, indicating factor independency (Freiberg Hoffmann, de la Iglesia, Stover, &
Fernández Liporace, 2014). Correlations above 0.85 indicate convergent validity and suggest merging the
correlated factors into one (Rial Boubeta, Varela Mallou, Abalo Piñeiro, & Lévy Mangin, 2006). By contrast,
correlations below 0.50 indicate evidence of discriminant validity, suggesting including new dimensions into the
model (Rial Boubeta et al., 2006).
Furthermore, to analyze each scale’s internal consistency we computed Cronbach’s αs (for WAI‐I total score
and subscales) using the psych package (Revelle, 2018). As another indicator of reliability, we computed adjusted
item‐total correlations for the WAI‐I items. Finally, as an indicator of external validity, we ran correlations among
WAI‐I posttreatment scores, APOI baseline scores, and ZUF posttreatment scores, using the Hmisc package
(Harrell, 2018). As these correlations were at the subscale or total score level and not at the item level, we used
Pearson’s correlations for these analyses.

2 | RES U LTS

2.1 | Sample characteristics

Table 1 shows the mean and standard deviation of 12 items that constitute the WAI‐I. Additionally, in Table 2 we
present descriptive statistics for the WAI‐I, APOI, and ZUF‐8 total and subscales scores. The results showed high
scores on WAI‐I at posttreatment, similar to previous findings for online interventions and face‐to‐face therapy
(see Berger, 2017).

2.2 | Construct validity

To analyze the construct validity of the WAI‐I, we ran a CFA based on the 223 observations gathered at
posttreatment. First, we ran a two‐factor model with the eight items of goals and tasks loading on the same
dimension (i.e., task and goal agreement with program dimension) and the four bond items loading on a separate
dimension (i.e., bond with therapist dimension). Overall, the model showed adequate goodness‐of‐fit indices.
Although the χ2 test was significant,1 χ2(53) = 156.192, p < .001, the χ2/degrees of freedom ratio, that is, 2.95, was
within the recommended range between 2 and 5 (Marsh & Hocevar, 1985). Both the CFI (0.996) and TLI (0.995)
were above the 0.95 threshold (Hu & Bentler, 1999). Furthermore, the SRMR (0.062) was below the recommended
threshold of 0.08 (Hu & Bentler, 1999). Finally, the RMSEA (0.097) was below the recommended threshold and
therefore indicates an acceptable goodness of fit (i.e., 0.10; Kenny, Kaniskan, & McCoach, 2015). We present all the
standardized factor loadings from the two‐factor model in Table 3.
For comparison purposes, we ran a three‐factor model that disaggregated the task and goal agreement with
program dimension into two factors (i.e., separating task and goals items). Disaggregating this dimension into tasks
and goals neither significantly improved the overall model fit, χ2(2) = 156.19, p = .48, nor the goodness of fit indexes,
χ2(51) = 155.008, p < .001, CFI = 0.996, TLI = 0.995, RMSEA, 0.099, SRMR = 0.062.
8 | PENEDO ET AL.

T A B L E 2 Means and standard deviations of the WAI‐I, APOI, and ZUF‐8 total scores and subscales
Mean SD
WAI‐I
Total scale 3.32 0.90
(sub) Task and goal agreement with program 3.17 0.91
(sub) Bond with therapist 3.56 1.15
ZUF‐8
Total scale 3.63 0.59
APOI
Total scale 3.48 0.43
(sub) Scepticism and Perception of Risks 2.36 0.57
(sub) Confidence in effectiveness 4.15 0.57
(sub) Technologization threat 2.97 0.59
(sub) Anonymity benefits 3.12 0.74
Abbreviations: APOI, Attitudes towards Psychological Online‐Interventions Questionnaire; SD, standard deviation; (sub),
subscale; WAI‐I, Working Alliance Inventory for Internet Interventions; ZUF‐8, Patient satisfaction questionnaire.

2.3 | Convergent and discriminant validity

For the two‐factor model, the correlation between the task and goal agreement with program dimension and the bond
with therapist dimension was within the recommended range of 0.50–0.85, r = .67, SE = 0.04, 95% CI [0.59, 0.74],
z = 17.43, p < .001. This degree of association suggests that the two factors are independent, and that it is not
necessary to include more dimensions in the model (Freiberg Hoffmann et al., 2014). However, when running the
three‐factor model, the correlation between the tasks and goals dimension was above the 0.85 threshold, r = 0.98,
SE = 0.01, 95% CI [0.96, 1.00], z = 98.89, p < .001, indicating convergent validity and suggesting merging both
factors into one (Rial Boubeta et al., 2006). Nevertheless, in this model the associations between the bond and tasks
dimensions, r = .66, SE = 0.04, 95% CI [0.58, 0.75], z = 15.95, p < .001, and bond and goals dimensions, r = .67,
SE = 0.04, 95% CI [0.59, 0.74], z = 16.89, p < 0.001, were within the recommended range.

2.4 | Internal consistency

The Cronbach’s α for the WAI‐I total score was 0.93. Regarding the two subscales, the Cronbach’s αs were 0.93 for
the task and goal agreement with program dimension and 0.89 for the bond with therapist dimension.
The adjusted item‐total correlations for the WAI‐I total scale were within the range of 0.51–0.81, with a mean
adjusted correlation of 0.73. In the task and goal agreement with the program dimension, the adjusted item‐total
correlations ranged from 0.47 to 0.86, with a mean correlation of 0.77. Finally, for the bond with the therapist
dimension, the adjusted item‐total correlations were within the range of 0.76–0.88, with a mean correlation of 0.81.

2.5 | External validity

Pearson’s correlations showed that the WAI‐I total score was significantly associated with the APOI total
score (r = .29, p < .001; See Table 4). Furthermore, the APOI total score was also significantly related
to WAI‐I’s task and goal agreement with program (r = .27, p < .001) and to the bond with therapist subscale
(r = .26, p < .001).
PENEDO ET AL. | 9

T A B L E 3 Standardized factor loadings

Standardized Standard Confidence Confidence
Item ← factor regression weights error interval lower limit interval upper limit p Value
WAI‐I 3 ← bond factor .860 .027 0.807 0.913 <.001
WAI‐I 5 ← bond factor .815 .030 0.756 0.874 <.001
WAI‐I 7 ← bond factor .936 .021 0.896 0.977 <.001
WAI‐I 9 ← bond factor .894 .026 0.843 0.944 <.001
WAI‐I 1 ← T&G factor .887 .016 0.855 0.919 <.001
WAI‐I 2 ← T&G factor .892 .016 0.861 0.922 <.001
WAI‐I 4 ← T&G factor .935 .012 0.912 0.957 <.001
WAI‐I 6 ← T&G factor .858 .020 0.820 0.897 <.001
WAI‐I 8 ← T&G factor .520 .048 0.426 0.615 <.001
WAI‐I 10 ← T&G factor .844 .021 0.804 0.885 <.001
WAI‐I 11 ← T&G factor .899 .016 0.867 0.930 <.001
WAI‐I 12 ← T&G factor .941 .013 0.917 0.966 <.001
Note: For item reference, see Table 1.
Abbreviations: T&G, Task and goal agreement with program dimension from WAI‐I; WAI‐I, Working Alliance Inventory for
Internet Interventions.

Additionally, the WAI‐I total score was significantly correlated with patients’ satisfaction with treatment (ZUF‐8
total score), r = .79, p < .001. When analyzing the correlations of the subscales, WAI‐I task and goal agreement with
the program dimension had a stronger association with patients’ satisfaction, r = .82, p < .001, than the bond with the
therapist dimension, r = .54, p < .001.
Furthermore, as a follow‐up analysis of external validity, we conducted correlations among WAI‐I scores
and indicators of compliance/adherence to the online intervention. These analyses showed significant
associations among WAI‐I and (a) the amount of modules done by the participants (r = .16, p = .02), (b) the
time in minutes the participants used the intervention (r = .18, p = .006), and (c) the number of messages
sent by the participant to the supporter (r = .18, p = .02). However, the proportion of emails sent by
the supporters that were read by the participants was not significantly associated with WAI‐I total scores,
r = .08, p = .32.
Additionally, to have a more fine‐grained analysis of the associations among WAI‐I and the attitudes towards
online interventions, we also conducted correlations among WAI‐I scores and APOI’s subscales. The WAI‐I total
score was significantly associated with the APOI subscales of SCE (r = − .25, p < .001), CON (r = .34, p < .001), and
TET (r = −.26, p < .001). However, it was not significantly associated with APOI’s ABE subscale, r = .02, p = .80. The
same pattern of relationships was observed at WAI‐I’s subscales level, WAI‐I’s task and goal agreement with program
was related to the subscales SCE (r = −.20, p < .001), CON (r = .30, p < .001), and TET (r = −.23, p < .001), but not with
ABE (r = .05, p = .43). Further, the WAI‐I’s bond with therapist subscale was associated with the APOI total score
(r = .26, p < .001), and the subscales SCE (r = −.24, p < .001), CON (r = .31, p < .001), and TET (r = −.24, p < .001), but
not ABE (r = −.01, p = .91). However, these analyses at APOI’s subscale level should be consider exploratory and be
interpret extremely cautiously considering that APOI’s subscales have a considerable lower reliability than APOI’s
total score.
10 | PENEDO ET AL.

T A B L E 4 Correlations between the total scores of targeted variables of the study

Variables 1 2 3 4 5 6 7 8 9
1 WAI‐I total –
2 WAI‐I T&G – –
3 WAI‐I bond – .62** –
4 ZUF‐8 total .79** .82** .54** –
5 APOI total .29** .27** .26** .31** –
6 APOI SCE −0.25** −0.20** −0.24** −.27** – –
7 APOI CON .34** .30** .31** .33** – −.57** –
8 APOI TET −.26** −.23** −.24** −.28** – .49** −.36** –
9 APOI ABE .02 .05 −.01 −.03 – −.04 .21* −.20* –
Abbreviations: ABE, Anonymity Benefits; APOI, Attitudes towards Psychological Online‐Interventions Questionnaire;
bond, bond with therapist dimension from WAI‐I; CON, Confidence in Effectiveness SCE, Scepticism and Perception of
Risks; TET, Technologization Threat; T&G, Task and goal agreement with program dimension from WAI‐I; WAI‐I, Working
Alliance Inventory for Internet Interventions; ZUF‐8, Patient satisfaction questionnaire.
*p < .05.
**p < .001.

3 | D IS C U S S IO N

The aim of this study was to evaluate the suitability of the WAI‐I as an instrument to assess the working alliance in
the context of guided Internet interventions and to analyze its psychometric properties. Overall, the results
supported the WAI‐I as a valid and reliable measure for this purpose.
The WAI‐I showed evidence of construct validity for a two‐factor structure, with the task and goal agreement
with program and bond with therapist dimensions, which showed adequate model fit based on all the goodness‐of‐fit
indices explored (CFI, TLI, SRMR, and RMSEA; Hu & Bentler, 1999; Kenny et al., 2015) with the exception of the
more conservative the χ2 test (Bentler & Bonett, 1980). This was also the case for previous studies analyzing WAI‐
SR construct validity via CFA (e.g., Hatcher & Gillaspy, 2006; Munder et al., 2010). Additionally, in this sample, an
alternative three‐factor solution did not improve the model fit.
Furthermore, the convergent and discriminant validity analysis indicated that the two factors of the model were
independent. The results of this analysis did not suggest that the factors of the model should be merged or that a
new factor needs to be incorporated (Rial Boubeta et al., 2006). This finding provides further evidence to support a
final two‐factor structure. Moreover, the reliability analysis based on these two factors showed evidence of good
internal consistency for the WAI‐I, at the total and subscales levels (α’s ranging between .92–.94), as well as the
item level (with all the adjusted item‐total rs above .30; Rattray & Jones, 2007).
Finally, evidence emerged in support of the external validity of the WAI‐I. Participants who held more positive
attitudes towards Internet interventions reported higher scores on the WAI‐I (total score and both dimensions) at
the end of treatment. In addition, participants who scored higher on the WAI‐I total scale and subscales were more
satisfied with the intervention at the end of treatment. Interestingly, the task and goal agreement with program
dimension of the WAI‐I were more strongly correlated with patient satisfaction (r = .82) than the bond with therapist
dimension (r = .54). This finding is in line with recent studies suggesting that the agreement with tasks and goals
provided by the program may be more important in this context than agreement with the supporting clinician
(Berger et al., 2014; Meyer et al., 2015).
The present study has several limitations. First, the WAI‐I was only measured at the end of treatment. We do
not have information to establish measurement invariance from other moments during the treatment. Thus, the
PENEDO ET AL. | 11

factorial structure and item loadings might change if the instrument is measured at other time‐points during or
after therapy. In addition, as a consequence of measuring WAI‐I at posttreatment, we do not have data of those
participants who dropped out from the study between pre‐ and posttreatment. Future research might benefit from
analyzing WAI‐I in different phases of the intervention and running measurement invariance tests based on such
data. Second, we did not include other sources to measure the WAI‐I, such as observer‐based or therapist ratings.
Further studies might evaluate the WAI‐I’s concurrent validity by analyzing other perspectives of the alliance in
online interventions. Third, the sample consisted of participants with a rather narrow range of initially moderate
depressive symptoms. Thus, generalizability to patients with other levels of depression severity or other diagnoses
might need to be examined in future studies. Fourth, the difference between the patients assigned to deprexis (with
or without e‐mail support) and the ones that completed the WAI‐I at the end of treatment, might raise concerns
regarding the generalizability of the results to the population of patients treated with a guided Internet
intervention. Although comparison analysis only showed differences on the age of the participants, further research
is necessary to replicate these results and establish if they are not a consequence of a sample‐specific effect. For
example, future studies using multiple datasets to test the measurement invariance of the WAI‐I factorial structure,
would enhance the generalizability of these results. Fifth, it is worthy to note that the analysis on the psychometric
properties of the WAI‐I were conducted based on a sample of German‐speaking patients. Thus, generalizability to
other cultures might be limited. Further studies would need to explore the transcultural validity and factorial
invariance of the WAI‐I in other contexts. Sixth, in this study we did not explore the relationship between WAI‐I
and outcome. Future research might need to provide evidence regarding the associations of the alliance in guided
Internet interventions, as measured by the WAI‐I, and therapeutic outcome. Finally, the psychometric properties of
the WAI‐I were analyzed in a sample of participants who used a specific Internet intervention (i.e., deprexis) with
guided email support, so it remains unclear whether these findings generalize to other guided Internet
interventions or other guidance formats (e.g., real‐time phone or mobile text‐based support). Future research would
need to explore the generalizability of these results and the possibility of applying the WAI‐I in guided Internet
interventions with other methods of guidance.
Notwithstanding these limitations, this study provided the first evidence to suggest that the WAI‐I is a reliable
and valid instrument to evaluate the working alliance in the context of guided Internet interventions. Using this
instrument in process‐outcome research could yield insights that help to further optimize these treatments and
improve outcomes.

E ND NO T E

1
Note that the χ2 test is considered to be a too conservative indicator of model fit because it is extremely sensitive to
minimum misspecifications (Bentler & Bonett, 1980; Falkenström et al., 2015). In fact, as pointed out by Falkenström
et al. (2015) other studies analyzing the WAI‐SR’s construct validity via testing multiple models with CFAs found
significant χ2 tests in all their models, although some of them presented excellent model fit, based on approximate fit
indices (e.g., Hatcher & Gillaspy, 2006; Munder et al., 2010).

OR CID

Juan Martín Gómez Penedo http://orcid.org/0000-0001-7304-407X

Jan Philipp Klein http://orcid.org/0000-0001-9882-2261

REFERENC ES

Andersson, G., Carlbring, P., Berger, T., Almlöv, J., & Cuijpers, P. (2009). What makes internet therapy work? Cognitive
Behaviour Therapy, 38, 55–60. https://doi.org/10.1080/16506070902916400
12 | PENEDO ET AL.

Andersson, G., Paxling, B., Wiwe, M., Vernmark, K., Felix, C. B., Lundborg, L., & Carlbring, P. (2012). Therapeutic alliance in
guided internet‐delivered cognitive behavioural treatment of depression, generalized anxiety disorder and social
anxiety dis‐order. Behaviour Research & Therapy, 50, 544–550.
Andrade‐González, N., & Fernández‐Liria, A. (2015). Spanish adaptation of the working alliance inventory (WAI).
Psychometric properties of the patient and therapist forms (WAI‐P and WAI‐T). Anales de Psicología, 31, 524. https://
doi.org/10.6018/analesps.31.2.177961
Andrews, G., Basu, A., Cuijpers, P., Craske, M. G., McEvoy, P., English, C. L., & Newby, J. M. (2018). Computer therapy for the
anxiety and depression disorders is effective, acceptable and practical health care: An updated meta‐analysis. Journal of
Anxiety Disorders, 55, 70–78. https://doi.org/10.1016/j.janxdis.2018.01.001
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures.
Psychological Bulletin, 88, 588–606. https://doi.org/10.1037/0033‐2909.88.3.588
Berger, T. (2017). The therapeutic alliance in internet interventions: A narrative review and suggestions for future
research. Psychotherapy Research, 27, 511–524. https://doi.org/10.1080/10503307.2015.1119908
Berger, T., & Andersson, G. (2009). Internetbasierte psychotherapien: Besonderheiten und empirische Evidenz.
Psychotherapie, Psychosomatik, Medizinische Psychologie, 59, 159–170.
Berger, T., Boettcher, J., & Caspar, F. (2014). Internet‐based guided self‐help for several anxiety disorders: A randomized
controlled trial comparing a tailored with a standardized disorder‐specific approach. Psychotherapy, 51, 207–219.
https://doi.org/10.1037/a0032527
Berger, T., Hämmerli, K., Gubser, N., Andersson, G., & Caspar, F. (2011). Internet‐Based Treatment of Depression: A
randomized controlled trial comparing guided with unguided self‐help. Cognitive behaviour therapy, 40, 251–266.
https://doi.org/10.1080/16506073.2011.616531
Bergman Nordgren, L., Carlbring, P., Linna, E., & Andersson, G. (2013). Role of the working alliance on treatment outcome in
tailored internet‐based cognitive behavioural therapy for anxiety disorders: Randomized controlled pilot trial. JMIR
Research Protocols, 2, e4. https://doi.org/10.2196/resprot.2292
Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory,
Research and Practice, 16(3), 252–260.
Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., & Hedman‐Lagerlöf, E. (2018). Internet‐based vs. face‐to‐face cognitive
behavior therapy for psychiatric and somatic disorders: An updated systematic review and meta‐analysis. Cognitive
Behaviour Therapy, 47, 1–18. https://doi.org/10.1080/16506073.2017.1401115
Castonguay, L. G., Constantino, M. J., & Holtforth, M. G. (2006). The working alliance: Where are we and where should we
go? Psychotherapy: Theory, Research, Practice, Training, 43, 271–279.
Crits‐Christoph, P., Gibbons, M. B. C., & Mukherjee, D. (2013). Psychotherapy process‐outcome research. In M. J. Lambert
(Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 298–340). Hoboken, NY: Wiley.
Dallery, J., Jarvis, B., Marsch, L., & Xie, H. (2015). Mechanisms of change associated with technology‐based interventions for
substance use. Drug and Alcohol Dependence, 150, 14–23. https://doi.org/10.1016/j.drugalcdep.2015.02.036
Doran, J. M. (2016). The working alliance: Where have we been, where are we going? Psychotherapy Research, 26, 146–163.
https://doi.org/10.1080/10503307.2014.954153
Doss, B. D. (2004). Changing the way we study change in psychotherapy. Clinical Psychology: Science and Practice, 11,
368–386. https://doi.org/10.1093/clipsy/bph094
Falkenström, F., Hatcher, R. L., & Holmqvist, R. (2015). Confirmatory factor analysis of the patient version of the working
alliance inventory–short form revised. Assessment, 22, 581–593. https://doi.org/10.1177/1073191114552472
Flückiger, C., Del Re, A. C., Wampold, B. E., & Horvath, A. O. (2018). The alliance in adult psychotherapy: A meta‐analytic
synthesis. Psychotherapy, 55(4), 316–340. https://doi.org/10.1037/pst0000172
Freiberg Hoffmann, A., de laIglesia, G., Stover, J. B., & Fernández Liporace, M. (2014). Paradoxical Personality Scale: Its
development and construct validity analysis. International Journal of Psychological Research, 7, 49–72.
Freiberg Hoffmann, A., Stover, J. B., de laIglesia, G., & Fernández Liporace, M. (2013). Correlaciones policóricas y
tetracóricas en estudios factoriales exploratorios y confirmatorios. Ciencias Psicológicas, 7(2), 151–164.
Harrell, F. E. (2018). Hmisc: Harrell Miscellaneous. R package version 4.1‐1. Retrieved from The Comprehensive R Archive
Network website https://cran.r‐project.org/web/packages/Hmisc/Hmisc.pdf
Hatcher, R. L., & Barends, A. W. (1996). Patients’ view of the alliance of psychotherapy: Exploratory factor analysis of three
alliance measures. Journal of Consulting and Clinical Psychology, 64, 1326–1336. https://doi.org/10.1037//0022‐006x.64.
6.1326
Hatcher, R. L., & Gillaspy, J. A. (2006). Development and validation of a revised short version of the working alliance
inventory. Psychotherapy Research, 16, 12–25. https://doi.org/10.1080/10503300500352500
Heim, E., Rötger, A., Lorenz, N., & Maercker, A. (2018). Working alliance with an avatar: How far can we go with internet
interventions? Internet Interventions, 11, 41–46. https://doi.org/10.1016/j.invent.2018.01.005
PENEDO ET AL. | 13

Horvath, A. O., & Bedi, R. P. (2002). The Alliance. In Norcross, J. C. (Ed.), Psychotherapy relationships that work (pp. 37–70).
New York, NY: Oxford University Press. https://doi.org/10.1017/CBO9781107415324.004.
Horvath, A. O., & Greenberg, L. S. (1989). Development and validation of the Working Alliance Inventory. Journal of
Counseling Psychology, 36, 223–233. https://doi.org/10.1037/0022‐0167.36.2.223
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus
new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. https://doi.org/10.1080/
10705519909540118
Jasper, K., Weise, C., Conrad, I., Andersson, G., Hiller, W., & Kleinstauber, M. (2014). The working alliance in a randomized
controlled trial comparing internet‐based self‐help and face‐to‐face cognitive behavior therapy for chronic tinnitus.
Internet Interventions, 1, 49–57. https://doi.org/10.1016/j.invent.2014.04.002
Kenny, D. A., Kaniskan, B., & McCoach, D. B. (2015). The performance of RMSEA in models with small degrees of freedom.
Sociological Methods and Research, 44, 486–507. https://doi.org/10.1177/0049124114543236
Kiluk, B. D., Serafini, K., Frankforter, T., Nich, C., & Carroll, K. M. (2014). Only connect: The working alliance in computer‐
based Cognitive Behavioral Therapy. Behaviour Research and Therapy, 63, 139–146. https://doi.org/10.1016/j.brat.2014.
10.003.
Kiluk, B. D., Serafini, K., Frankforter, T., Nich, C., & Carroll, K. M. (2016). Only connect: The working alliance in computer‐
based cognitive behavioral therapy. Behaviour Research and Therapy, 63, 139–146. https://doi.org/10.1016/j.brat.2014.
10.003
Kiropoulos, L. A., Klein, B., Austin, D. W., Gilson, K., Pier, C., Mitchell, J., & Ciechomski, L. (2008). Is internet‐based CBT for
panic disorder and agoraphobia as effective as face‐to‐face CBT? Journal of Anxiety Disorders, 22, 1273–1284.
Klein, B., Austin, D., Pier, C., Kiropoulos, L., Shandley, K., Mitchell, J., & Ciechomski, L. (2009). Internet‐based treatment for
panic disorder: Does frequency of therapist contact make a difference? Cognitive Behaviour Therapy, 38, 100–113.
Klein, B., Mitchell, J., Abbott, J., Shandley, K., Austin, D., Gilson, K., & Redman, T. (2010). A therapist‐assisted cognitive
behavior therapy internet intervention for posttraumatic stress disorder: Pre‐, post‐ and 3‐month follow‐up results
from an open trial. Journal of Anxiety Disorders, 24, 635–644.
Klein, J. P., Berger, T., Schröder, J., Späth, C., Meyer, B., Caspar, F., & Hohagen, F. (2016). Effects of a psychological internet
intervention in the treatment of mild to moderate depressive symptoms: Results of the evident study, a randomized
controlled trial. Psychotherapy and Psychosomatics, 85, 218–228. https://doi.org/10.1159/000445355
Klein, J. P., Berger, T., Schröder, J., Späth, C., Meyer, B., Caspar, F., & Moritz, S. (2013). The EVIDENT‐trial: Protocol and
rationale of a multicenter randomized controlled trial testing the effectiveness of an online‐based psychological
intervention. BMC Psychiatry, 13, 239. https://doi.org/10.1186/1471‐244X‐13‐239
Kroenke, K., Spitzer, R. L., Williams, J. B., & Lowe, B. (2010). The patient health questionnaire somatic, anxiety, and
depressive symptom scales: A systematic review. General Hospital Psychiatry, 32, 345–359.
Lindner, P., Olsson, E., Johnsson, A., Dahlin, M., Andersson, G., & Carlbring, P. (2014). The impact of telephone versus e‐mail
therapist guidance on treatment outcomes, therapeutic alliance and treatment engagement in internet‐delivered CBT
for depression: A randomised pilot trial. Internet Interventions, 1, 182–187.
Marsh, H. W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self‐concept: First‐ and
higher order factor models and their invariance across groups. Psychological Bulletin, 97, 562–582.
Meyer, B., Berger, T., Caspar, F., Beevers, C. G., Andersson, G., & Weiss, M. (2009). Effectiveness of a novel integrative
online treatment for depression (Deprexis): Randomized controlled trial. Journal of Medical Internet Research, 11, e15.
https://doi.org/10.2196/jmir.1151
Meyer, B., Bierbrodt, J., Schroder, J., Berger, T., Beevers, C., Weiss, M., & Klein, J. (2015). Effects of an internet intervention
(Deprexis) on severe depression symptoms: Randomized controlled trial. Internet Interventions, 2, 48–59.
Munder, T., Wilmers, F., Leonhart, R., Linster, H. W., & Barth, J. (2010). Working Alliance Inventory‐Short Revised (WAI‐SR):
Psychometric properties in outpatients and inpatients. Clinical Psychology and Psychotherapy, 17, 231–239. https://doi.
org/10.1002/cpp.658
Paxling, B., Lundgren, S., Norman, A., Almlöv, J., Carlbring, P., Cuijpers, P., & Andersson, G. (2013). Therapist behaviours in
internet‐delivered cognitive behaviour therapy: Analyses of e‐mail correspondence in the treatment of generalized
anxiety disorder. Behavioural and Cognitive Psychotherapy, 41, 280–289. https://doi.org/10.1017/S1352465812000240
R Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing,
Retrieved from https://www.R‐project.org/
Rattray, J., & Jones, M. C. (2007). Essential elements of questionnaire design and development. Journal of Clinical Nursing,
16, 234–243. https://doi.org/10.1111/j.1365‐2702.2006.01573.x
Revelle, W. (2018). psych: Procedures for personality and psychological research. Retrieved from The Comprehensive R Archive
Network website https://CRAN.R‐project.org/package=psych
14 | PENEDO ET AL.

Rial Boubeta, A., Varela Mallou, J., Abalo Piñeiro, J., & Lévy Mangin, J. P. (2006). El Análisis Factorial Confirmatorio. In J. P.
Lévy Mangin, & J. Varela Mallou (Eds.), Modelización con estructuras de covarianzas en ciencias sociales (pp. 119–154).
Netbiblo: Coruña.
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36. https://
doi.org/10.18637/jss.v048.i02
Safran, J. D., & Muran, J. C. (2006). Has the concept of the therapeutic alliance outlived its usefulness? Psychotherapy:
Theory, Research, Practice, Training, 43, 286–291. https://doi.org/10.1037/0033‐3204.43.3.286
Schmidt, J., Lamprecht, F., & Wittmann, W. W. (1989). Satisfaction with inpatient management. Development of a
questionnaire and initial validity studies. Psychotherapie, Psychosomatik, Medizinische Psychologie, 39, 248–255.
Schröder, J., Sautier, L., Kriston, L., Berger, T., Meyer, B., Späth, C., & Moritz, S. (2015). Development of a questionnaire
measuring Attitudes Towards Psychological Online Interventions‐the APOI. Journal of Affective Disorders, 187,
136–141. https://doi.org/10.1016/j.jad.2015.08.044
Tracey, T. J., & Kokotovic, A. M. (1989). Factor structure of the Working Alliance Inventory. Psychological Assessment, 1,
207–210. https://doi.org/10.1037/1040‐3590.1.3.207
World Medical Association. (2000). World medical association declaration of Helsinki: Ethical principles for medical
research involving human subjects. Journal of the American Medical Association, 284, 3043–3045.

How to cite this article: Gómez Penedo JM, Berger T, grosse Holtforth M, et al. The Working Alliance
Inventory for guided Internet interventions (WAI‐I). J. Clin. Psychol. 2019;1–14.
https://doi.org/10.1002/jclp.22823