See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/347950819

Psychometric Properties of the Environmental

Reward Observation Scale: Study on Its Internal
Structure,
Factor Invariance, and Method Effe....

Article in Psychological Reports · December 2020

DOI: 10.1177/0033294120981930

CITATION
READS
1
54

6 authors, including:

Wildman Vilca Robert Ivan Echabaudes Ilizarbe

Universidad Privada Norbert Wiener Universidad Peruana Unión
54 PUBLICATIONS 371 CITATIONS 3 PUBLICATIONS 4 CITATIONS

Jannia Aquino Hidalgo

Jose Luis Ventura Leon
Universidad Peruana Unión
Universidad Privada del Norte (Perú)
1 PUBLICATION 1 CITATION
156 PUBLICATIONS 1,064 CITATIONS

Some of the authors of this publication are also working on these related projects:

Validación de Escala Emocionales Infantiles en el Perú View project

Uso de programas estadísticos en investigación psicológica: ¿software comercial o software libre? View
project
All content following this page was uploaded by Wildman Vilca on 15 February 2021.

The user has requested enhancement of the downloaded file.

Article

Psychometric Psychological Reports

0(0) 1–27
A The Author(s) 2020
Properties of the Article reuse guidelines:
sagepub.com/journals-permissions
Environmental Reward DOI: 10.1177/0033294120981930
journals.sagepub.com/home/prx

Observation Scale:
Study on Its Internal
Structure, Factor
Invariance, and Method
Effect Associated With
Its Negative Items

Lindsey W. Vilca ,
Robert I. Echebaudes-Ilizarbe
and Jannia M. Aquino-Hidalgo
Universidad Peruana Unio´n, Lima, Per´u

Jose´ Ventura-Leo´n
and Renzo Martinez-
Munive
Universidad Privada del Norte, Lima, Per´u

Michael White
Universidad Peruana Unio´n, Lima, Per´u

Abstract
The aim of this study was to assess the factorial structure of the scale, the method’s
effect associated with its negative items, its temporal invariance, and factorial invari-
ance according to sex. For this purpose, three samples were collected, an initial

Corresponding Author:
Lindsey W. Vilca, Departamento de Psicolog´ıa, Universidad Peruana Unio´n, Carretera Central Km.
19, N˜a~na, Lima, Per´u.
Email: lwquiro@gmail.com
2 Psychological Reports

sample of 200 participants, a second sample of 461 participants and a third sample of
107 participants; making a total of 768 Peruvian university students. Other instru-
ments were applied together with the EROS scale in order to measure
satisfaction with life, anxiety, stress and depression. Regarding the results, in the
initial sample it was found that the original scale containing positive and negative
items does ade- quately fit the data (RMSEA .19; CFI .77; TLI .71) and also evidence
¼
was found supporting the existence of a¼methodological
¼ effect associated with the
negative items. It was also found that version B of the scale which only has
positive items data fits the data (RMSEA .13; CFI .96; TLI .95). In the second
¼
sample it was found that version B still¼had a good
¼ fit to the data in a larger sample
(RMSEA .07; CFI .98; TLI .98). In addition, it was found that the scale can¼ be
¼
considered ¼
invariant according to sex and presents validity based on other
constructs. In the third sample it was found that the test-retest reliability of the scale
was adequate (.70 [CI95% .593–.788]) and also evidence was found in favor of the
temporal invariance of the scale. It is concluded that the scale formed only by
positive items presents more robust psychometric properties and constitutes a
better alternative to mea- sure the level of reward provided by the
environment.

Keywords
Reverse worded items, construct validity, factorial invariance, test-retest
reliability, Environmental Reward Observation Scale

Introduction
Behavioral Activation is a therapy set out in the third generation of cognitive-
behavioral therapy whose main objective is to carry out activities that increase
the contact with the contingencies of the environment, and thus feel reinforced
by the environment (Hayes, 2016), which is based on the assumption that the
behavioral change causes a change in state of life, in perceived rewards, in
solution of problems (P´erez-Alvares, 2014) and in emotions (Barraca & P
´erez- A´ lvarez, 2015). In addition, it is a therapy that uses a version of
functional
analysis and a philosophy based on functional contextualism to explain
human behavior (Kanter et al., 2011). Its effectiveness has been demonstrated
in several meta-analysis studies (Barth et al., 2013; Mazzucchelli et al., 2009,
2010; Zabihi et al., 2020) and in randomized clinical studies (Dimidjian et al.,
2006; Gawrysiak et al., 2009; Hopko et al., 2011). Furthermore, it is as effective
as pharmacological treatment (paroxetine) with less treatment abandonment or
refusal (Dimidjian et al., 2006) and is more effective than Cognitive Behavioral
Therapy (Richards et al., 2016). Also, several studies have demonstrated its
Vilca et 3

efficacy in group interventions (Mahen et al., 2019; Simmonds-Buckley et al.,

2019) and through mobile applications (Ly et al., 2014).
Since the development of this treatment protocol, one of the main needs was
the development of original evaluation instruments that would allow for an
effective assessment of the main objectives of the therapy: to measure the
increase in behavior and to measure the access to positive reinforcement
(Martell et al., 2013). However, the absence of instruments with robust psycho-
metric properties may make it difficult to evaluate these objectives and to objec-
tively assess the effectiveness of therapy.
Responding to this need, Armento and Hopko (2007) developed the
Environmental Reward Observation Scale (EROS), which is a brief ten-item
instrument to evaluate increased behavior and access to positive environmental
reinforcement. The instrument has been widely used in randomized clinical
studies (Ferna´ ndez-Rodr´ıguez et al., 2020; Mira et al., 2019; Va´ zquez et
al., 2019) and explanatory studies (Aoki et al., 2019; Kern et al., 2019;
Maitland et al., 2019; Mart´ınez-Vispo et al., 2019; Otero et al., 2020).
Regarding the psychometric properties of the EROS scale, the original study
conducted with university students (Armento & Hopko, 2007) showed that a
one-dimensional model fits well with the data (RMSEA .06; GFI .92;
¼ ¼
NFI .90) with good reliability in its scores (a .88). Furthermore, EROS
¼ ¼
has several adaptations. Its French version (Wagener & Blairy, 2015) showed
good evidence based for internal structure (RMSEA .06; GFI .99;
NFI .99) and reliability of scores (a .89). In the Spanish ¼version (Barraca ¼
¼ ´ ¼
& P´erez-A lvarez, 2010), its internal structure was reviewed by means of an
Exploratory Factor Analysis (EFA) showing that the items formed two factors,
although the first factor explained most of the variance (45.7%). Regarding its
reliability, it showed an adequate level of internal consistency (a .86). In
¼
Colombia (Valderrama-D´ıaz et al., 2016) the EFA showed that the items
formed two factors, where the first factor explained most of the variance
(46.8%) and the reliability of the scores was good (a .87). Additionally,
there is a Spanish version only for breast cancer¼ survivors (Ferna´
ndez- Rodr´ıguez et al., 2020), where the evidence suggests a non-adjustment of
the one-dimensional model (RMSEA .116 [IC90% .096–.137); in spite of that,
its reliability was excellent (a 91). ¼
¼
With respect to the psychometric studies carried out in Spanish-speaking
populations, there are several methodological limitations, such as the use of
the principal components method as an EFA estimator, which only takes into
account the total variance and does not differentiate between the common var-
iance and the error variance to form the factors (Watkins, 2018). In addition, the
use of Kaiser’s rule overestimates the number of factors as a function of the
number of items, that is, a greater number of items increases the number of
factors estimated (Watson, 2017). Additionally, the use of varimax rotation is
not appropriate since it assumes that the factors are not related to each other
4 Psychological Reports

(Lloret et al., 2017). Furthermore, it is worth noting that only one study
(Ferna´ ndez-Rodr´ıguez et al., 2020) carried out a Confirmatory
Factorial Analysis (CFA) to evaluate the internal structure of the scale.
However, it was carried out on a very specific sample of women, which limits
the general- izability of the results. Therefore, it can be seen that there is a lack
of conclusive psychometric studies to confirm the internal structure of the scale
in the Spanish-speaking population.
It is important to mention that all the psychometric studies carried out use
Cronbach’s Alpha coefficient (a) as a reliability indicator and there is no
evidence of the use of compound coefficients that are typically used in factorial
models and that are resistant to tau-equivalence non-compliance (Viladrich et
al., 2017). Nor has psychometric evidence been found on the test-retest
reliability of the scale, which is important for evaluating the stability of scores
over time.
Furthermore, no psychometric studies have been found that assure the fac-
torial invariance of the scale according to the sex of those evaluated; several
studies have found that women are more sensitive to reinforcements and
rewards, because they obtain a higher score in behavioral events and participate
in more behavioral domains than men (Ryba & Hopko, 2012; Tull et al., 2010).
Nor has psychometric evidence been found on the temporal invariance of the
scale that guarantees that the changes observed over time are the product of a
real change in the level of the construct.
Another important aspect is the lack of psychometric studies that evaluate
the possible methodological effect associated with the presence of negative
items in the scale, since these types of items make it difficult to adequately
understand the items (van Sonderen et al., 2013), cannot control for
acquiescence bias in the factor structure (Savalei & Falk, 2014), and generally
create an additional factor in the structure of the scale (Lindwall et al., 2012).
Finally, the review of previous psychometric studies shows that SEM has not
been used to assess validity based on the relationship to other constructs. This is
even though it is a very useful tool that provides information about the rela-
tionship between the observed variables (items) and the constructs they repre-
sent and the relationship between the measured constructs themselves
(McCoach et al., 2013). For this study, according to the literature review,
EROS was expected to have a positive relationship with life satisfaction
(Hopko et al., 2011) and a negative relationship with depression and anxiety
(Safra et al., 2019; Va´ zquez et al., 2019).
Faced with these methodological and psychometric limitations, this study has
the following objectives: (a) evaluate the internal structure of the scale through
CFA, (b) evaluate the method’s effect associated with negative items, (c) eval-
uate reliability using more robust indicators, (d) evaluate temporal reliability, (e)
evaluate temporal invariance, (f) evaluate factor invariance according to sex,
and (g) evaluate validity related to other constructs.
 Vilca et 5

Method
Participants
For this study, three samples were collected, starting with an initial sample of
200 participants in order to compare two versions of the test. Version A
involved 100 students (37% male and 63% female) aged 17–29 (M 20.8,
¼
SD 2.5) and version B involved 100 students (62% male and 38% female)
¼
aged 17–28 (M 20.4, SD 2.1). In both cases they were university students
¼ ¼
attending a private university in Lima, Peru. The sample size used for both
versions (A and B) is sufficient considering that it is a one-dimensional model
with ten indicators whose factor weights are equal to or greater than .50 (Wolf
et al., 2013).
To confirm the psychometric properties of version B, a second sample of 461
university students (58.4% male and 41.6% female) between 17 to 28 years old
(M ¼ 20.7, SD ¼ 2.4) was collected, where no significant differences were
found
between the ages of males (M ¼ 20.2, SD ¼ 2.4) and females (M ¼ 19.9,
SD 2.3) and the effect size was low (t(459) 1.06, p .288, d .24, CI95%
¼ ¼ ¼ ¼
.20–.68). Finally, to evaluate the temporal invariance of version B, a third
—
sample was collected which consisted of 107 university students of both sexes
(21.5% male and 78.5% female) from 18 to 29 years old (M 21.2, SD 2.5).
¼ ¼
The time between the first and second application was 36 days.
The following inclusion criteria were used for the collection of the three
samples: (a) informed consent of the participants, (b) ability to read and
write in Spanish, (c) not older than 40 years of age, and (d) being enrolled in a
uni- versity degree program. The following exclusion criteria were also used:
(a) failure to complete all tests, and (b) failure to complete the second
assessment in the temporal invariance.

Measures
Environmental reward observation scale (EROS). This 10-item instrument was devel-
oped by Armento and Hopko (2007) to measure the degree of reward provided
by the environment. The version adapted to Spanish by Barraca and P
´erez-
A´ lvarez (2010) was used for this study, where it showed a high reliability
(a .86). The items have four response categories that range from “totally dis-
¼
agree” (1) to “totally agree” (4), where a higher score shows greater experience
of the environment.

Depression, anxiety and stress scale (DASS-21). The short version of 21 items
adapted to Spanish by Bados et al. (2005) was used, where it showed adequate
reliability and validity indexes. Also, other studies have corroborated the exis-
tence of a model of three related factors (Patias et al., 2016; Wang et al., 2016).
The scale has three dimensions that evaluate the presence and intensity of the
6 Psychological Reports

negative emotional states of depression, anxiety and stress. In addition, each

dimension is made up of 7 items which have four categories of response ranging
from: “Does not apply to me” (0) to “Applies a lot or most of the time” (3). The
present study also evaluated its psychometric properties, where it showed ade-
quate reliability indices for the depression (a . 93; x .91), anxiety (a . 91;
¼ ¼ ¼
x .88) and stress components (a . 89; x .87). In addition, it showed that
¼ ¼ ¼
the model of three related factors presented adequate indexes of adjustment
(CFI ¼ .98; TLI ¼ .98; RMSEA ¼ .067, SRMR ¼ .043).

Diener’s satisfaction with life scale (SWLS). The short version of five items was used
for the study, adapted to Spanish by Atienza et al. (2000) where it showed
adequate reliability (a .84) and validity indexes based on the internal structure
¼
(GFI .98; NFI .99; NNFI .99; RMR .02). Also, different studies in Latin
¼ ¼ ¼ ¼
America evaluated the metric quality of the instrument in university students,
evidencing the existence of a unifactorial model (Oliver et al., 2018; Padro´s et
al., 2015). Regarding the structure of the scale, the five items have five response
categories ranging from: “Strongly disagree” (1) to “Strongly agree” (5), where a
higher score on the scale evidences greater satisfaction with life. In the present
study, it showed adequate indices of reliability (a ¼ . 90; x ¼ 91) and validity
based on internal structure (CFI ¼ .99; TLI ¼ .99; RMSEA ¼ .088;
SRMR ¼ .018).

Procedure and statistical analysis. The study was approved by an ethics committee
of the Center for Health Research and Innovation (CIISA) of the Faculty of
Health Sciences at a private university in Lima, Peru and following the rules of
the Declaration of Helsinki (World Medical Association, 2013) did not represent
a risk for participants. A cross-sectional design was used for data collection and
the instruments were applied collectively in the classrooms. In each case, three
fourth-year psychology students were trained in data collection before applying
the instruments. During the application process, anonymity and confidentiality
of the results were ensured, where the university students signed an informed
consent, the objectives of the study were explained and doubts regarding the
procedure were resolved. The average time for answering the instruments was
20 minutes.
The process followed to study the psychometric properties of the scale is
made up of three stages: (a) content-based validity, (b) study of method
effect, and (c) confirmation of psychometric properties (see Figure 1).

Content based validity. For version A of the scale, five judges participated:
four psychologists with the academic degree of Master in Psychology and one
Doctor in Psychology. Similarly, for version B, five judges participated: four
psychologists with the academic degree of Master in Psychology and one Doctor
in Psychology. All the judges had at least 15 years of professional experience.
 Vilca et 7

Figure 1. Stages of the EROS scale adaptation process.

To evaluate the structure and content of the items, all the judges used four
criteria: (a) relevance, (b) representativeness, (c) clarity, and (d) context of the
items. Aiken’s V coefficient (Aiken, 1980) was used for its quantification and an
ad hoc program in MS ExcelA format (Ventura-Leo´n, 2019) was used for
its calculation. Then thirteen psychology students in their last year of studies
eval-
uated the clarity of the items in both versions (A and B).

Study on the method’s effect. As shown in Figure 2, four models were pro-
posed, the A1 model maintains the original approach with positive and negative
items, the A2 model adds a specific factor for negative items and the A 3 model
adds a specific factor for positive items. A B model of the scale was also tested,
where the 10 items are all written in a positive way.
The Diagonally Weighted Least Squares with Mean and Variance corrected
(WLSMV) estimator was used to evaluate these models since the items are all
8 Psychological Reports

Figure 2. Models of the A and B versions of the scale.

ordinal (Brown, 2015). The chi-square test (v2), the RMSEA (Root Mean
Square Error of Approximation) and the SRMR (Standardized Root Mean
Square Residual) were used to evaluate model fit, in which case values less
than .05 indicate good fit, and between .05 and .08 are considered acceptable
(Kline, 2015). In addition, the CFI (Comparative Fit Index), TLI (Tucker-
Lewis Index) was used for these cases, values higher than .95 indicate a good fit
and higher than .90 an acceptable fit (Schumacker & Lomax, 2015). The WRMR
(Weighted Root Mean Square Residual) index was also used where values below
1.0 are adequate (DiStefano et al., 2018). For bi-factor models, the hierarchical
omega coefficient (xh) and the total omega (xt) were used to calculate reliability
(Zinbarg et al., 2005). For the one-dimensional model, the omega coefficient
was used (McDonald, 1999). All statistical analyses were performed with the
“lavaan” package (Rosseel, 2012) using the RStudio environment (RStudio
Team, 2018) for R (R Core Team, 2019).

Confirmation of psychometric properties. For the descriptive analyses of the

items (mean [M], standard deviation [SD], asymmetry [g1] and kurtosis
[g2]) the program SPSS 22.0 for Windows was used. For Confirmatory
Factor Analysis (CFA) the package “lavaan” was used (Rosseel, 2012), for
factor invariance models of the scale the package “semTools” was used
(Jorgensen et al., 2018) and for evaluating test-retest reliability the package
“irr” was used (Gamer et al., 2019). In all cases the RStudio environment
(RStudio Team, 2018) was used for R (R Core Team, 2019).
 Vilca et 9

As part of the confirmatory factor analysis, the WLSMV estimator (Weighted

Least Squares with Mean and Variance corrected) was used and the same
adjust- ment indicators made in the pilot test were taken into account. In
addition, to evaluate the relevance of the modification indexes (MI) in the
model, the Saris et al. (2009) method was used, where “M” represents that the
parameter is wrongly specified, “NM” represents that the parameter is not
wrongly specified, “EPC: M” represents a misspecification using the expected
change in the param- eter and “EPC: NM” represents that there is no
misspecification using the expected change in the parameter.
To evaluate the internal consistency of the scale, Cronbach’s alpha coefficient
(Cronbach, 1951) and omega coefficient (McDonald, 1999) were used, where a
value of x > .80 is adequate (Raykov & Hancock, 2005). The H coefficient was
also used (Mueller & Hancock, 2001) because it is robust for correlated errors.
A
value greater than .70 is acceptable (Geldhof et al., 2014). In addition, to
evaluate the test-retest reliability of the scale, the Intraclass Correlation
Coefficient (ICC) was used, for which a mixed effects bidirectional model (ICC3)
was used, for single measurement and the type of relationship was in absolute
agreement (Koo & Li, 2016). A value greater than .60 is considered acceptable
(Fleiss et al., 2013). Finally, in order to evaluate the invariance of the scale
according to sex and time invariance, a sequence of hierarchical variance
models was proposed, which were increasingly restrictive. To this end, first the
configural invariance (reference model) was tested, followed by the metric
invariance (equality of factorial loads), the scalar invariance (equality of
factorial loads and intercepts) and finally the strict invariance was tested
(equality of factorial loads, intercepts and residuals). Two strategies were used
to compare the sequence of models: first a formal sta-
tistical test was used, for which the chi-square difference (Dv2) was used where
non-significant values (p > .05) suggest invariance between groups. Second, a
modeling strategy was employed, using differences in the CFI (DCFI) where
values less than <.010 evidence model invariance between groups (Chen, 2007).
Finally, to evaluate the relationship of the EROS scale with other variables, a
structural equation model was proposed, where the degree of reward provided
by the environment (EROS) is related to the level of stress, anxiety, depression
and satisfaction with life. To estimate the model, the WLSMV estimator
(Weighted Least Squares with Mean and Variance corrected) was used and the
same adjustment indicators made in the confirmatory factor analysis were taken
into account.

Results
Content based validity
Table 1 shows the modifications made following the suggestions of the judges
and students in the items of version A and version B. In item 2 the expression
1 Table 1. Content validity of both scale versions.

Spanish adaptation Version A VRel VRep VCla VCon Version B VRel VCoh VCla VCon
1. Muchas actividades Muchas actividades de .93 .93 .93 .93 Muchas actividades de .93 .93 .93 .93
de mi vida son mi vida son mi vida son
agradables agradables agradables
2. U´ ltimamente, me he U´ ltimamente, me he .93 .93 .87 .80 U´ ltimamente, me he .93 .87 .87 .80
dado cuenta de que dado cuenta que las dado cuenta que las
las cosas que vivo experiencias que experiencias que
me hacen infeliz vivo, me hacen vivo, me hacen feliz
infeliz
3. En general, estoy En general, estoy muy 1.00 .87 .87 .93 En general, estoy muy .93 .87 .80 .93
muy satisfecho con satisfecho con la satisfecho con la
la forma en que forma en que utilizo forma en que utilizo
empleo mi tiempo mi tiempo mi tiempo
4. Me resulta fa´ cil Me resulta fa´ cil .93 .93 .73 .73 Me resulta fa´ cil .93 .93 .73 .73
encontrar con qu´e encontrar moti- encontrar moti-
disfrutar en la vida vos para disfrutar vos para disfrutar
de la vida de la vida
5. Otras personas par- A comparacio´ n de .87 .87 .73 .73 Mi vida es tan gratifi- .80 .87 .80 .73
ecen tener vidas m´ı, otras personas cante como la de
ma´ s plenas parecen tener vidas otras personas
ma´ s gratificantes
6. Ya no me resultan Ya no me resultan 1.00 .87 .93 .93 Las actividades que .93 .87 .93 .93
gratificantes activi- gratificantes activi- antes hac´ıa me
dades que antes me dades que antes me siguen resultando
agradaban agradaban gratificantes
(continued)
 Table 1. Continued
Spanish adaptation Version A VRel VRep VCla VCon Version B VRel VCoh VCla VCon
7. Desear´ıa encontrar Desear´ıa encontrar .87 .87 .73 .73 Encuentro activi- .80 .87 .80 .73
ma´ s aficiones que ma´ s dades que me
actividades
me divirtiesen que me diviertan divierten
8. Estoy satisfecho con Estoy satisfecho con .93 .80 .73 .80 Estoy satisfecho con .93 .80 .73 .80
mis logros mis logros mis logros
9. Mi vida es aburrida Mi vida es aburrida 1.00 1.00 .87 .93 Mi vida es interesante .93 1.00 .87 .93
10. Las actividades en Las actividades que .80 .93 .80 .73 Las actividades que .73 .93 .87 .80
que tomo parte realizo normal- realizo normal-
normalmente salen mente salen bien mente salen bien
bien
Nota. VRel ¼ Relevance; VRep ¼ Representativity; VCla ¼ Clarity; VCont ¼ Context.
1
 1 Psychological Reports

“experiencias que vivo” (“experiences I live”) was used in both versions and in
version B the item was written as a positive. In item 4, the expression “encontrar
motivos” (“find reasons”) was added in both versions. In item 5, the expression
“A comparacio´n de mi” (“Compared to me”) was added at the beginning of
the item and “vidas ma´s gratificantes” (“more rewarding lives”) at the end of
it for version A, in version B the item was written as a positive, keeping the
content of the item. In item 6, the wording was kept in version A and in version
B it was written as a positive. In item 7, the term “aficiones” (“hobbies”) was
changed to “actividades” (“activities”) in both versions, and in version B the
item was writ- ten as a positive. In item 9, the wording in version A was kept
and in version B it was written as a positive. Finally, in item 10 the expression
“actividades que realizo” (“activities I carry out”) was added in both versions.

Study on the method’s effect

Table 2 shows that the one-dimensional structure in the A1 model presents
inadequate indexes of adjustment to the data (RMSEA ¼ .19; CFI ¼ .77;
TLI ¼ .71; WRMR ¼ 1.51), where negative loads associated with the inverse
items are observed. On the other hand, the A2 model, where a specific factor

Table 2. Comparison of adjustment indicators of both scale versions.

Adjustment rates

Models X2 gl p X2/gl RMSEA SRMR CFI TLI WRMR

Model A1 166.77 35 .00 4.76 .19 .14 .77 .71 1.51
Model A2 59.23 30 .00 1.97 .10 .06 .95 .93 .73
Model A3 91.76 30 .00 3.05 .14 .08 .89 .84 .90
Model B 99.93 35 .00 2.85 .13 .06 .96 .95 .86
Standardized estimates of parameters

Items Model A1 Model A2 Model A3 Model B

Item 2 (—) —.58 —.37 .48 —.63 .82

Item 5 (—) —.59 —.20 .75 —.69 .80
Item 6 (—) —.74 —.41 .72 —.83 .83
Item 7 (—) —.54 —.18 .75 —.66 .78
Item 9 (—) —.65 —.55 .36 —.72 .88
Item 1 (þ) .66 .73 .49 .49 .78
Item 3 (þ) .55 .69 .23 .75 .67
Item 4 (þ) .33 .40 .14 .42 .69
Item 8 (þ) .73 .82 .48 .63 .78
Item 10 (þ) .67 .76 .32 .73 .86
Note: Reliability level of the A2 model: xH ¼ .17; xt ¼ .72 Reliability level of model B (x) ¼ .91.
 Vilca et 1

was added for negative items, presents better adjustment indexes

(RMSEA ¼ .10; CFI ¼ .95; TLI ¼ .93; WRMR ¼ .73). In addition, it can be
seen that the reliability of the A2 model, considering only the general factor
as true variance, presents a very low level (xH .17). On the other hand, when
¼
the specific factor is included as part of the true variance, the reliability of the
model increases notably (xt .72). This change could be attributed to the meth-
¼
od’s effect associated with the negative items. Following this line of thinking,
the A3 model was evaluated, where a specific factor was added for the positive
items. As shown in Table 2, this model does not present adequate indexes of
adjustment to the data (RMSEA .14; CFI .89; TLI .84; WRMR .90),
¼ ¼ ¼ ¼
evidencing that the presence of positive items in the scale does not impact the
model’s adjustment. These results show that there is a methodological effect
associated with the presence of negative items in the scale.
In light of this, a B model was proposed, where all the items are written in
positive way. As shown in Table 2, this model has adequate adjustment indexes
(RMSEA ¼ .13; CFI ¼ .96; TLI ¼ .95; WRMR ¼ .86) and has a high level of
reliability (x .91) which is higher than the A2 model. It is also noteworthy
¼
that the items in model B show a higher factorial weight than in the other
models. Taking into account these findings, the B model was chosen for the
final version in the process of adaptation of the scale.

Confirmation of psychometric properties

Descriptive analysis of the scale items. Table 3 shows that item 7 has the highest
average score in the total sample (M ¼ 3.17) and similarly in the group of
men (M ¼ 3.15) and women (M ¼ 3.20). It is also observed that item 3 presents the
lowest average score in the total sample (M ¼ 2.78) and in the different groups
of

Table 3. Descriptive analysis of the items.

Total sample (n ¼ 461) Males (n ¼ 269) Women (192)

Items M SD g1 g2 M SD g1 g2 M SD g1 g2
E1 3.13 .76 —.92 1.04 3.11 .80 —.98 1.02 3.17 .71 —.77 .91
E2 3.12 .72 —.76 .88 3.09 .76 —.82 .87 3.15 .68 —.60 .72
E3 2.78 .82 —.28 —.42 2.80 .85 —.26 —.58 2.77 .77 —.32 —.15
E4 3.01 .73 —.66 .66 2.98 .78 —.66 .37 3.05 .66 —.59 1.08
E5 2.95 .75 —.43 —.00 2.91 .77 —.38 —.16 3.01 .72 —.50 .31
E6 2.99 .74 —.57 .34 2.97 .76 —.61 .36 3.01 .72 —.50 .31
E7 3.17 .71 —.76 .81 3.15 .74 —.75 .65 3.20 .69 —.77 1.11
E8 2.99 .81 —.51 —.21 2.95 .84 —.45 —.42 3.06 .76 —.59 .17
E9 3.05 .78 —.64 .20 3.05 .80 —.65 .07 3.06 .74 —.62 .44
E10 2.99 .66 —.66 1.33 2.96 .70 —.65 .92 3.03 .59 —.60 2.03
Note: M¼Mean; SD¼Standard Deviation; g1¼ Skewness; g2¼ Kurtosis.
 1 Psychological Reports

Table 4. One-dimensional model fit rates and invariance models by sex.

One-dimensional
model v2 df p TLI IFC SRMR RMSEA Dv2 Ddf p DCFI

Total sample 114.24 32 .000 .98 .99 .028 .075 — — — —

According to sex
Men 81.27 32 .000 .98 .99 .031 .076 — — — —
Women 90.39 32 .000 .97 .98 .042 .089 — — — —
Configural 171.23 64 .000 .98 .98 .039 .085 — — — —
Metric 172.74 73 .000 .98 .98 .039 .077 7.17 9 .618 .001
Scalar 170.30 92 .000 .98 .98 .039 .061 13.36 19 .819 .007
Strict 144.02 93 .000 .99 .99 .039 .049 .34 1 .557 .004
According to time
Test 56.13 32 .005 .96 .97 .062 .084 — — — —
Retest 57.98 32 .003 .96 .97 .068 .977 — — — —
Configural 113.20 64 .000 .96 .97 .070 .085 — — — —
Metric 108.35 73 .005 .97 .98 .071 .068 4.82 9 .849 .007
Scalar 118.87 89 .019 .98 .98 .073 .056 13.47 16 .637 .003
Strict 112.05 90 .058 .98 .98 .073 .048 .29 1 .586 .004

men (M ¼ 2.80) and women (M ¼ 2.77). In addition, it can be seen that the
items present adequate indices of asymmetry and kurtosis ( 1.5) in the total
sample and in all the specific groups.

Validity based on internal structure

It can be seen in Table 4 that the proposed one-dimensional model presents
adequate adjustment indexes (CFI ¼ .98; TLI ¼ .98; RMSEA ¼ .075). It can
also be seen that the factorial weight of the latent variable with each one of its
observed variables is high and significant (see Figure 3). Furthermore, following
Saris et al. (2009) method to evaluate the relevance of the modification indexes
(MI) in the model and according to the item content analysis, a correlation
was specified between the errors of items 1 and 2 (.34), 4 and 6 (.25), and 9
and 10 (.33).

Factorial invariance according to sex and time invariance of the scale

Table 4 shows the adjustment indexes for the sequence of invariance models
proposed and its difference with the least restrictive model in the sequence.
Regarding factor invariance by sex, the configural model was evaluated first,
which fit the data well (TLI ¼ .98; CFI ¼ .98; RMSEA ¼ .085; SRMR ¼ .039)
and can therefore be used as a reference model for the evaluation of the follow-
ing models. Next, the metric invariance was evaluated, where it can be seen that
only the RMSEA index improved slightly (TLI ¼ .98; CFI ¼
.98;
 Vilca et 1

Figure 3. Confirmatory factor analysis of the scale.

RMSEA ¼ .077; SRMR ¼ .039). Also, it can be seen that the differences in chi-
square were statistically not significant (Dv2 ¼ 7.17; p > .05) and there were no
important changes when comparing both models (DCFI ¼ .001). Therefore, the
metric invariance of the scale is affirmed. After evaluating the scalar invariance,
where only the RMSEA index improved again (TLI ¼ .98; CFI ¼ .98;
RMSEA ¼ .061; SRMR ¼ .039), no significant differences were observed
between the two models (Dv2 ¼ 13.36; p > .05) and no important changes are
seen in regards to the metric model either (DCFI ¼ .003). Finally, strict
invari-
ance was calculated, where the adjustment indices improved slightly (TLI ¼ .99;
CFI ¼ .99; RMSEA ¼ .049; SRMR ¼ .039), no significant differences are
observed between the two models (Dv2 ¼ .34; p > .05), and no important
changes are seen with respect to the scalar model (DCFI ¼ .003). Therefore, the
scale has proven to be strictly invariant for both groups (men and women).
Similarly, the
scale also showed evidence of being invariant over time, in the invariance
models proposed: metric (DCFI ¼ .007), scalar (DCFI ¼ .003) and strict
invariance (DCFI ¼ .004).

Scale reliability
The scale evidences adequate levels of reliability since it presents a high omega
coefficient (x ¼ .93 [IC95% ¼ .92–.94]), a Cronbach’s alpha coefficient that
can be considered excellent (a ¼ .93 [IC95% ¼ .92–.94]) and an adequate H coeffi-
cient (.94). Furthermore, the ICC for test-retest reliability was adequate (.70
[CI95% .593–.788]).
 1 Psychological Reports

Table 5. Explanatory model.

Structural model

X2 gl p X2/gl RMSEA IQ 90%. SRMR IFC TLI

828.14 581 .00 1.43 .03 .026–.035 .04 .96 .96

Measurement models

EROS Stress Anxiety Depression Satisfaction

k (item) k (item) k (item) k (item) k (item)

.62 (E1) .42 (D1) .48 (D2) .71 (D3) .88 (S1)
.66 (E2) .67 (D6) .64 (D4) .70 (D5) .80 (S2)
.70 (E3) .71 (D8) .74 (D7) .76 (D10) .87 (S3)
.75 (E4) .77 (D11) .73 (D9) .79 (D13) .72 (S4)
.78 (E5) .74 (D12) .79 (D15) .77 (D16) .64 (S5)
.69 (E6) .74 (D14) .78 (D19) .76 (D17)
.74 (E7) .75 (D18) .74 (D20) .76 (D21)
.73 (E8)
.75 (E9)
.68 (E10)
Note: k: Factorial load.

Validity based on relationship to other variables

Based on the literature review, a SEM model was proposed to evaluate the latent
relationship between the EROS scale and the levels of stress, anxiety, depression
and life satisfaction. As shown in Table 5, the structural model presents ade-
quate adjustment indexes (RMSEA ¼ .03; CFI ¼ .96; TLI ¼ .96) and the mea-
surement models are adequately represented by their items.
Figure 4 shows that the degree of reward provided by the environment is
negatively related to the level of depression (.34; p < .01), anxiety (.22; p <
.01) and stress (.21; p < .01). Furthermore, it has a positive relationship with the
degree of satisfaction with life (.51; p < .01). Taking into account these
results, it can be concluded that the scale presents validity related to other
constructs.

Discussion
The aim of this study was to adapt and evaluate the psychometric properties of
the Spanish version of the EROS scale in Peruvian university students, because
it is one of the most used instruments in the framework of behavioral activation
(BA) as a treatment for depression (Barth et al., 2013; Richards et al., 2016) and
other mental health problems (Ferna´ ndez & Mairal, 2017; Hirayama et
al., 2019).
 Vilca et 1

Figure 4. Model of latent relationships.

In the initial sample, four models were tested, three models (A 1, A2, A3) that
maintain the original approach of positive and negative items; and a B model
where all items were written in a positive way. It was found that the original A 1
model presents inadequate adjustment indexes (RMSEA .19; CFI .77;
¼ ¼
TLI .71; WRMR 1.51), although this result does not coincide with what
¼ ¼
was found in the original study by Armento and Hopko (2007) where they evi-
denced that the one-dimensional model did present adequate adjustment indexes
(RMSEA .06; GFI .92; NFI .90) and neither does it coincide with the psy-
¼
chometric study ¼ out in France
carried ¼ (Wagener & Blairy, 2015) where it also
showed adequate fit indexes (RMSEA .06; GFI .99; NFI .99).
This difference in model fit could¼ be attributed ¼ ¼
to cultural differences
between countries. People living in highly individualistic countries prioritize
autonomy and self-realization while people living in less individualistic
countries prioritize social interaction and cohesion (Hofstede et al., 2010). In
addition, less individ- ualistic countries show greater empathic concern,
kindness, life satisfaction, pro- social behavior (Chopik et al., 2017) and lower
levels of loneliness (Heu et al., 2019). All of this can influence the way in which
people perceive and value the degree of reward that their environment provides;
that is, the cultural aspect can condition the response to the items and affect the
factor structure of the scale between countries with marked individualism,
such as the United States (IC 91) and France (IC 71) and more
¼
collectivist countries such ¼as Peru (IC 16). The classification of countries
¼
according to their level of individual- ism was based on the information
provided by (Hofstede Insights, 2020).
Regarding the adaptation of the instrument in less individualistic countries
such as Spain (compared with the rest of the European countries) and Latin
America, it was found that in the Spanish adaptation (Barraca & P´erez-A´
lvarez,
2010) the Exploratory Factor Analysis (EFA), showed the possible existence of
 1 Psychological Reports

two factors, where items 5 (.31), 6 (.28), 7 (.65) and 10 (-.34) entered in both
factors, most of which are written in a negative way. Similarly, in the adaptation
to Colombia (Valderrama-D´ıaz et al., 2016) the EFA showed the existence of
two factors that explained 56.9% of the accumulated variance, and again item 7
(.47) showed a high factorial weight in both factors. In another study of adap-
tation for women survivors of breast cancer in Spain (Ferna´ ndez-Rodr
´ıguez et al., 2020), the one-dimensional model presented adjustment problems
(RMSEA .116) and the values of the Unidimensional Congruence
(UniCo .89) ¼ and Explained Common Variance (ECV .81) did not support
¼ ¼
the presence of a one-dimensional model in the data (UniCo > .95; ECV >
.85).
These adjustment problems observed in the present and previous studies
could be due to the method’s effect caused by the presence of inverse items,
since these can form additional factors not associated with the construct
(Brown, 2015; Toma´ s et al., 2013; Zhang et al., 2016). Taking this into
account, the A2 model was tested where a second specific factor was added only
for negative items (2, 5, 6, 7 and 9). This model improved the fit
indexes
(RMSEA .10; CFI .95; TLI .93; WRMR .73) and the total omega coef-
¼
ficient drastically and¼showed that ¼ including the ¼ specific factor as part of the
true variance notably improves the reliability of the model (xt ¼ .72). This was
not the case in the A3 model where including a specific factor for positive items
did not contribute to the fit indexes of the model (RMSEA .14; CFI .89;
¼ ¼
TLI .84; WRMR .90). So these findings confirm the existence of method
¼ ¼
bias associated with negative items.
To overcome this limitation, the B model of the scale was proposed with all
positively phrased items, where the results showed adequate adjustment indexes
in their majority (RMSEA .13; CFI .96; TLI .95; WRMR .86) and a
¼ ¼ ¼ ¼
better level of reliability (x .91) than the previous models. In addition, the
¼
items in model B show a higher factorial weight than in the other models.
In the large sample, the B model of the EROS scale continued to show ade-
quate adjustment indicators (RMSEA .07; SRMR .02; CFI .98; TLI .98;
WRMR .78) and all items showed a high ¼ factorial weight
¼ ¼ evidencing
(k>.70) ¼
¼
that negative items transformed to positive continue to adequately measure the
degree of reward provided by the environment. However, some correlated errors
were found, which can be attributed to similar conceptual content among the
items (Brown, 2015). In addition, the analysis of the modification indexes (MI)
following the method of Saris et al. (2009) showed that it is pertinent to include
these correlated errors in the model.
Regarding items 1 (“Many activities in my life are pleasant”) and 2 (“Lately, I
have realized that the experiences I live, make me happy”), both items refer to
activities or experiences that produce some degree of satisfaction. In the case of
item 4 (“I find it easy to find reasons to enjoy life”) and item 6 (“The activities I
used to do are still rewarding”), both items refer to some degree of satisfaction
with life. Finally, items 8 (“I am satisfied with my achievements”) and 9 (“My
 Vilca et 1

life is interesting”) have in common that both refer to a positive affectivity of

their personal experience.
Furthermore, these results constitute the first empirical evidence of the
existence of a one-dimensional model for this scale in Spanish for university
students, since in previous studies (Barraca & P´erez-A´ lvarez, 2010;
Valderrama- D´ıaz et al., 2016) only exploratory analyses were carried out,
where the existence
of a one-dimensional model was not clear. It is important to note that no other
psychometric studies of the scale have been found in Latin America or in other
countries of the world besides the United States, France and Spain.
Regarding the reliability of the scale, the a coefficient shows an excellent level
of reliability (a .93) which is similar to that found in previous studies, where
they also used the ¼ same coefficient (Armento & Hopko, 2007; Barraca & P
´erez-
A´ lvarez, 2010; Ferna´ ndez-Rodr´ıguez et al., 2020; Valderrama-D´ıaz et al.,
2016;
Wagener & Blairy, 2015) although this indicator can lead to biased estimates of
reliability since it is difficult to guarantee the tau-equivalence of items and it is
sensitive to the presence of correlated errors in the model (Dunn et al., 2014;
Yang & Green, 2010). Furthermore, its use is not recommended for ordinal
items (Elosua & Zumbo, 2008) . Therefore, to overcome these limitations, the
omega coefficient (x .91) was used, which showed adequate reliability indices.
The H coefficient was ¼ also used since it is robust regarding the presence of
correlated errors (Dominguez-Lara, 2016). In this indicator the scale also
showed an adequate level of reliability (.94). In addition, the test-retest
method shows evidence that the scale scores are reliable over time
(ICC .60). This result allows for the objective contrast of a participant’s
changes ¼ through time and presents objective data on the effectiveness of the
therapy. All the coefficients used show an adequate level of reliability, thus
ensuring less measurement error and greater accuracy of the scores obtained.
These results also constitute the first empirical evidence on the reliability of the
scale, through other indicators more robust than the traditional Cronbach’s a.
Additionally, the sequence of hierarchical variance models proposed in the
invariance study show that the scale can be considered invariant according to
sex and over time, for this purpose the DCFI index was used as the main crite-
rion since DRMSEA can be affected by the sample size (Chen, 2007). This
allows for the comparison of the scores between men and women, as well as
the changes that can occur over time, asuring that the differences found are due
to differences in the actual level of the construct. These results are particularly
important when evaluating the effectiveness of intervention therapies.
Furthermore, it is important to note that these results constitute the first empir-
ical evidence on factorial invariance of the scale.
Regarding the validity in relation to other variables, it was found that the
degree of reward provided by the environment is positively related to the degree
of satisfaction with life (q .51; p < .01). This result is coherent since if people
¼
positively value each aspect of their lives then that would allow them to feel
 2 Psychological Reports

reinforced by their environment. In addition, it was found that the EROS scale
has a negative relationship with the level of depression (q ¼ —.34; p < .01), anx-
iety (q ¼ —.22; p < .01) and stress (q ¼ —.21; p < .01). This relationship can be
explained because depressive symptoms are strongly associated with reduced
availability and sensitivity to environmental reward (Eshel & Roiser, 2010;
Huys et al., 2013; Safra et al., 2019). Similarly, escape and avoidance behaviors
associated with anxiety and stress reduce a person’s exposure to sources of
environmental reward (Grant & White, 2016; Harl´e et al., 2017).
Regarding the limitations of the study, a non-probabilistic sampling was used
for convenience which limits the generalization of the results. The metric quality
of the scale was not evaluated in a clinical sample. Despite these limitations, the
results of the study are important not only for psychometric research but also
for therapies based on behavioral activation.
In conclusion, this study represents a significant contribution to measurement
in the context of behavioral activation therapy as it will facilitate a rapid and
efficient assessment of the degree of environmental reward perceived by the
person, thus allowing for a systematic monitoring of progress in therapy. In
addition, psychometric evidence has been shown for the existence of a method’s
effect regarding negative items in the adaptations to Spanish. Finally, the results
of the study provide a statistical and methodological basis for the development
of the EROS scale in future psychometric studies. Further studies could use the
scale to evaluate the effectiveness of therapy in different mental health
problems.

Declaration of Conflicting Interests

The author(s) declare(s) that there is no conflict of interest with respect to the research,
authorship, and/or publication of this article.

Funding
The author(s) received no financial support for the research, authorship, and/or publi-
cation of this article.

ORCID iDs
Lindsey W. Vilca https://orcid.org/0000-0002-8537-9149
Robert I. Echebaudes-Ilizarbe https://orcid.org/0000-0002-1746-2530
Michael White https://orcid.org/0000-0003-4530-8167

References
Aiken, L. R. (1980). Content validity and reliability of single items or questionnaires.
Educational and Psychological Measurement, 40(4), 955–959.
https://doi.org/10.1177/ 001316448004000419
Aoki, S., Doi, S., Horiuchi, S., Takagaki, K., Kawamura, A., Umeno, R., Fujita, M.,
Kitagawa, N., & Sakano, Y. (2019). Mediating effect of environmental rewards on the
 Vilca et 2

relation between goal-directed behaviour and anhedonia [Mediating effect of environ-

mental rewards on the relation between goal-directed behaviour and anhedonia].
Current Psychology. https://doi.org/10.1007/s12144-019-00312-y
Armento, M. E. A., & Hopko, D. R. (2007). The environmental reward observation scale
(EROS): Development, validity, and reliability. Behavior Therapy, 38(2), 107–119.
https://doi.org/10.1016/j.beth.2006.05.003
Atienza, F. L., Pons, D., Balaguer, I., & Garc´ıa-Merita, M. (2000). Propiedades psi-
com´etricas de la escala de satisfaccio´n con la vida en adolescents [Psychometric
prop- erties of the life satisfaction scale in adolescents]. Psicothema, 12(2), 314–319.
https:// www.unioviedo.es/reunido/index.php/PST/article/view/7597
Bados, A., Solanas, A., & Andr´es, R. (2005). Psychometric properties of the
Spanish
version of depression, anxiety and stress scales (DASS). Psicothema, 17(4), 679–683.
http://www.psicothema.com/psicothema.asp?id = 3165
Barraca, J., & P´erez-A´ lvarez, M. (2010). Adaptacio´n espan~ola del environmental
reward
observation (EROS) [Translation results Spanish adaptation of the environmental
reward observation (EROS)]. Ansiedad y Estre´s, 16(1), 95–107.
https://dialnet.uni rioja.es/servlet/articulo?codigo = 3191048
Barraca, J., & P´erez-A´ lvarez, M. (2015). Activacio´n Conductual para el tratamiento
de la
depresio´n [Translation results behavioral activation for the treatment of
depression]. Editorial Sintesis S.A.
Barth, J., Munder, T., Gerger, H., Nu¨ esch, E., Trelle, S., Znoj, H., Ju¨ ni, P., & Cuijpers,
P. (2013). Comparative efficacy of seven psychotherapeutic interventions for patients
with depression: A network. PLoS Medicine, 10(5), e1001454. https://doi.org/10.
1371/journal.pmed.1001454
Brown, T. A. (2015). Confirmatory factor analysis for applied research (Vol. 43). The
Guilford Press.
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invari-
ance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504.
https://doi.org/10.1080/10705510701301834
Chopik, W. J., Brien, E. O., & Konrath, S. H. (2017). Differences in empathic concern
and perspective taking across 63 countries. Journal of Cross-Cultural Psychology,
48(1), 23–38. https://doi.org/10.1177/0022022116673910
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests.
Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis,
M. E., Gallop, R., McGlinchey, J. B., Markley, D. K., Gollan, J. K., Atkins, D. C.,
Dunner, D. L., & Jacobson, N. S. (2006). Randomized trial of behavioral activation,
cognitive therapy, and antidepressant medication in the acute treatment of adults with
major depression. Journal of Consulting and Clinical Psychology, 74(4), 658–670.
https://doi.org/10.1037/0022-006X.74.4.658
DiStefano, C., Liu, J., Jiang, N., & Shi, D. (2018). Examination of the weighted root
mean square residual: Evidence for trustworthiness? Structural Equation Modeling: A
Multidisciplinary Journal, 25(3), 453–466. https://doi.org/10.1080/10705511.2017.
1390394
Dominguez-Lara, S. (2016). Evaluacio´n de la confiabilidad del constructo
mediante el coeficiente H: Breve revisio´n conceptual y aplicaciones [Evaluation of
the reliability of
 2 Psychological Reports

the construct using the H coefficient: Brief conceptual review and applications].
Psychologia: Avances de La Disciplina, 10(2), 87–94. https://doi.org/10.21500/
19002386.2134
Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical
solution to the pervasive problem of internal consistency estimation. British Journal of
Psychology (London, England: 1953), 105(3), 399–412.
https://doi.org/10.1111/bjop.12046
Elosua, P., & Zumbo, B. D. (2008). Coeficientes de fiabilidad para escalas de respuesta
catego´rica ordenada. Psicothema, 20(4), 896–901. Retrieved from
http://www.psicoth ema.com/psicothema.asp?id=3572
Eshel, N., & Roiser, J. P. (2010). Reward and punishment processing in depression.
Biological Psychiatry, 68(2), 118–124. https://doi.org/10.1016/j.biopsych.2010.01.
027
Ferna´ ndez, N. E., & Mairal, B. J. (2017). Behavioral activation versus cognitive restruc-
turing to reduce automatic negative thoughts in anxiety generating situations.
Psicothema, 29(2), 172–177. https://doi.org/10.7334/psicothema2016.325
Ferna´ ndez-Rodr´ıguez, C., Gonza´ lez-Ferna´ ndez, S., Coto-Lesmes, R., & Pedrosa, I.
(2020). Behavioral activation and acceptance and commitment therapy in the treat-
ment of anxiety and depression in cancer survivors: A randomized clinical trial.
Behavior Modification. https://doi.org/10.1177/0145445520916441
Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical methods for rates and proportions
(3rd ed.). Wiley.
Gamer, M., Fellows, I., & Singh, P. (2019). Various coefficients of interrater reliability
and agreement. https://cran.r-project.org/web/packages/irr/irr.pdf
Gawrysiak, M., Nicholas, C., & Hopko, D. (2009). Behavioral activation for moderately
depressed university students: Randomized controlled trial. Journal of Counseling
Psychology, 56(3), 468–475. https://doi.org/10.1037/a0016383
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a
multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–
91. https://doi.org/10.1037/a0032138
Grant, D. M., & White, E. J. (2016). The influence of anxiety on cognitive control
processes. Oxford Research Encyclopedia of Psychology. https://doi.org/10.1093/acre
fore/9780190236557.013.74
Harl´e, K. M., Guo, D., Zhang, S., Paulus, M. P., & Yu, A. J. (2017). Anhedonia and
anxiety underlying depressive symptomatology have distinct effects on reward-based
decision-making. PLoS One, 12(10), e0186473. https://doi.org/10.1371/journal.pone.
0186473
Hayes, S. C. (2016). Acceptance and commitment therapy, relational frame theory, and
the third wave of behavioral and cognitive therapies. Behavior Therapy, 47(6), 869–
885. https://doi.org/10.1016/j.beth.2016.11.006
Heu, L. C., Zomeren, M., Van., & Hansen, N. (2019). Lonely alone or lonely together ? A
cultural-psychological examination of individualism—Collectivism and loneliness in
five European countries. Personality & Social Psychology Bulletin, 45(5), 780–793.
https://doi.org/10.1177/0146167218796793
Hirayama, T., Ogawa, Y., Yanai, Y., Suzuki, S., & Shimizu, K. (2019). Behavioral acti-
vation therapy for depression and anxiety in cancer patients: A case series study.
BioPsychoSocial Medicine, 13, 9. https://doi.org/10.1186/s13030-019-0151-6 (2019)
 Vilca et 2

Hofstede Insights. (2020). Country comparison. https://www.hofstede-insights.com/coun

try-comparison/
Hofstede, G., Hofstede, G. J., & Minkov, M. (2010). Cultures and organizations:
Software of the mind (3rd ed.). McGraw-Hill Education.
https://books.google.com.pe/books? id = o4OqTgV3V00C
Hopko, D. R., Armento, M. E. A., Robertson, S. M. C., Ryba, M. M., Carvalho, J. P.,
Colman, L. K., Mullane, C., Gawrysiak, M., Bell, J. L., McNulty, J. K., & Lejuez,
C. W. (2011). Brief behavioral activation and problem-solving therapy for depressed
breast cancer patients: Randomized trial. Journal of Consulting and Clinical
Psychology, 79(6), 834–849. https://doi.org/10.1037/a0025450
Huys, Q. J., Pizzagalli, D. A., Bogdan, R., & Dayan, P. (2013). Mapping anhedonia onto
reinforcement learning: A behavioural meta-analysis. Biology of Mood & Anxiety
Disorders, 3(1), 12–16. https://doi.org/10.1186/2045-5380-3-12
Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2018).
semTools: Useful tools for structural equation modeling. R package version 0.5–1.
https://cran.r-project.org/package = semTools
Kanter, J. W., Busch, A. M., Rusch, L. C. (2011). Activacio´n conductual: Refuerzos
positivos ante la depresio´n [Behavioral activation: Positive reinforcements for
depres- sion]. Alianza Editorial. https://books.google.com.pe/books?id =
zKj0ZwEACAAJ
Kern, D., Busch, A., Schneider, K. L., Miller, S. A., Appelhans, B. M., Waring, M. E.,
Whited, M. C., & Pagoto, S. (2019). Psychosocial factors associated with treatment
outcomes in women with obesity and major depressive disorder who received behav-
ioral activation for depression. Journal of Behavioral Medicine, 42(3), 522–533.
https:// doi.org/10.1007/s10865-018-9993-9
Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). The
Guilford Press.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass corre-
lation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2),
155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Lindwall, M., Barkoukis, V., Grano, C., Lucidi, F., Raudsepp, L., Liukkonen, J., &
Thøgersen-Ntoumani, C. (2012). Method effects: The problem with negatively
versus positively keyed items. Journal of Personality Assessment, 94(2), 196–204.
https://doi.org/10.1080/00223891.2011.645936
Lloret, S., Ferreres, A., Herna´ ndez, A., & Toma´ s, I. (2017). The exploratory
factor analysis of items: Guided analysis based on empirical data and software. Anales
de Psicolog´ıa, 33(2), 417–432. https://doi.org/10.6018/analesps.33.2.270211
Ly, K. H., Tru¨ schel, A., Jarl, L., Magnusson, S., Windahl, T., Johansson, R., Carlbring,
P., & Andersson, G. (2014). Behavioural activation versus mindfulness-based guided
self-help treatment administered through a smartphone application: A randomised
controlled trial. BMJ Open, 4(1), e003440. https://doi.org/10.1136/bmjopen-2013-
003440
Mahen, H. A. O., Moberly, N. J., & Wright, K. A. (2019). Trajectories of change in a
group behavioral activation treatment for severe, recurrent depression. Behavior
Therapy, 50(3), 504–514. https://doi.org/10.1016/j.beth.2018.08.007
Maitland, D. W. M., Neilson, E. C., Munoz, E. A., Ybanez, A., & Murray, A. L. (2019).
The impact of an enriched environment on the relationship between activation and
 2 Psychological Reports

depression in Latinx and Non-Latinx students. The Psychological Record, 69(4), 541–
550. https://doi.org/10.1007/s40732-019-00351-8
Martell, C. R., Dimidjian, S., & Herman-Dunn, R. (2013). Activacio´n conductual Para la
depresio´n: Una gu´ıa cl´ınica [Behavioral activation for depression: A clinical
guide]. Descl´ee De Brouwer.
Mart´ınez-Vispo, C., Senra, C., Lo´pez-Dura´ n, A., Ferna´ ndez del R´ıo, E., &
Becon~a, E. (2019). Boredom susceptibility as predictor of smoking cessation
outcomes: Sex differ- ences. Personality and Individual Differences, 146(March),
130–135. https://doi.org/10. 1016/j.paid.2019.03.026
Mazzucchelli, T. G., Kane, R. T., & Rees, C. S. (2010). Behavioral activation interven-
tions for well-being: A meta-analysis. The Journal of Positive Psychology, 5(2), 105–
121. https://doi.org/10.1080/17439760903569154
Mazzucchelli, T. G., Kane, R., & Rees, C. (2009). Behavioral activation treatments for
depression in adults: A meta-analysis and review. Clinical Psychology: Science and
Practice, 16(4), 383–411. https://doi.org/10.1111/j.1468-2850.2009.01178.x
McCoach, D. B., Gable, R. K., & Madura, J. P. (2013). Evidence based on relations to
other variables: Bolstering the empirical validity arguments for constructs. In D. B.
McCoach, R. K. Gable, & J. P. Madura (Eds.), Instrument development in the affective
domain (pp. 209–248). Springer. https://doi.org/10.1007/978-1-4614-7135-6
McDonald, R. P. (1999). Test theory: A unified treatment. Taylor & Francis.
Mira, A., D´ıaz-Garc´ıa, A., Castilla, D., Campos, D., Romero, S., Breto´n-Lo´pez,
J., & Botella, C. (2019). An internet-based intervention for depressive symptoms:
Preliminary data on the contribution of behavioral activation and positive psycho-
therapy strategies. Lecture Notes of the Institute for Computer Sciences, Social-
Informatics and Telecommunications Engineering, LNICST, 19(133), 129–146.
https://doi.org/10.1007/978-3-030-25872-6_10
Mueller, R. O., & Hancock, G. R. (2001). Rethinking construct reliability within latent
variable systems. In Structural equation modeling: Past and present. A festschrift in
honor of Karl G. Jo€reskog (pp. 195–261). Scientific Software International.
Oliver, A., Galiana, L., & Bustos, V. (2018). Validacio´n de la escala de satisfaccio´n con la
vida y su relacio´n con las dimensiones de autoconcepto en universitarios
peruanos [Translation results Validation of the life satisfaction scale and its
relationship with the dimensions of self-concept in Peruvian university students].
Persona, 2(021), 29–44. https://doi.org/10.26439/persona2018.n021.3018
Otero, P., Hita, I., Torres, A´ . J., & Va´ zquez, F. L. (2020). Brief psychological interven-
tion through mobile app and conference calls for the prevention of depression in non-
professional caregivers: A pilot study. International Journal of Environmental Research
and Public Health, 17(12), 4578–4515. https://doi.org/10.3390/ijerph17124578
Padro´s, F, F., Guti´errez, C. Y, C. Y., & Medina, M. A, M. A. (2015).
Propiedades
psicom´etricas de la escala de satisfaccio´n con la vida (SWLS) de diener en
poblacio´n de michoaca´ n (m´exico) [Psychometric properties of Diener’s life
satisfaction scale (SWLS) in a population of Michoaca´ n (Mexico)].
Avances En Psicolog´ıa Latinoamericana, 33(2), 223–230.
https://doi.org/10.12804/apl33.02.2015.04
Patias, N. D., Machado, W. L., Bandeira, D. R., & Dell´Aglio, D. D. (2016). Depression
anxiety and stress scale (DASS-21)—Short form: Adaptac¸ ~ao e validac¸ ~ao Para
ado- lescentes brasileiros [Depression anxiety and stress scale (DASS-21)—Short
 Vilca et 2

form: Adaptation and validation For Brazilian adolescents]. Psico-USF, 21(3), 459–
469. https://doi.org/10.1590/1413-82712016210302
P´erez-Alvares, M. (2014). Las terapias de tercera generacio´n como terapias
contextuales
[Third-generation therapies as contextual therapies]. Editorial Sintesis S.A.
R Core Team. (2019). A language and environment for statistical computing. R
Foundation for Statistical Computing. http://www.r-project.org/
Raykov, T., & Hancock, G. R. (2005). Examining change in maximal reliability
for multiple-component measuring instruments. The British Journal of
Mathematical and Statistical Psychology, 58(Pt 1), 65–82. https://doi.org/10.1348/
000711005X38753
Richards, D. A., Ekers, D., McMillan, D., Taylor, R. S., Byford, S., Warren, F. C.,
Barrett, B., Farrand, P. A., Gilbody, S., Kuyken, W., O’Mahen, H., Watkins, E. R.,
Wright, K. A., Hollon, S. D., Reed, N., Rhodes, S., Fletcher, E., & Finning, K.
(2016). Cost and outcome of behavioural activation versus cognitive behavioural
therapy for depression (COBRA): A randomised, controlled, non-inferiority trial.
The Lancet, 388(10047), 871–880. https://doi.org/10.1016/S0140-6736(16)31140-0
Rosseel, Y. (2012). Lavaan : An R package for structural equation modeling. Journal of
Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
RStudio Team. (2018). RStudio: Integrated Development Environment for R. RStudio,
Inc. http://www.rstudio.com/
Ryba, M. M., & Hopko, D. R. (2012). Gender differences in depression : Assessing
mediational effects of overt behaviors and environmental reward through daily
diary monitoring. Depression Research and Treatment, 2012, 865679. https://doi.org/
10.1155/2012/865679
Safra, L., Chevallier, C., & Palminteri, S. (2019). Depressive symptoms are associated
with blunted reward learning in social contexts. PLoS Computational Biology, 15(7),
e1007224. https://doi.org/10.1371/journal.pcbi.1007224
Saris, W. E., Satorra, A., & van der Veld, W. M. (2009). Testing structural equation
models or detection of misspecifications? Structural Equation Modeling: A
Multidisciplinary Journal, 16(4), 561–582. https://doi.org/10.1080/10705510903203433
Savalei, V., & Falk, C. F. (2014). Recovering substantive factor loadings in the presence
of acquiescence bias: A comparison of three approaches. Multivariate Behavioral
Research, 49(5), 407–424. https://doi.org/10.1080/00273171.2014.931800
Schumacker, R. E., & Lomax, R. G. (2015). A beginner’s guide to structural equation
modeling: Fourth edition (4th ed.). Routledge.
Simmonds-Buckley, M., Kellett, S., & Waller, G. (2019). Acceptability and efficacy of
group behavioral activation for depression among adults: A meta-analysis. Behavior
Therapy, 50(5), 864–885. https://doi.org/10.1016/j.beth.2019.01.003
Toma´ s, J. M., Oliver, A., Galiana, L., Sancho, P., & Lila, M. (2013). Explaining method
effects associated with negatively worded items in trait and state global and domain-
specific self-esteem scales. Structural Equation Modeling: A Multidisciplinary
Journal, 20(2), 299–313. https://doi.org/10.1080/10705511.2013.769394
Tull, M. T., Gratz, K. L., Latzman, R. D., Kimbrel, N. A., & Lejuez, C. W. (2010).
Reinforcement sensitivity theory and emotion regulation difficulties : A multimodal
investigation. Personality and Individual Differences, 49(8), 989–994. https://doi.org/
10.1016/j.paid.2010.08.010
 2 Psychological Reports

Valderrama-D´ıaz, M. A., Bianchi-Salguero, J. M., & Villalba-Garzo´n, J. A.

(2016). Validacio´n de la environmental reward observation scale (EROS) en
poblacio´n colombiana [Validation of the environmental reward observation scale
(EROS) in Colombian population]. Universitas Psychologica, 15(4).
https://doi.org/10.11144/ Javeriana.upsy15-4.vero
van Sonderen, E., Sanderman, R., & Coyne, J. C. (2013). Ineffectiveness of reverse word-
ing of questionnaire items: Let’s learn from cows in the rain. PLoS One, 8(7), e68967.
https://doi.org/10.1371/journal.pone.0068967
Va´ zquez, F. L., Lo´pez, L., Blanco, V., Otero, P., Torres, A´ . J., & Ferraces, M. J.
(2019).
The impact of decreased environmental reward in predicting depression severity in
caregivers. Annals of Psychology, 35(3), 357–363. https://doi.org/10.6018/analesps.35.
3.329131
Va´ zquez, F. L., Torres, A´ . J., Otero, P., Blanco, V., Lo´pez, L., Garc´ıa-Casal,
A., &
Arrojo, M. (2019). Cognitive-behavioral intervention via interactive multimedia
online video game for active aging: Study protocol for a randomized controlled
trial. Trials, 20(1), 1–15. https://doi.org/10.1186/s13063-019-3859-5
Ventura-Leo´n, J. (2019). Back to content-based validity. Adicciones, 0(0). https://doi.org/
10.20882/adicciones.1213
Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A journey around alpha and
omega to estimate internal consistency reliability. Anales de Psicolog´ıa,
33(3), 755–782. https://doi.org/10.6018/analesps.33.3.268401
Wagener, A., & Blairy, S. (2015). Validation and psychometric properties of the French
versions of the environmental reward observation scale and of the reward probability
index. Psychologica Belgica, 55(2), 71–86. https://doi.org/10.5334/pb.bg
Wang, K., Shi, H.-S., Geng, F.-L., Zou, L.-Q., Tan, S.-P., Wang, Y., Neumann, D. L.,
Shum, D. H. K., & Chan, R. C. K. (2016). Cross-cultural validation of the depression
anxiety stress. Psychological Assessment, 28(5), e88–e100. https://doi.org/10.1037/
pas0000207
Watkins, M. W. (2018). Exploratory factor analysis: A guide to best practice. Journal of
Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807
Watson, J. C. (2017). Development establishing evidence for internal structure using
exploratory factor analysis factor analysis. Measurement and Evaluation in
Counseling and Development, 50(4), 232–238. https://doi.org/10.1080/07481756.2017.
1336931
Wolf, E. j., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample size require-
ments for structural equation models: An evaluation of power, bias, and solution
propriety. Educational and Psychological Measurement, 76(6), 913–934. https://doi.
org/10.1177/0013164413495237
World Medical Association. (2013). World medical association declaration of Helsinki
ethical principles for medical research involving human subjects. Clinical Review &
Education Special, 27(20), 2191–2194. https://doi.org/10.1001/jama.292.11.1359
Yang, Y., & Green, S. B. (2010). A note on structural equation modeling estimates of
reliability. Structural Equation Modeling: A Multidisciplinary Journal, 17(1), 66–81.
https://doi.org/10.1080/10705510903438963
Zabihi, S., Lemmel, F. K., & Orgeta, V. (2020). Behavioural activation for depression in
informal caregivers: A systematic review and meta-analysis of randomised controlled
 Vilca et 2

clinical trials. Journal of Affective Disorders, 274, 1173–1183.

https://doi.org/10.1016/j. jad.2020.03.124
Zhang, X., Noor, R., & Savalei, V. (2016). Examining the effect of reverse worded items
on the factor structure of the need for cognition scale. PLoS One, 11(6), e0157795.
https://doi.org/10.1371/journal.pone.0157795
Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s, a revelle’s b and
McDonald’s x H: Their relations with each other and two alternative conceptualiza-
tions of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-
003-0974-7

Author Biographies
Lindsey W. Vilca, Mg, Professor and Researcher at the Universidad Peruana
Union in Lima, Peru. He is a member of the National System of Researchers,
level Mar´ıa Rostworowski I of the National Council of Science, Technology
and Technological Innovation (Peruvian Government).

Robert I. Echebaudes-Ilizarbe, Psychologist graduated from Universidad

Peruana Union. He is a member of the Scientific Society of Psychology
Students.

Jannia M. Aquino-Hidalgo, Student of Psychology at the Universidad Peruana

Union in Lima, Peru. She is president of the Scientific Society of Psychology
Students.

Jos´e Ventura-Le´on, PhD, professor in the School of Psychology at

the Universidad Privada del Norte. He is a member of the National
System of Research, level Carlos Monge II of the National Counceil for
Sciencie and Technology (Peruvian Government).

Renzo Martinez-Munive, PhD, professor in psychology at the Universidad

Privada del Norte in Lima, Peru. He is a member of the Association for
Contextual Behavioral Science (ACBS).

Michael White, M.Div, Head of Scientific Publications and Translations at

the Universidad Peruana Unio´n in Lima, Peru.
2 Psychological Reports
View publication stats