See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/275835037

Effectiveness of Internet-Based Affect Induction Procedures: A Systematic

Review and Meta-Analysis

Article  in  Emotion · May 2015

DOI: 10.1037/emo0000035 · Source: PubMed

CITATIONS READS
35 1,433

3 authors:

Rebecca A Ferrer Emily Grenen

National Institutes of Health London School of Hygiene and Tropical Medicine
138 PUBLICATIONS   3,216 CITATIONS    7 PUBLICATIONS   87 CITATIONS   

SEE PROFILE SEE PROFILE

Jennifer M. Taber
Kent State University
73 PUBLICATIONS   936 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Rebecca A Ferrer on 04 November 2015.

The user has requested enhancement of the downloaded file.

Emotion
Effectiveness of Internet-Based Affect Induction
Procedures: A Systematic Review and Meta-Analysis
Rebecca A. Ferrer, Emily G. Grenen, and Jennifer M. Taber
Online First Publication, May 4, 2015. http://dx.doi.org/10.1037/emo0000035

CITATION
Ferrer, R. A., Grenen, E. G., & Taber, J. M. (2015, May 4). Effectiveness of Internet-Based
Affect Induction Procedures: A Systematic Review and Meta-Analysis. Emotion. Advance
online publication. http://dx.doi.org/10.1037/emo0000035
Emotion In the public domain
2015, Vol. 15, No. 2, 000 http://dx.doi.org/10.1037/emo0000035

Effectiveness of Internet-Based Affect Induction Procedures: A Systematic

Review and Meta-Analysis
Rebecca A. Ferrer, Emily G. Grenen, and Jennifer M. Taber
National Cancer Institute/National Institutes of Health

Procedures used to induce affect in a laboratory are effective and well-validated. Given recent method-
ological and technological advances in Internet research, it is important to determine whether affect can
be effectively induced using Internet methodology. We conducted a meta-analysis and systematic review
of prior research that has used Internet-based affect induction procedures, and examined potential
moderators of the effectiveness of affect induction procedures. Twenty-six studies were included in final
analyses, with 89 independent effect sizes. Affect induction procedures effectively induced general
positive affect, general negative affect, fear, disgust, anger, sadness, and guilt, but did not significantly
induce happiness. Contamination of other nontarget affect did not appear to be a major concern. Video
inductions resulted in greater effect sizes. Overall, results indicate that affect can be effectively induced
in Internet studies, suggesting an important venue for the acceleration of affective science.

Keywords: emotion induction, mood induction, Internet experiments, meta-analysis

Supplemental materials: http://dx.doi.org/10.1037/emo0000035.supp

Hundreds of studies have effectively induced affective or emo-
tional states among human participants in experimental psycho-
logical studies (Lench et al., 2011; Westermann et al., 1996).
These studies have resulted in important discoveries about the role
of emotion for judgment and decision-making, sensation and per-
ception, interpersonal attribution and interaction, and behavioral
action tendencies. Such studies have been conducted primarily in
controlled laboratory settings, and the effectiveness of different
affect inductions in the laboratory has been validated through
careful analysis (Lench et al., 2011; Westermann et al., 1996).
In recent years, methodological and technological advances in
Internet research have shifted the focus from the laboratory as the
sole, or even most desirable, venue for experimental psychology
research. Indeed, experimental studies may be among the most
common type of psychological research that takes place online
(Skitka & Sargis, 2006). This shift has the potential to revolution-
ize and positively disrupt the field of experimental psychology.
The advent of readily available Internet panels, in which people
who sign up to be in a pool of individuals selected to complete
Rebecca A. Ferrer, Basic Biobehavioral and Psychological Sciences
Branch, Behavioral Research Program, Division of Cancer Control and
Population Sciences, National Cancer Institute/National Institutes of
Health; Emily G. Grenen and Jennifer M. Taber, Behavioral Research
Program, Division of Cancer Control and Population Sciences, National
Cancer Institute/National Institutes of Health.
Emily G. Grenen is now at ICF International.
Correspondence concerning this article should be addressed to Rebecca
A. Ferrer, Basic Biobehavioral and Psychological Sciences Branch, Be-
havioral Research Program, Division of Cancer Control and Population
Sciences, National Cancer Institute/National Institutes of Health, Rock-
ville, MD 20850. E-mail: ferrerra@mail.nih.gov
online studies are paid small sums (e.g., $0.10 –$1) for answering
a question online (for example, Amazon mTurk [https://www.mturk
.com/mturk/welcome], Survey Monkey Audience [https://www
.surveymonkey.com/mp/audience], and GfK Knowledge Panel),
have exponentially accelerated the number of experiments con-
ducted online (Berinsky, Huber, & Lenz, 2012; Buhrmester,
Kwang, & Gosling, 2011; Horton, Rand, & Zeckhauser, 2011;
Paolacci & Chandler, 2014). In parallel with this acceleration,
software specific to conducting Internet research has been devel-
oped by the research community (e.g., psiTurk 2.0) and industry
(for example, Qualtrics [http://www.qualtrics.com/], Survey Mon-
key). Moreover, the growth of Internet research is reflected in
conference workshops specifically aimed at identifying advances,
challenges, and opportunities in experimental psychology con-
ducted via the Internet (e.g., Suri, Mason, & Goldstein, 2014).
Given the advent of online research, it is important to understand
whether emotion can be effectively induced using Internet meth-
odology.
Internet-based experimental studies have several advantages.
Internet samples tend to be more representative of the general
public on characteristics such as age, income, and education than
student or convenience samples (Berinsky et al., 2012; Paolacci,
Chandler, & Ipeirotis, 2010; Ross et al., 2010), perhaps because
they can reach more geographically and otherwise diverse popu-
lations (Birnbaum, 2004; Reips, 2000). This is particularly true
when complex sampling strategies are used to yield nationally
representative samples (e.g., providing computer and Internet ac-
cess by companies such as GfK), but is even the case when the
sample is an Internet convenience sample or targeted population
(Birnbaum, 2004; Reips, 2000). Moreover, the lower costs and
greater efficiency of online experimentation allow more junior
researchers and faculty at smaller or less resourced universities to
conduct research and contribute to psychological scholarship (Hor-
ton & Chilton, 2010; Kraut et al., 2004). In contrast to traditional

1
2 FERRER, GRENEN, AND TABER
laboratory settings where the cost of experiment administration
can be quite high, most online studies are automated to perform the
activities that would normally require human labor: recruitment,
instruction delivery, experimental manipulation, and data collec-
tion. Moreover, the relative anonymity of the Internet can lead to
higher response rates when questions are perceived to be sensitive
(Birnbaum, 2004; Skitka & Sargis, 2006). Internet surveys are also
lower in measurement error and social desirability bias (Skitka &
Sargis, 2005, 2006), and online study participants have reported
being intrinsically motivated to engage in studies (Ipeirotis, 2010;
Paolacci et al., 2010; Ross et al., 2010).
Despite these well-demonstrated advantages, there have been
concerns that experimental designs administered over the Internet
lack control and precision (see Kraut et al., 2004; Skitka & Sargis,
2006). In a laboratory, researchers can control the research envi-
ronment and paradigm administration. Thus, even experiments
conducted on a laboratory computer may be fundamentally differ-
ent than experiments conducted via the Internet, given that re-
searchers cannot control for elements in the participant’s physical
environment (e.g., noise, lighting, or other people), and as such can
have less confidence that participants are fully engaged in exper-
imental paradigms. Furthermore, in the laboratory, an experi-
menter can monitor participants’ behavior to ensure that they are
taking the task seriously and understanding directions. However,
evidence suggests that Internet studies can replicate the results of
laboratory paradigms (Berinsky et al., 2012; Krantz & Dalal,
2000), although it is unknown whether the specific strategy of
inducing emotion or manipulating participants’ emotional or af-
fective state can be effectively reproduced over the Internet.
Given the critical importance of carefully considering method-
ological differences of Internet research compared with laboratory
studies (Mathy, Kerr, & Haydin, 2003), one primary goal of this
meta-analysis is to identify methods of induction that are particu-
larly promising for inducing affective states in Internet studies.
Thus, we defined and coded inductions based on categories that
have been examined in larger meta-analyses synthesizing primar-
ily laboratory studies (e.g., Lench et al., 2011; Westermann et al.,
1996).
We also sought to examine whether certain types of affect are
more difficult to induce over the Internet. Positive affect is typi-
cally more difficult to induce than negative affect (Augustine &
Hemenover, 2009; Ito et al., 1998), and we expected this finding to
replicate with Internet inductions. Finally, because of concerns
over experimental control in Internet-based experimental manipu-
lations (see Kraut et al., 2004; Skitka & Sargis, 2006), we exam-
ined whether affective states could be induced over the Internet
without contamination of other affective states—that is, whether
an induction targeting one affective state induces only that affec-
tive state, but not others. If Internet-based inductions can have
sufficient experimental control and are effective, they should ex-
hibit significant overall effect sizes for target, but not nontarget,
affective states.
Method
We followed consensus guidelines for conducting and reporting
meta-analyses of observational studies in epidemiology (Stroup et
al., 2000), as appropriate and adapted to the current meta-analysis
of experimental psychological research (e.g., omitting recommen-
dations specific to epidemiology studies, such as the evaluation of
the comparability of cases and controls).
Search Strategy
Relevant publications, in press manuscripts, dissertations, post-
ers presented at conferences, and raw data were identified through:
(a) searches of electronic databases (PsycINFO, PsycARTICLES);
(b) email requests to professional organization listservs (Society
for Judgment and Decision Making, Society for Personality and
Social Psychology, Social Personality and Health Network, Soci-
ety for Medical Decision Making); (c) searching reference lists
from reviews or published studies; (d) emails to authors who have
published in the field; and (e) a search of publicly available data on
the Time Sharing Experiments in the Social Sciences Website
(TESS; http://www.tess.org). The keywords used for the initial
database search were combinations of the terms [emotion; “posi-
tive affect”; “negative affect”; affective; mood; stress; sadⴱ; hapⴱ;
fear; anxⴱ; angⴱ; gratⴱ; prideⴱ; disgustⴱ], [online; mturk; amazon;
“knowledge networks”; Web; Internet], and [manipⴱ; inductⴱ; elic-
itⴱ; influenⴱ].
Studies were included if they fit all of the following criteria: (a)
took place entirely online in a nonlaboratory setting; (b) included
experimental inductions of discrete emotions, mood, or stress; (c)
contained manipulation checks that assessed participants’ self-
reported affective state after the induction; and (d) included a
comparison condition (between-subjects design).1 All articles lo-
cated in the search were published in English. Authors were
contacted directly when studies did not include sufficient statistical
information to calculate effect size. Two studies were excluded
because the authors were unable to provide the necessary infor-
mation. Studies that fulfilled the above criteria as of May 1, 2014,
were included; the final sample included 26 studies, for which 89
independent effect sizes could be extracted. Figure 1 contains the
PRISMA diagram of the manuscript search, retrieval, and coding
process (Moher, Liberati, Tetzlaff, & Altman, 2009).
Coding and Reliability
Studies were coded for (a) publication status (i.e., published
journal article vs. dissertation or unpublished data); (b) year of
publication (or year study was conducted, if unpublished); (c)
academic department of first author; (d) explicit description of the
theoretical framework underlying study design; (e) the behavioral
or disciplinary domain and context of the study (e.g., financial,
health, risk perception, or valuation); (f) reporting of use of a cover
story to blind participants to affective condition; and (g) reporting
of participant randomization to experimental condition. We also
coded characteristics of the participants in each sample, including:
(a) mean age; (b) percent female; (c) percent White; (d) percent of
sample currently living in the United States; (e) percent with some
1
One between-subjects study with no neutral group and no clear com-
parisons was excluded (Egermann & McAdams, 2013); in this study, there
were 24 musical affect induction conditions, but no explicit hypotheses
about which conditions should be different from one another or whether
each musical selection should induce positive or negative affect. As such,
we were unable to determine which conditions would be expected to differ
and how, and including it in analyses could introduce noise in effect sizes.
 INTERNET-BASED AFFECT INDUCTION PROCEDURES
Potentially relevant study titles
identified by searches and
screened for relevance (3,741
studies)
Abstracts read/ data examined for
more detailed evaluation (900
studies)
Studies obtained for more
detailed evaluation (412 studies)
Potentially appropriate studies to
be included in meta-analysis (58
studies)
Studies in final meta-analysis
(26 studies; k = 89)
Figure 1. PRISMA diagram of study selection.
college education or higher; and (f) population (e.g., Internet
panel, student sample).
The affective induction procedure in each study was coded for:
(a) the affective state targeted in the experimental condition(s) as
specified by the study authors; (b) the affective state targeted in the
experimental condition(s) as specified based on coder judgment;
(c) the affective state targeted in the comparison condition (e.g.,
neutral affect) as specified by the study authors; (d) the affective
state targeted in the comparison condition as specified based on
coder judgment; (e) type of affect induction (e.g., autobiographical
recall); and (f) type of comparison induction.
Induction categories were selected so that they could be com-
pared with previous meta-analyses of affect induction and mood
(e.g., Lench et al., 2011; Westermann et al., 1996). Categories
were not always mutually exclusive; for example, one study used
pictures and text inductions simultaneously (e.g., Boudewyns et
al., 2013).
We identified a variety of techniques used to induce affect, and
describe each in turn. A novel category we identified was supra-
liminal word presentation, in which participants are exposed to a
series of positive or negative affectively laden words. Word pre-
sentation is supraliminal, which distinguishes it from subliminal
priming in previous meta-analyses. In autobiographical recall,
participants are instructed to write in detail about an experience
3
Studies excluded (2,841 studies)
Reasons: Search term “online” referred to brain
processes/ related behaviors
(neuroscience)
Studies involved clinical samples with
severe psychopathy
Studies excluded (488 studies)
Reasons: No affect induction
Study took place in laboratory
No comparison condition
Induction part of larger intervention
Studies excluded (354 studies)
Reasons: No affect induction
Study took place in laboratory
No comparison condition
Studies excluded (32 studies)
Reasons: No manipulation check
Inadequate statistical information
that made them feel particularly strongly on a specified affective
state. Such inductions often involve instructing participants to list
3–5 events that made them feel most emotional before choosing
one to write about (e.g., Lerner & Keltner, 2001; Strack, Schwarz,
& Gschneidinger, 1985). Video elicitations involve presenting a
film clip, usually excerpted from a feature length film and often
selected from a standardized set of clips (Gross & Levenson,
1995).
Picture inductions involve presenting previously validated (and
often taken from standard databases) affectively laden images
either supraliminally or subliminally (e.g., International Affective
Pictures System [http://csea.phhp.ufl.edu/media.html]; Bradley &
Lang, 1999). All studies identified here used supraliminal presen-
tation. Real-life inductions involve using procedures that are os-
tensibly outside study protocols to influence affect. In Internet
research, real-life affect inductions often involve manipulating the
Web site interface in such a way that it is not meant to be
obviously part of the experiment. For example, such inductions
could involve purposely degrading the usability of the Web site
interface to induce negative mood (Hängii, 2004). Text inductions
involve presenting paragraphs of text with affectively laden con-
tent, such as newspaper articles, jokes, or portions of textbooks or
articles. Velten mood induction procedures involve instructing
participants to put themselves in a target mood state, and to
4 FERRER, GRENEN, AND TABER
subsequently read a series of self-referent statements (Velten,
1968).
Each study was coded by the lead author (RF). A second
coder (EG) independently coded 47 effect sizes (68.12% of k).
Rater agreement was high (percent agreement ⬎93.62, ␬s ⬎
.48). Discrepancies were resolved through discussion among
the coders. The coding form can be obtained from the corre-
sponding author.
From these coding categories, we further coded: (a) whether the
affective state targeted was positively or negatively valenced; (b)
whether the affective state targeted was a discrete emotion or
mood state; (c) whether the author and coder ratings of affective
state targeted were discrepant; and (d) whether the author and
coder ratings of comparison affective state targeted were discrep-
ant.
Study Outcomes and Effect Sizes
Effect sizes were calculated based on the primary manipulation
check item or scale (target affective state). Effect sizes were also
calculated for any secondary affective states assessed. A conser-
vative approach to quantifying contamination to other nontarget
affective states would take into account the nontarget affective
state most affected by the induction. Therefore, we calculated one
secondary outcome using the highest effect size for nontarget
affective state (highest nontarget affective state), as well as sec-
ondary outcomes for each specific nontarget affective state.
Effect sizes (Cohen’s d) were calculated as the mean differences
between the experimental and comparison condition divided by the
pooled SD. If means and SDs were not included in the publication,
other statistical information (e.g., t- or F values) was used to
calculate d (Johnson & Eagly, 2000; Lipsey & Wilson, 2001). A
positive effect size indicated an increase in the target affect com-
pared with the comparison condition (in the one study that targeted
a decrease in emotion, a positive effect size indicated a decrease of
the target emotion). To correct for sample size bias, effect sizes
were weighted by the inverse of the variance of d (Hedges, 1981).
When a neutral condition was included in the design, effect sizes
were calculated by comparing each experimental condition to the
neutral condition. When more than one neutral condition was
included, the condition that most closely resembled the experimen-
tal induction was used for comparison (e.g., a neutral video would
be selected to compare with a video induction instead of a no
treatment control when both were included).
When studies included multiple conditions and no neutral com-
parison condition, effect sizes were calculated based on the most
appropriate comparison. For example, in a study with the follow-
ing four conditions: (a) positive affect induced with a Velten
procedure; (b) positive affect induced with a picture induction; (c)
negative affect induced with a Velten procedure; and (d) negative
affect induced with a picture induction; Conditions 1 and 3 were
compared, as were Conditions 2 and 4. When a study included no
neutral comparison condition and no clearly delineated compari-
sons (as described), we calculated effect sizes for comparisons
between each combination of experimental conditions.
To prevent interdependence of effect sizes included in the same
analysis, when multiple comparisons in the study shared a com-
parison condition (e.g., an experiment had an anger, sadness, and
neutral condition, and we compared both anger and sadness to
neutral) we divided the size of the comparison condition n in-
cluded in calculations for each comparison by the number of
comparisons in which the comparison condition’s data were used
(Borenstein et al., 2009). In subgroup analyses, we recalculated
effect sizes to include the full n for comparison or experimental
groups, even if those n were divided in main analyses (e.g., when
an experiment had an anger, sadness, and neutral condition, the
neutral condition n was halved in calculations for main analyses,
but was not divided in effect size calculations used in subgroup
analyses of studies with sadness inductions).
Previous meta-analyses examining the effectiveness of affect
inductions have handled the presence of nonneutral comparison
conditions differently. Westerman and colleagues (1996) excluded
studies without a neutral comparison condition, given that changes
in affective state postinduction cannot be isolated to an induction
of a specific affective state (i.e., changes could be either because
the experimental induction increased the target affective state, or
because the comparison condition reduced that affective state). As
such, our primary analyses focus on studies that include a neutral
comparison condition (Neutral Comparison Sample). However,
consistent with the approach of Lench and colleagues (2011),
because the number of experiments inducing affective states via
the Internet was limited, we also report supplementary analyses
that capitalize on the full sample of studies (Full Sample).
From the 26 studies included in the meta-analysis, 89 indepen-
dent comparisons (effect sizes, or k) of the main dependent vari-
able (target affective state) were calculated. Of these, 59 effect
sizes involved a neutral comparison condition (as opposed to
another affective state comparison).
Analysis Strategy
The Q statistic was calculated to examine whether effect sizes
were homogenous (i.e., whether all studies produced a statistically
equitable effect size for the comparison of experimental condition
to comparison condition on the target affective state manipulation
check). Because heterogeneity in effect sizes was detected, and to
generalize our findings, random-effects procedures were used
throughout (Hedges & Vevea, 1998).
All analyses were performed using Comprehensive Meta-
analysis (Biostat, 2014). First, we examined main effects of ex-
perimental inductions across studies by examining the overall
effect size. Separate analyses were conducted on each outcome
(target affective state, highest nontarget affective state, and each
affective state for which a manipulation check was included).
These were conducted within the Neutral Comparison Sample and
the Full Sample. Main effects analyses were also conducted within
each subsample targeting a specific affective state (e.g., anger)
when there were at least three studies targeting that affective state.
Similarly, main effects analyses were conducted within each sub-
sample targeting a specific affective state for highest nontarget
affective state effect size when there were at least three studies
targeting that affective state.
Because previous meta-analyses of emotion inductions reported
g (Lench et al., 2011) or r (Westermann et al., 1996), we also
calculated these statistics to facilitate direct comparisons between
our effects and those previously reported. Specifically, we calcu-
lated both g and r for the overall effect size in both the Full Sample
and Neutral Comparison Sample. We also calculated g in main
 INTERNET-BASED AFFECT INDUCTION PROCEDURES
effects analyses among studies targeting happiness, sadness, anger,
and fear, consistent with Lench and colleagues (2011), and r for
studies targeting general positive affect and general negative af-
fect, consistent with Westermann and colleagues (1996).
Then, we examined whether coded factors moderated the mag-
nitude of the effect size, using metaregression techniques. Within
each sample of studies, and for each outcome, each coded factor
was entered into a univariate metaregression (supplemental mate-
rials Appendix, Tables S2 and S3). Then (again within each
sample of studies and for each outcome), those factors that were
significant predictors of effect size magnitude at the univariate
level were simultaneously entered into comprehensive metaregres-
sions. Those coded factors that remained significant when entered
simultaneously were retained in the final models, while those that
became nonsignificant were trimmed from the final models.
Metaregressions were undertaken within each subsample targeting
a specific affective state when there were at least 10 studies that
could contribute to a given analysis.
Results
Overview of Included Studies
Included studies are denoted with an asterisk in the reference
section, and specific details about each study are included in the
supplementary online materials (supplemental materials Appen-
dix, Table S1). Studies were published or conducted from 2004
through 2014. Of the 26 studies included in the meta-analysis, 15
(57.7%) were published; the remainder were dissertations or un-
published data. Of the 89 independent effect sizes, 61 (68.5%)
came from published studies. Samples were roughly equally pro-
portioned in gender (54.4% female across studies). Mean age
across studies was 31.12 (SD ⫽ 8.17). The majority of studies
were first-authored by researchers housed in psychology depart-
ments (57.7%). Studies predominantly included samples residing
in the United States at the time of participation (73.1%). Only 22%
of studies targeted a student sample; 8.7% included a nationally
representative sample. The proportion of independent effect sizes
involving a given emotion target were as follows: general positive
affect (24.7%); general negative affect (22.5%); anger (18.0%);
disgust (4.5%); sadness (9.0%); guilt (7.9%); shame (1.1%); fear
(9.0%); hope (1.1%); and happiness (4.5%).
Analyses of Homogeneity
The overall effect sizes for the affective state induced in the
experimental conditions were heterogeneous for both the full sam-
ple: Q (88) ⫽ 1899.27, p ⬍ .0001; ␶2 ⫽ .55, I2 ⫽ 95.37, and the
neutral comparison sample: Q (58) ⫽ 1738.23, p ⬍ .0001; ␶2 ⫽
.67, I2 ⫽ 96.66. The overall effect size for the nontarget affective
state postinduction was also heterogeneous for both the full sample
of studies: Q (46) ⫽ 405.97, p ⬍ .0001; ␶2 ⫽ .20, I2 ⫽ 88.67; and
the neutral comparison sample: Q (25) ⫽ 78.105, p ⬍ .0001; ␶2 ⫽
.05, I2 ⫽ 67.99. These heterogeneous effect sizes support the
strategy of identifying moderators of the effect size magnitude.
Main Effects
Target affective state. The overall effect size for the target
affective state was significant across analyses: full sample: d ⫽
5
.862, 95% (confidence interval) CI ⫽ .696, 1.027, p ⬍ .0001, g ⫽
.822, 95% CI ⫽ .601, 1.04, r ⫽ .317; neutral comparison sample:
d ⫽ .828, 95% CI ⫽ .611, 1.044, p ⬍ .0001, g ⫽ .850, 95% CI ⫽
.687, 1.105, r ⫽ .355. These represent uniformly large effect sizes
in accordance with standard interpretation of Cohen’s (1988) d.
Main effects analyses stratified by target affective state (see
Table 1) indicated that, with the exception of happiness inductions,
inductions targeting all specific affective states were effective,
producing significant effect sizes that ranged from medium (pos-
itive affect, fear) to large (negative affect, disgust, anger, sadness,
or guilt). To facilitate comparisons to previous meta-analyses
(Lench et al., 2011; Westermann et al., 1996), Cohen’s g for
relevant stratified analyses were: happiness g ⫽ .284, 95% CI ⫽
.099, .468; sadness g ⫽ .851, 95% CI ⫽ .728, .974; anger g ⫽
.917, 95% CI ⫽ .484, 1.35; fear g ⫽ .643, 95% CI ⫽ .285, 1.00;
r for relevant stratified analyses were: general positive affect r ⫽
.207; general negative affect r ⫽ .366.
Highest nontarget affective state. The overall effect size for
the highest nontarget affective state was nonsignificant for the full
sample: d ⫽ ⫺.016, 95% CI ⫽ ⫺.166, .135, p ⫽ .836, but
significant among the neutral comparison sample: d ⫽ .189, 95%
CI ⫽ .074, .303, p ⫽ .001. These represent uniformly small effect
sizes (Cohen, 1988).
Main effects analyses stratified by target affective state (see
Table 1) indicate a fairly low degree of contamination, such that
effect sizes for nontarget affective states range from nonsignificant
(positive affect, happiness, anger, sadness, fear, guilt, or disgust) to
negative and significant (negative affect).
Moderator Analyses
Moderator analyses were conducted to examine the heterogene-
ity across studies. We first conducted univariate metaregressions to
test whether study and population characteristics moderated effect
sizes (results available in the online supplementary materials Ta-
bles S2 and S3). Significant moderators identified in the univariate
analyses were retained as predictors in multivariate metaregres-
sions. Results of these metaregressions are reported here.
Target affective state. Full statistical analyses for the final
metaregression models (that included only predictors that were
significant in univariate metaregressions) are located in Table 2.
There were several notable findings regarding study design and
type of induction. First, across analyses, effect sizes for the target
affective state were smaller when it was explicitly specified that
participants were randomized to conditions. Second, also across
analyses, studies in which a video induction was used had signif-
icantly higher effect sizes. Third, among the full sample, research
in a health domain resulted in lower effect sizes for the target
affective state; however, this effect was not found in the neutral
comparison sample.
There were also several notable findings regarding the type of
affective state targeted. The pattern of analyses suggests that
inducing positive affective states produces lower effect sizes.
Specifically, among studies with a neutral comparison, targeting
any positively valenced emotion (including positive affect and
happiness) resulted in lower effect sizes; whereas among the full
sample of studies, targeting general positive affect resulted in
lower effect sizes. Moreover, among the neutral comparison sam-
ple, targeting guilt resulted in significantly higher effect sizes;
6 FERRER, GRENEN, AND TABER

Table 1
Main Effects Stratified by Target Affective State

Studies targeting general positive affect (PA) Studies targeting general negative affect (NA)
k d 95% CI p k d 95% CI p

Target affective state 21 .422 [.256, .589] ⬍.0001 20 .926 [.469, 1.384] ⬍.0001
Highest nontarget affective state 4 ⫺.017 [⫺.254, .220] .866 4 ⫺.411 [⫺.785, ⫺.036] .032
Nontarget NA 4 ⫺.017 [⫺.254, .220] .866 — — — —
Nontarget PA — — — — 4 ⫺.411 [⫺.785, ⫺.036] .032

Studies targeting happiness Studies targeting disgust

Target affective state 3 3.353 [⫺1.005, 7.710] .132 4 1.240 [.420, 2.061] .003
Highest nontarget affective state 3 ⫺2.388 [⫺5.719, .943] .160
Nontarget anger 3 ⫺2.455 [⫺5.621, .710] .128
Nontarget sadness 3 ⫺2.671 [⫺6.259, .918] .145

Studies targeting anger Studies targeting sadness

Target affective state 15 .924 [.487, 1.360] ⬍.0001 8 1.535 [.766, 2.304] ⬍.0001
Highest nontarget affective state 13 ⫺.109 [⫺.361, .143] .396 7 .407 [⫺.103, .918] .118
Nontarget fear 6 ⫺.235 [⫺.549, .079] .143
Nontarget anger — — — — 5 ⫺.146 [⫺.766, .473] .643
Nontarget sadness 7 ⫺.025 [⫺.407, .356] .897 — — — —
Nontarget guilt 4 ⫺.298 [⫺.584, ⫺.012] .041 — — — —
Nontarget happiness 5 ⫺.770 [⫺1.566, .026] .058 3 ⫺1.268 [⫺3.103, .568] .176
Nontarget hope 3 ⫺.075 [⫺.245, .095] .386 — — — —

Studies targeting fear Studies targeting guilt

Target affective state 8 .647 [.287, 1.007] ⬍.0001 8 1.017 [.070, 1.965] .035
Highest nontarget affective state 7 .063 [⫺.264, .390] .705 5 .162 [⫺.110, .433] .243
Nontarget anger 7 .039 [⫺.281, .359] .810 4 ⫺.181 [⫺.431, .070] .158

however, this effect was not uncovered in the analysis of the full
sample of studies (that includes nonneutral comparisons).
The magnitude of effect sizes between the neutral comparison
sample and the full sample did not differ for anger inductions (B ⫽
.063, p ⫽ .894), fear inductions (B ⫽ .389, p ⫽ .390), sadness
inductions (B ⫽ ⫺.606, p ⫽ .508), or negative affect inductions
(B ⫽ ⫺.084, p ⫽ .888). As such, metaregressions within studies
targeting these affective states were undertaken on all studies,
regardless of the presence of a comparison condition (see Table 3).
Guilt inductions with neutral comparisons had significantly higher
effect sizes than those using the full sample (B ⫽ 2.127, p ⫽ .001);
presence of a neutral comparison condition was the only signifi-
cant moderator in the final metaregression on guilt inductions.
Positive affect inductions with neutral comparisons had signifi-
cantly lower effect sizes in univariate analyses (B ⫽ ⫺.660, p ⫽
.005); however, in the final regression model, the presence of a
neutral comparison was no longer a significant moderator and was
trimmed from analyses presented in Table 3.
Results indicate that, among studies targeting positive affect,
inclusion of a cover story decreased the magnitude of the effect
size, whereas using picture inductions increased effect size mag-
nitude. Among studies targeting negative affect, discrepant coder
interpretation of target affective state significantly decreased the
magnitude of the effect size. Among studies targeting anger, when
randomization procedures were specified the effect size signifi-
cantly decreased. Also among studies targeting anger, examining
emotion in the context of valuation and reward was associated with
increased effect size. Finally, among studies targeting both anger
and sadness, video inductions significantly increased effect size.
Highest nontarget affective state. Full statistical analyses for
the final metaregression models are located in Table 4. Across
analyses, video inductions also increased the magnitude of the
effect size for nontarget affective states. However, beyond this
consistency, findings about significant effect size moderators were
discrepant between analyses that included the full sample versus
those that included the neutral comparison sample. One potential
reason for this discrepancy is the finding that having a neutral
comparison condition significantly reduced the magnitude of the
effect size, suggesting that there may be inherent differences
between these samples. Among the neutral comparison sample, the
only other significant moderator of effect size magnitude for
nontarget affective state in the final metaregression was targeting
a discrete emotion, which increased the magnitude of effect sizes.
Among the full sample of studies, discrete emotion induction was
not a significant moderator; however, effect sizes in the full sample
were increased when the study was explicitly theory-based and
decreased for happiness inductions.
Publication Bias
Publication bias analyses were undertaken among the full sam-
ple of studies, first by calculating fail-safe N (Rosenthal, 1979).
The fail-safe N was 8,538, suggesting that even if a great number
of additional relevant studies with null results were included, the
overall effect size would remain significant.
However, because fail-safe N is biased toward overestimating
the number of null studies necessary to render the overall effect
size nonsignificant (Orwin, 1983), we also generated a funnel plot
 INTERNET-BASED AFFECT INDUCTION PROCEDURES 7

Table 2
Target Affective State Effect Size—Final Meta-Regression Models

Studies with neutral comparison All studies

Moderator B 95% CI p B 95% CI p

Participant randomization specified 59 ⫺1.244 [⫺1.643, ⫺.846] ⬍.0001 89 ⫺1.244 [⫺1.615, ⫺.873] ⬍.0001
Positively valenced affective state targeted 59 ⫺.543 [⫺.905, ⫺.182] .003
General positive affect targeted 59 89 ⫺.442 [⫺.767, ⫺.117] .008
Guilt targeted 59 1.001 [.256, 1.746] .008
Video induction 59 1.307 [.703, 1.911] ⬍.0001 89 1.513 [1.046, 1.980] ⬍.0001
Health domain 59 89 ⫺.560 [⫺.985, ⫺.136] .010

of the standard error by the standard mean difference (see Figure
2). The distribution is asymmetrical, suggesting the potential of
publication bias. However, trim-and-fill calculations (Duval &
Tweedie, 2000) indicate that removing the 24 effect sizes contrib-
uting to the asymmetry does not render the effect size nonsignif-
icant (d ⫽ 0.348, 95% CI ⫽ .152, 545).
Discussion
The present meta-analysis suggests that emotion can be effec-
tively experimentally induced via the Internet. Overall, manipula-
tions effectively induced the target affective state, and this finding
was consistent across all main effects analyses. The effect sizes for
Internet inductions seem comparable with those calculated in
previous meta-analyses that examined inductions in primarily lab-
oratory settings (Lench et al., 2011 g ⫽ 0.83 compared to our
overall g ⫽ .822 and .850; Westermann et al., 1996 r ⫽ .48
compared with our r ⫽ .317 and .355).
The inductions were effective overall for most affective states.
The one exception was the body of studies that attempted to induce
happiness; among these studies, the overall effect size was positive
but not significant. It may be that it is more difficult to induce
happiness via the Internet than other affective states. This stands in
contrast with previous meta-analysis of laboratory inductions,
where happiness inductions had effect sizes comparable with those
targeting discrete negative emotions (Lench et al., 2011). How-
ever, lower effect sizes for happiness are consistent with hypoth-
eses regarding negativity bias, which posit that negative affect is
easier to induce than positive affect (Augustine & Hemenover,
2009; Ito et al., 1998). It is also possible that the small number of
studies targeting happiness (k ⫽ 3) did not yield adequate power to
detect an effect size. The overall effect size was influenced by one
study with a negative effect size. Without additional power, it is
impossible to determine whether this study is an exception, or
whether there are systematic reasons why some studies or induc-
tions may be able to influence happiness via the Internet, whereas
others are not. Future research is necessary to test whether happi-
ness can be effectively induced via the Internet.
Similarly, it has been hypothesized that positive affect may be
particularly difficult in Internet inductions, as individuals may be
in better moods at home when completing a study on their own
timeline, than when completing a study in a laboratory (Göritz et
al., 2007). Consistent with this hypothesis, positively valenced
affective state inductions had lower effect sizes than negatively
valenced affective state inductions. Indeed, the effect sizes for
Internet inductions of positive affect in the present analyses are
lower than those reported in a previous meta-analysis of mood
inductions largely administered in the laboratory (Westermann et
al., 1996; r ⫽ .41 compared with our r ⫽ .207). The lower effect
size for positive affect inductions is also consistent with the
assertion that positive affective states are less differentiated than
negative affective states (Shiota et al., 2014), and as such, positive
affect may be more difficult to precisely induce. Targeting discrete
positive emotions (e.g., pride, gratitude) may produce higher effect
sizes. However, the previously described findings concerning the
difficulty of effectively inducing one specific positive emotion,
happiness, do not support this hypothesis. It may be that inducing
any positive affective state, general or discrete, is more difficult
than inducing negative affect because of relatively low levels of
differentiation among positive affect.
The same may also be true for negative affect induction: It may
be more difficult to induce negative affect via the Internet than in
the laboratory (Westermann et al., 1996; r ⫽ .53 compared with
our r ⫽ .366). However, discrete negative emotions induced via
Internet manipulations yielded similar effect sizes to those induced
in a laboratory (Anger: g ⫽ .80 vs. .917; Sadness: g ⫽ .73 vs. .851;
for Lench et al., 2011 compared with our analyses, respectively).

Table 3
Highest Nontarget Affective State Effect Size (Final Models)

Studies with neutral comparison All studies

Moderator k B 95% CI p k B 95% CI p

Theory-based 47 .323 [.047, .599] .023

Discrete emotion targeted 26 .390 [.154, .626] .001
Happiness targeted 47 ⫺1.074 [⫺1.619, ⫺.529] .001
Video induction 26 .571 [.261, .881] .001 47 .450 [.102, .798] .011
Neutral comparison condition — — — — 47 ⫺.479 [.228, .730] .001
8 FERRER, GRENEN, AND TABER

Table 4
Final Meta-Regressions Stratified by Target Emotion (DV ⫽ Target Emotion)

Studies targeting general positive Studies targeting general negative

affect (PA) affect (NA)
Moderator k B 95% CI p k B 95% CI p

Participant randomization specified — — — 20 ⫺2.871 [⫺4.029, ⫺1.712] ⬍.0001

Discrepant coder interpretation of target affective state — — — ⫺.880 [⫺1.749, ⫺.011] .047
Cover story 21 ⫺.367 [⫺.679, ⫺.054] .021 — — —
Picture induction 21 .455 [.086, .824] .016

Studies targeting anger Studies targeting sadness

Participant randomization specified 15 ⫺.108 [⫺1.600, ⫺.416] .0009 8 — — —
Video induction 1.640 [1.010, 2.269] ⬍.0001 1.614 [.216, 3.013] .024
Valuation domain .842 [.253, 1.431] .005 — — —
Note. No significant moderators of fear effect size magnitude uncovered in analyses.

The exception was fear, which may be more difficult to induce via
the Internet (Lench et al., 2011 g ⫽ .1.05 compared with our .643).
We also examined whether other study, design, and sample
characteristics were associated with significantly increased (or
decreased) effect sizes. Video inductions were uniformly associ-
ated with higher effect sizes for the target affective state, particu-
larly for anger and sadness, consistent with Westermann et al.
(1996). Videos are considered to be a robust means of eliciting
affective states in laboratory research (Kring & Gordon, 1998;
Rottenberg, Ray, & Gross, 2007; see Lench et al., 2011). However,
contamination was more likely when using video inductions, con-
sistent with previous work on emotion elicitation using films,
which found that films elicit a range of unintended emotions
(Gross & Levenson, 1995). Thus, even though videos were most
effective at inducing the target affective state, the goals of the
study should be carefully considered when selecting video induc-
tions for Internet research on affect, particularly when hypotheses
rely on isolating the influence of one particular affective state.
Studies had lower effect sizes when the authors specified that
the study involved randomization. It is possible that more highly
controlled studies are more adequately balanced in terms of par-
ticipants’ baseline affect, and as such, are less able to induce
emotion among some participants, contributing to a lower effect
size. However, because some studies may have used randomiza-
tion but failed to specify in the article, it is difficult to interpret this
association. Unlike in previous meta-analyses, the proportion of
women (Lench et al., 2011) and presence of cover story techniques
(Lench et al., 2011; Westermann et al., 1996) did not influence the
magnitude of the overall effect size. However, the presence of a
cover story did produce reductions in the overall magnitude of
effect size for studies targeting positive affect.
More important, among negative affect inductions, studies
where the coders rated the experimental condition as discrepant
from the authors’ report (e.g., rated a negative affect induction as
relevant to anger) were associated with lower effect size magni-
tudes. This result makes sense intuitively; when coders rated an
induction as less precise than authors classified it to be, it follows
that it might be less effective in inducing the target affective state.
These results underscore the importance of carefully considering
the nature and target of the induction when writing and interpreting
an article using an Internet negative affect induction. However,
this imprecision seemed to be a problem only for negative affect
inductions in the present meta-analysis; although this variable was
associated with smaller effect sizes at the univariate level among

Funnel Plot of Standard Error by Std diff in means

0.0

0.2

0.4
Standard Error

0.6

0.8

1.0

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6

Std diff in means

Figure 2. Funnel plot of full sample.

 INTERNET-BASED AFFECT INDUCTION PROCEDURES
studies with a neutral comparison condition (see online supple-
mentary materials), the moderator coefficient was no longer sig-
nificant in the final meta-regression models. For negative affect
inductions in particular, imprecision may be a problem because
there are more distinctions in negative than positive affective states
(Shiota et al., 2014), potentially leading to greater likelihood that
different negative affect inductions will unintentionally target very
different discrete emotions.
There was some degree of contamination of inductions. That is,
the effect sizes for nontarget affective states were not always null
or negative, which would be expected if an induction influenced
the target affective state but not other nontarget affective states.
Specifically, the overall effect size for nontarget affective states
among studies with a neutral comparison condition was signifi-
cant, although small. Although contamination is a concern, re-
search suggests that contamination may also be an issue in labo-
ratory studies (Gross & Levenson, 1995). Moreover, the small
magnitude of such effect sizes for highest nontarget affective state
suggests this is not a critical concern for future Internet research
involving affective inductions.
However, note that null effect sizes for nontarget affective states
should be interpreted with caution, as they depend on the measures
authors included in their study. Only 52% of the comparisons
included a manipulation check for a nontarget emotion. Thus,
these analyses can be a guide, but should be interpreted with
caution. For example, analyses indicate that positive affect induc-
tions had nonsignificant effect sizes for other nontarget affective
states. However, most positive affect inductions included only
measures of positive and negative affect. As such, we cannot
determine whether positive affect inductions would involve con-
tamination of discrete emotion states such as pride or hope.
The present meta-analysis has a number of limitations in addi-
tion to the caution over interpreting effect sizes for nontarget
affective states described above. A chief limitation concerns the
self-reported nature of manipulation checks. Although self-report
remains the predominant method for establishing the effectiveness
of an affect induction across the extant literature (Lench et al.,
2011), research suggests that individuals may have some difficulty
identifying and labeling affective states (e.g., Buck, 1990; Buck,
Miller, & Caul, 1974), and often self-reported affective states do
not correspond with other indicators of affect, such as physiolog-
ical arousal or facial expressions (Barrett, 2006; e.g., Cacioppo et
al., 2000; Lindquist et al., 2012). This limitation is applicable to
the present meta-analysis, but also more broadly to the studies of
Internet-delivered emotion inductions that contribute to the anal-
ysis (as well as studies involving emotion inductions in laborato-
ries).
Another limitation concerns publication bias; although trim-
and-fill analyses indicate that removing the 24 effect sizes con-
tributing to the asymmetrical funnel plot does not render the
overall effect size nonsignificant, such trimming does reduce the
magnitude of the effect size. Thus, conclusions about the magni-
tude of effect size for affect inductions administered via the
Internet should be interpreted with some caution. However, this
limitation is offset by the inclusion of a high proportion (40%) of
unpublished studies (dissertations and unpublished data), as well
as the finding that publication status did not moderate effect size.
Moreover, asymmetry in effect sizes by sample size can result
from inherent differences in the types of studies rather than indi-
9
cate publication bias (Lau, Ioannidis, Terrin, Schmid, & Olkin,
2006). Finally, in the context of interpreting publication bias, it is
important to note that we are not making the case that all studies
that try to induce mood via the Internet are effective. Rather, this
meta-analysis suggests that it is possible to successfully induce
mood online.
This synthesis highlights several gaps in knowledge that should
be addressed in future research. No studies reviewed used sublim-
inal priming or imagination inductions, only one used a music
induction (in combination with other components), and only one
study attempted to induce stress. Future research should examine
the feasibility of a wider variety of affect inductions, consistent
with the extant laboratory-based literature. Moreover, future re-
search is necessary to identify potent means of inducing affective
states that do not involve contamination; although video inductions
uniformly increased the magnitude of effect sizes for target affec-
tive states, they also increased effect sizes for nontarget affective
states. This may be a particular problem when a study hypothesis
involves very specific predictions about different affective states.
Future research should also address the shortcomings of Internet
inductions for general positive affect, general negative affect, and
sadness; although effect sizes were lower for Internet studies than
for the extant laboratory literature, it is possible that different types
of Internet inductions would be more effective in successfully
inducing these states.
This study has important implications for future affective sci-
ence research. Because of recent methodological and technological
advances, Internet research has revolutionized the process of con-
ducting psychological experiments (Berinsky, Huber, & Lenz,
2012; Buhrmester, Kwang, & Gosling, 2011; Horton, Rand, &
Zeckhauser, 2011; Paolacci & Chandler, 2014; Skitka & Sargis,
2006). However, skepticism remains concerning researchers’ abil-
ity to achieve control and precision in Internet experiments (see
Kraut et al., 2004; Skitka & Sargis, 2006). The present meta-
analysis suggests that affective scientists can confidently acceler-
ate their programs of research by leveraging the Internet to take
advantage of important benefits such as increased sample repre-
sentativeness (Berinsky et al., 2012; Paolacci et al., 2010; Ross et
al., 2010), cost and resource conservation (Kraut et al., 2004),
reduced measurement error and social desirability bias (Skitka &
Sargis, 2005, 2006), and increased speed in data collection
(Buhrmester, Kwang, & Gosling, 2011).
References
ⴱ
References marked with an asterisk indicate studies included in the
meta-analysis.
Augustine, A. A., & Hemenover, S. H. (2009). On the relative effective-
ness of affect regulation strategies: A meta-analysis. Cognition and
Emotion, 23, 1181–1220. http://dx.doi.org/10.1080/0269993080
2396556
ⴱ
Baron, J. (April 3, 2014). Provided via personal email communication.
Unpublished data.
Barrett, L. F. (2006). Are emotions natural kinds? Perspectives on Psy-
chological Science, 1, 28 –58. http://dx.doi.org/10.1111/j.1745-6916
.2006.00003.x
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online
labor markets for experimental research: Amazon.com’s Mechanical
Turk. Political Analysis, 20, 351–368. http://dx.doi.org/10.1093/pan/
mpr057
10 FERRER, GRENEN, AND TABER
Biostat. (2014). Comprehensive meta-analysis software. Retrieved from
http://www.meta-analysis.com/index.php
Birnbaum, M. H. (2004). Human research and data collection via the
internet. Annual Review of Psychology, 55, 803– 832. http://dx.doi.org/
10.1146/annurev.psych.55.090902.141601
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009).
Introduction to meta-analysis. West Sussex, United Kingdom: Wiley.
http://dx.doi.org/10.1002/9780470743386
ⴱ ⴱ
Boudewyns, V., Turner, M. M., & Paquin, R. S. (2013). Shame-free guilt
appeals: Testing the emotional and cognitive effects of shame and guilt
appeals. Psychology & Marketing, 30, 811– 825. http://dx.doi.org/
10.1002/mar.20647
Bradley, M. M., & Lang, P. J. (1999). International affective digitized
sounds (IADS): Stimuli, instruction manual and affective ratings (Tech.
Rep. No. B-2). Gainesville, FL: The Center for Research in Psychophys-
iology, University of Florida.
Buck, R. (1990). Rapport, emotional education, and emotional compe-
tence. Psychological Inquiry, 1, 301–302. http://dx.doi.org/10.1207/
s15327965pli0104_4
Buck, R., Miller, R. E., & Caul, W. F. (1974). Sex, personality, and
physiological variables in the communication of affect via facial expres-
sion. Journal of Personality and Social Psychology, 30, 587–596. http://
dx.doi.org/10.1037/h0037041
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechan-
ical Turk: A new source of inexpensive, yet high-quality, data? Per-
spectives on Psychological Science, 6, 3–5. http://dx.doi.org/10.1177/
1745691610393980
Cacioppo, J. T., Berntson, G. G., Larsen, J. T., Poehlmann, K. M., & Ito,
T. A. (2000). The psychophysiology of emotion. In R. Lewis & J. M.
Haviland-Jones (Eds.), The handbook of emotion (2nd ed., pp. 173–191).
New York, NY: Guilford Press.
Carter, L. E., McNeil, D. W., Vowles, K. E., Sorrell, J. T., Turk, C. L.,
Ries, B. J., & Hopko, D. R. (2002). Effects of emotion on pain reports,
tolerance and physiology. Pain Research & Management, 7, 21–30.
ⴱ
Chen, R., & Sakamoto, Y. (2013). Perspective matters: Sharing of crisis
information in social media. Paper presented at the Hawaii International
Conference on System Sciences.
ⴱ
Choi, Y. (2013). Jurors’ subjective certainty and standards of proof: The
role of emotion and severity of carge in subjective probability judgment.
Theses, Dissertations, and Student Research: Department of Psychology,
Paper 61. http://digitalcommon.unl.edu/psychdiss/61.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences.
Hillsdale, NJ: Erlbaum.
ⴱ
Cohen, P. M., & Knobe, J. (April 6, 2014). Provided via personal email
communication. Unpublished data.
ⴱ
Coleman, M. D. (2010). Sunk cost, emotion, and commitment to educa-
tion. Current Psychology, 29, 346 –356. http://dx.doi.org/10.1007/
s12144-010-9094-6
ⴱ
Coleman, M. D. (2011). Emotion and the self-serving bias. Current
Psychology, 30, 345–354. http://dx.doi.org/10.1007/s12144-011-9121-2
ⴱ
Coleman, M. D. (2013). Emotion and the ultimate attribution error.
Current Psychology, 32, 71– 81. http://dx.doi.org/10.1007/s12144-013-
9164-7
ⴱ
Denson, T. F. (April 9, 2014). Provided via personal email communica-
tion. Unpublished data.
Duval, S., & Tweedie R.. (2000). Trim and fill: A simple funnel-plot-based
method of testing and adjusting for publication bias in meta-analysis.
Biometrics, 56, 455– 463.
Egermann, H., & McAdams, S. (2013). Empathy and emotional contagion
as a link between recognized and felt emotions in music listening. Music
Perception, 31, 139 –156. http://dx.doi.org/10.1525/mp.2013.31.2.139
ⴱ
Gadarian, S. K. (2008). Elicit cues and emotion. Unpublished data.
Retrieved from: http://www.tessexperiments.org/data/gadarian568.html,
April 9 2015.
ⴱ
Göritz, A. S., & Moser, K. (2006). Web-based mood induction. Cognition
and Emotion, 20, 887– 896. http://dx.doi.org/10.1080/026999305
00405386
ⴱ
Göritz, A. S. (2007). The induction of mood via the WWW. Motivation
and Emotion, 31, 35– 47. http://dx.doi.org/10.1007/s11031-006-9047-4
Gross, J. J., & Levenson, R. W. (1995). Emotion elicitation using films.
Cognition and Emotion, 9, 87–108. http://dx.doi.org/10.1080/
02699939508408966
Hängii, Y. (2004). Stress and emotion recognition: An internet experiment
using stress induction. Swiss Journal of Psychology, 63, 113–125. http://
dx.doi.org/10.1024/1421-0185.63.2.113
Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect
size and related estimators. Journal of Educational Statistics, 6, 107–
128. http://dx.doi.org/10.2307/1164588
Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in
meta-analysis. Psychological Methods, 3, 486 –504. http://dx.doi.org/
10.1037/1082-989X.3.4.486
ⴱ
Helweg-Larsen, M., & Vernon, D. (June 4, 2014). Provided via personal
email communication. Unpublished data.
Horton, J. J., & Chilton, L. (2010). The labor economics of paid crowd-
sourcing. Proceedings of the 11th ACM conference on electronic com-
merce. http://dx.doi.org/10.1145/1807342.1807376
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online
laboratory: Conducting experiments in a real labor market. Experimental
Economics, 14, 399 – 425. http://dx.doi.org/10.1007/s10683-011-9273-9
Ipeirotis, P. G. (2010) Demographics of Mechanical Turk (Tech. Rep. No.
CeDER-10 – 01). New York, NY: New York University. Retrieved from
http://hdl.handle.net/2451/29585
Ito, T. A., Larsen, J. T., Smith, N. K., & Cacioppo, J. T. (1998). Negative
information weighs more heavily on the brain: The negativity bias in
evaluative categorizations. Journal of Personality and Social Psychol-
ogy, 75, 887–900. http://dx.doi.org/10.1037/0022-3514.75.4.887
Johnson, B. T., & Eagly, A. H. (2000). Quantitative synthesis of social
psychological research. In H. T. Reis & C. M. Judd (Eds.), Handbook of
research methods in social and personality psychology (pp. 496 –528).
London: Cambridge University Press.
ⴱ
Koblitz, A. K., Ferrer, R. A., & Klein, W. M. P. (2012). Self-affirmation
and affect: Effects on implementation intentions. Unpublished data.
Retrieved from: http://www.tessexperiments.org/data/koblitz279.html,
April 9 2015.
Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological
research. In M. H. Birnbaum (Ed.), Psychological experiments on the
Internet (pp. 35– 60). San Diego, CA: Academic Press. http://dx.doi.org/
10.1016/B978-012099980-4/50003-4
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M.
(2004). Psychological research online: Report of Board of Scientific
Affiars’ Advisory Group on the conduct of research on the internet.
American Psychologist, 59, 105–117.
Kring, A. M., & Gordon, A. H. (1998). Sex differences in emotion:
Expression, experience, and physiology. Journal of Personality and
Social Psychology, 74, 686 –703. http://dx.doi.org/10.1037/0022-3514
.74.3.686
Lau, J., Ioannidis, J. P., Terrin, N., Schmid, C. H., & Olkin I. (2006). The
case of the misleading funnel plot. British Medical Journal, 333, 597–
600. http://dx.doi.org/10.1136/bmj.333.7568.597
ⴱ
Lee, G., Barrowclough, C., & Lobban, F. (2011). The influence of positive
affect on jumping to conclusions in delusional thinking. Personality and
Individual Differences, 50, 717–722. http://dx.doi.org/10.1016/j.paid
.2010.12.024
ⴱ
Lee, S. Y. (2010). Framing moral responsibility: The influence of moral
emotions on persuasive health messages. Unpublished doctoral disser-
tation, University of Wisconsin-Madison.
Lench, H. C., Flores, S. A., & Bench, S. W. (2011). Discrete emotions
predict changes in cognition, judgment, experience, behavior, and phys-
 INTERNET-BASED AFFECT INDUCTION PROCEDURES
iology: A meta-analysis of experimental emotion elicitations. Psycho-
logical Bulletin, 137, 834 – 855. http://dx.doi.org/10.1037/a0024244
ⴱ
Lerner, J. S., Gonzalez, R. M., Small, D. A., & Fischhoff, B. (2003).
Effects of fear and anger on perceived risks of terrorism: A national field
experiment. Psychological Science, 14, 144 –150. http://dx.doi.org/
10.1111/1467-9280.01433
Lerner, J. S., & Keltner, D. (2001). Fear, anger, and risk. Journal of
Personality and Social Psychology, 81, 146 –159. http://dx.doi.org/
10.1037/0022-3514.81.1.146
ⴱ
Lerner, J. S., Li, Y., & Weber, E. U. (2013). The financial costs of
sadness. Psychological Science, 24, 72–79.
Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., & Barrett,
L. F. (2012). The brain basis of emotion: A meta-analytic review.
Behavioral and Brain Sciences, 35, 121–143. http://dx.doi.org/10.1017/
S0140525X11000446
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. London,
United Kingdom: Sage
Mathy, R. M., Kerr, D. L., & Haydin, B. M. (2003). Methodological rigor
and ethical considerations in internet-mediated research. Psychotherapy:
Theory, Research, Practice, Training, 40, 77– 85. http://dx.doi.org/
10.1037/0033-3204.40.1-2.77
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G., & the PRISMA
Group. (2009). Preferred reporting items for systematic reviews and
meta-analyses: The PRISMA statement. PLoS Medicine, 6, e1000097.
http://dx.doi.org/10.1371/journal.pmed.1000097
ⴱ
Morgan, M. (2013). Playing against the odds: The role of mood, experi-
ence and risk in decision-making. (Doctoral dissertation). Retrieved
from http://hdl.handle.net/10292/23806
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal
of Educational Statistics, 8, 157–159. http://dx.doi.org/10.2307/1164923
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Me-
chanical Turk as a participant tool. Current Directions in Psychological
Science, 23, 184 –188. http://dx.doi.org/10.1177/0963721414531598
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments
on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–
419. psiTurk. Retrieved from http://gureckislab.org/blog/?p⫽4318
Reips, U.-D. (2000). The Web experiment method: Advantages, disadvan-
tages, and solutions. In M. H. Birnbaum (Ed.), Psychological experi-
ments on the Internet (pp. 89 –114). San Diego, CA: Academic Press.
http://dx.doi.org/10.1016/B978-012099980-4/50005-8
Rosenthal, R. (1979). The “File Drawer Problem” and tolerance for null
results. Psychological Bulletin, 86, 638 – 641. http://dx.doi.org/10.1037/
0033-2909.86.3.638
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010).
Who are the crowdworkers? Shifting demographics in mechanical turk.
In CHI ’10 Extended Abstracts on Human Factors in Computing Sys-
tems. (pp. 2863–2872). Paper Presented at ACM Special Interest Group
on Computer-Human Interaction, Atlanta, GA, 10 –15 April New York,
NY: Association for Computing Machinery.
Rottenber, J., Ray, R. D., & Gross, J. J. (2007). Emotion elicitation using
films. In J. A. Coan & J. J. B. Allen (Eds.), The handbook of emotion
elicitation and assessment (pp. 9 –28). London: Oxford University Press.
View publication stats
11
ⴱ
Ryan, T. J. (2012). What makes us click? Demonstrating incentives for
angry discourse with digital-age field experiments. The Journal of Pol-
itics, 74, 1138 –1152. http://dx.doi.org/10.1017/S0022381612000540
ⴱ
Scheibehenne, B., & von Helversen, B. (2015). Selecting decision strat-
egies: The differential role of affect. Cognition & Emotion, 29, 158 –167.
Shiota, M. N., Neufeld, S. L., Danvers, A. F., Osborne, E. A., Sng, O., &
Yee, C. I. (2014). Positive emotion differentiation: A functional ap-
proach. Social and Personality Psychology Compass, 8, 104 –117.
ⴱ
Sinclair, M., Ashkanasy, N. M., & Chattopadhyay, P. (2010). Affective
antecedents of intuitive decision making. Journal of Management &
Organization, 16, 382–398. http://dx.doi.org/10.5172/jmo.16.3.382
Skitka, L. J., & Sargis, E. G. (2005). Social psychological research and the
internet: The promise and peril of a new methodological frontier. In Y.
Amichai-Hamburger (Ed.), The social net: The social psychology of the
Internet (pp. 1–26). Oxford, United Kingdom: Oxford University Press.
Skitka, L. J., & Sargis, E. G. (2006). The internet as psychological
laboratory. Annual Review of Psychology, 57, 529 –555. http://dx.doi
.org/10.1146/annurev.psych.57.102904.190048
Strack, F., Schwarz, N., & Gschneidinger, E. (1985). Happiness and
reminiscing: The role of time perspective, affect, and mode of thinking.
Journal of Personality and Social Psychology, 49, 1460 –1469. http://
dx.doi.org/10.1037/0022-3514.49.6.1460
Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D.,
Rennie, D., . . . Thacker, S. B. (2000). Meta-analysis of observational
studies in epidemiology: A proposal for reporting. Meta-analysis Of
Observational Studies in Epidemiology (MOOSE) group. Journal of the
American Medical Association, 283, 2008 –2012. http://dx.doi.org/
10.1001/jama.283.15.2008
Suri, S., Mason, W. A., & Goldstein, D. G. (2014). The 2nd workshop on
crowdsourcing and online behavioral experiments. Workshop con-
ducted at the American Conference on Economics and Computation,
Palo Alto. Survey Monkey. Retrieved from http://www.surveymonkey
.com
ⴱ
Tsai, M.-H., & Young, M. J. (2010). Anger, fear, and escalation of
commitment. Cognition & Emotion, 24, 962–973. http://dx.doi.org/
10.1080/02699930903050631
Velten, E., Jr. (1968). A laboratory task for induction of mood states.
Behaviour Research and Therapy, 6, 473– 482. http://dx.doi.org/
10.1016/0005-7967(68)90028-4
ⴱ
Verheyen, C., & Goritz, A. S. (2009). Plain texts as an online mood-
induction procedure. Social Psychology, 40, 6 –15. http://dx.doi.org/
10.1027/1864-9335.40.1.6
Westermann, R., Spies, K., Stahl, G., & Hesse, F. W. (1996). Relative
effectiveness and validity of modd induction procedures: A meta-
analysis. European Journal of Social Psychology, 26, 557–580. http://
dx.doi.org/10.1002/(SICI)1099-0992(199607)26:4⬍557::AID-
EJSP769⬎3.0.CO;2-4
Received July 9, 2014
Revision received October 7, 2014
Accepted October 8, 2014 䡲