See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/305512554

Measuring emotions during epistemic activities: the Epistemically-Related

Emotion Scales

Article  in  Cognition and Emotion · July 2016

DOI: 10.1080/02699931.2016.1204989

CITATIONS READS

195 5,252

4 authors:

Reinhard Pekrun Elisabeth Vogl

Ludwig-Maximilian-University of Munich (LMU); Australian Catholic University (ACU) Ludwig-Maximilians-University of Munich
385 PUBLICATIONS   37,538 CITATIONS    44 PUBLICATIONS   945 CITATIONS   

SEE PROFILE SEE PROFILE

Krista R. Muis Gale M. Sinatra

McGill University University of Southern California
95 PUBLICATIONS   4,696 CITATIONS    139 PUBLICATIONS   8,936 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

DFG-Projekt "FEEL - Forschung zum Emotionalen Erleben im Lehr-Lern-Kontext" View project

ECoWeB: Assessing and Enhancing Emotional Competence for Well-Being in the Young: A principled, evidence-based, mobile-health approach to prevent mental
disorders and promote mental well-being View project

All content following this page was uploaded by Gale M. Sinatra on 25 July 2016.

The user has requested enhancement of the downloaded file.

 Cognition and Emotion

ISSN: 0269-9931 (Print) 1464-0600 (Online) Journal homepage: http://www.tandfonline.com/loi/pcem20

Measuring emotions during epistemic activities:

the Epistemically-Related Emotion Scales

Reinhard Pekrun, Elisabeth Vogl, Krista R. Muis & Gale M. Sinatra

To cite this article: Reinhard Pekrun, Elisabeth Vogl, Krista R. Muis & Gale M. Sinatra (2016):
Measuring emotions during epistemic activities: the Epistemically-Related Emotion Scales,
Cognition and Emotion, DOI: 10.1080/02699931.2016.1204989

To link to this article: http://dx.doi.org/10.1080/02699931.2016.1204989

View supplementary material

Published online: 22 Jul 2016.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=pcem20

Download by: [93.204.107.184] Date: 22 July 2016, At: 23:20

 COGNITION AND EMOTION, 2016
http://dx.doi.org/10.1080/02699931.2016.1204989

BRIEF ARTICLE

Measuring emotions during epistemic activities: the Epistemically-

Related Emotion Scales
Reinhard Pekruna,b†, Elisabeth Vogla†, Krista R. Muisc and Gale M. Sinatrad
a
Department of Psychology, University of Munich, Munich, Germany; bInstitute for Positive Psychology and Education, Australian
Catholic University, Sydney, Australia; cDepartment of Educational and Counselling Psychology, McGill University, Montreal,
Quebec, Canada; dRossier School of Education, University of Southern California, Los Angeles, CA, USA

ABSTRACT ARTICLE HISTORY

Measurement instruments assessing multiple emotions during epistemic activities are Received 23 July 2015
largely lacking. We describe the construction and validation of the Epistemically- Revised 12 June 2016
Accepted 15 June 2016
Downloaded by [93.204.107.184] at 23:20 22 July 2016

Related Emotion Scales, which measure surprise, curiosity, enjoyment, confusion,

anxiety, frustration, and boredom occurring during epistemic cognitive activities.
KEYWORDS
The instrument was tested in a multinational study of emotions during learning Epistemic emotion; surprise;
from conflicting texts (N = 438 university students from the United States, Canada, curiosity; confusion; value
and Germany). The findings document the reliability, internal validity, and external appraisal
validity of the instrument. A seven-factor model best fit the data, suggesting that
epistemically-related emotions should be conceptualised in terms of discrete
emotion categories, and the scales showed metric invariance across the North
American and German samples. Furthermore, emotion scores changed over time as
a function of conflicting task information and related significantly to perceived task
value and use of cognitive and metacognitive learning strategies.

Research has started to acknowledge the importance
of emotions for complex learning and cognitive per-
formance (see e.g. Pekrun & Linnenbrink-Garcia,
2014). In this nascent field of inquiry, researchers
have focused on examining achievement emotions,
such as hope, pride, anxiety, or shame related to
success and failure. However, emotions that are trig-
gered by the cognitive characteristics of tasks can
also be of fundamental importance for learning,
such as surprise, curiosity, or confusion about contra-
dictory information. As proposed by epistemologists,
these affective states represent epistemic emotions
because they relate to the knowledge-generating
qualities of cognitive activities (Brun, Doğuoğlu, &
Kuenzle, 2008; Morton, 2010). Empirical findings
support the view that epistemic emotions can strongly
impact learning and performance (e.g. D’Mello,
Lehman, Pekrun, & Graesser, 2014; Kang et al., 2009).
Further progress in research on epistemic emotions
requires measurement instruments to assess these
emotions. Specifically, instruments are needed that
measure multiple emotions during epistemic activity
and that are suited to track the dynamics of these
emotions over time. To date, measurement of these
emotions has focused on assessing curiosity (Jirout &
Klahr, 2012). In addition, researchers have used qualitat-
ive self-report and single-item instruments assessing
various emotions, such as emotion checklists (e.g.
D’Mello & Graesser, 2012). By contrast, systematic
multi-item instruments suited to assess a broader
range of these emotions are lacking. We aimed to
redress this deficit by developing multi-item self-report
scales measuring seven major emotions that occur
during epistemic activity and are believed to be of
primary importance for learning and the generation of
knowledge (Epistemically-Related Emotion Scales (EES)).

CONTACT Reinhard Pekrun pekrun@lmu.de

Supplemental data for this article can be accessed here doi:10.1080/02699931.2016.1204989.
†
The first two authors contributed equally to this work.
© 2016 Informa UK Limited, trading as Taylor & Francis Group
 2 R. PEKRUN ET AL.
We first outline the concept of epistemic emotions
that guided this research. Next, we describe the devel-
opment of the EES. We then report results from a mul-
tinational investigation that used conflicting texts
about climate change to arouse epistemic emotions.
Data from this study were employed to examine the
reliability, internal validity, and external validity of
the instrument.
Defining epistemic emotions
Epistemic emotions relate to the knowledge-generat-
ing qualities of cognitive tasks and activities (Brun
et al., 2008). For these emotions, knowledge and the
generation of knowledge are the objects of emotions.
This definition of epistemic emotions by object focus
Downloaded by [93.204.107.184] at 23:20 22 July 2016
is equivalent to the definition of other types of episte-
mic variables that share the same focus, such as epis-
temic cognition, epistemic metacognition, and
epistemic beliefs. Epistemic emotions differ from
these other epistemic variables by their affective
nature.
Epistemic emotions represent a major category of
human emotion serving evolutionary-based purposes
of acquiring knowledge about the world and the self.
A prototypical situation for the arousal of epistemic
emotions is discrepant information and appraisals of
cognitive incongruity that can trigger surprise and
curiosity (Kang et al., 2009), confusion, frustration,
and boredom when the incongruity cannot be
resolved (D’Mello & Graesser, 2012; D’Mello et al.,
2014), anxiety in the case of severe incongruity and
information that deeply disturbs existing beliefs
(Hookway, 2008), or enjoyment and delight when
the problem is solved. Appraisals of the positive
value of an epistemic activity should promote positive
epistemic emotions (curiosity, enjoyment) and reduce
boredom. Epistemic emotions are closely related to
the concepts of cognitive emotions (Scheffler, 1991)
and knowledge emotions (Silvia, 2010). We prefer to
use the term epistemic emotion because it is well
established in epistemology (Brun et al., 2008) and
aligned with terms denoting other constructs related
to epistemic processes, such as epistemic beliefs.
To conceptualise epistemic emotions, it is critical to
attend to the defining features of emotion. We
propose to adopt a multicomponent approach (e.g.
Scherer, 2009), which implies that emotions are
systems of coordinated psychological processes
including affective, cognitive, physiological, motiva-
tional, and behavioural components. An affective
state qualifies as an epistemic emotion if it comprises
these elements. Accordingly, surprise, enjoyment,
anxiety, frustration, and boredom related to knowl-
edge and knowing can be considered epistemic
emotions. This is also true for curiosity and confusion
which traditionally have not been classified as
emotions, but have been shown to involve affective
feelings, physiological arousal, motivational impulses,
as well as specific patterns of facial expression (e.g.
Markey & Loewenstein, 2014; Reeve, 1993; Rozin &
Cohen, 2003).
Epistemic emotions differ from other groups of
human emotions, such as social, moral, or achieve-
ment emotions, in terms of their object focus (Brun
et al., 2008). As noted, knowledge and the generation
of knowledge are the objects for epistemic emotions;
in social, moral, and achievement emotions, other
persons, moral norms, or success and failure, respect-
ively, are the objects. Some emotions are epistemic by
nature, such as curiosity and confusion, whereas
others can belong to various categories of emotion,
depending on the object focus of attention. For
example, during cognitive activities, some emotions
can be experienced as epistemic emotions or as
achievement emotions (Pekrun & Perry, 2014). A stu-
dent’s frustration at not deriving a correct solution
to a mathematics problem would be considered an
epistemic emotion if the focus is on the cognitive
incongruity resulting from the unsolved problem.
However, if the focus is on personal failure and the
inability to solve the problem, then the student’s frus-
tration would be considered an achievement emotion.
As such, it is important to acknowledge that epistemic
emotions can share affective properties with other cat-
egories of emotion but differ from them in terms of
their object focus (Brun et al., 2008).
Development of the EES
We aimed to develop a self-report instrument captur-
ing major emotions that occur frequently in epistemic
contexts and are functionally relevant for learning and
cognitive problem solving. Curiosity, joy, confusion,
anxiety, frustration, and boredom have been found
to be especially frequent and powerful in these con-
texts (see e.g. D’Mello, 2013; Pekrun & Stephens,
2012). We added surprise which represents an
immediate affective reaction to cognitive incongruity
and serves to promote subsequent epistemic cogni-
tion. As such, we developed scales measuring surprise,
curiosity, enjoyment, confusion, anxiety, frustration,

and boredom. The scales represent all of the emotions
targeted in D’Mello and Graesser’s (2012; D’Mello et al.,
2014) model of impasse-driven emotions and Pekrun
and Stephens’s (2012) classification of epistemic
emotions.
Our goal was to create an instrument that tracks
changes in these emotions across episodes of cogni-
tive activity and renders psychometrically sound
scores. Accordingly, it was necessary to create multiple
scales consisting of several items each. However, to
allow for an examination of these emotions as tem-
porally fluctuating states, the instrument needs to be
simple and quick to administer.
To serve these purposes, the EES uses emotion
adjectives as items, which allows for quick responses
and limits the influence of the items on respondents’
Downloaded by [93.204.107.184] at 23:20 22 July 2016
emotions. To keep scale length at a minimum, three
items were included per scale. As such, the instrument
contains 7 three-item scales (21 items overall; Appen-
dix). Selection of items was based on existing emotion
scales and information about frequently used emotion
words (see Supplemental Material). Items are
answered using a 5-point Likert scale that asks
respondents to report how strongly they feel the
emotion (1 = not at all to 5 = very strong). The instru-
ment includes contextualised instructions (Appendix),
which can be adapted to address different settings.
Additionally, a short version of the EES is available
that uses one item per emotion only (Appendix).
Items were selected for their semantic properties to
best represent the respective emotion, and all had
factor loadings >.69 (Supplemental Material; Table
S1). This version may be especially useful when it is
necessary to keep administration time short. Similar
to experience sampling methodology, it is minimally
invasive and can be employed to measure respon-
dents’ emotions while carrying out a task.
Aims of the present research
The present study examined item and scale statistics,
reliability, internal test validity, and external test val-
idity for the EES. Data from a multinational investi-
gation were used (Muis et al., 2015). The study
examined epistemic emotions in the context of a learn-
ing task in which students had to read and interpret
conflicting texts. In the conceptual change literature,
use of conflicting information presented in texts is a
standard procedure to trigger cognitive conflict (Muis
& Duffy, 2013), which is expected to induce epistemic
emotions. Correlational analysis and confirmatory
COGNITION AND EMOTION 3
factor analysis (CFA) were employed to test internal
validity in terms of the distinctiveness of the seven
emotions and the measurement equivalence of the
EES across the participant samples from different
countries.
To evaluate external validity, we analysed the
impact of the conflicting texts, which were supposed
to increase surprise, curiosity, and confusion. In
addition, we investigated relations between the
measured emotions, the perceived value of the learn-
ing task, and participants’ self-reported use of cogni-
tive and metacognitive learning strategies. We
expected perceived value of the task to relate posi-
tively to the intensity of the emotions except
boredom, and we expected activating emotions (i.e.
surprise, curiosity, enjoyment, confusion, anxiety, and
frustration) to correlate positively and boredom to cor-
relate negatively with strategy use (see Muis et al.,
2015; Pekrun & Stephens, 2012).
Method
Participants
The sample included N = 438 university students from
the United States (n = 138, 116 females), Canada (n =
152, 114 females), and Germany (n = 148, 129
females). Sample size was defined by the availability
of participants. At each of the three sites, we recruited
as many participants as possible until the end of the
predefined time period for the study, and the data
of all participants were included in the analysis.
Mean age was M = 21.76 years (SD = 4.28 years), with
M = 20.09, 21.81, and 23.27 (SD = 1.72, 3.87, and 5.67)
for the United States, Canadian, and German
samples, respectively.
Procedure
The EES was administered as part of Muis et al.’s (2015)
study on epistemic beliefs and epistemic processes
when reading conflicting texts about climate
change. Being a controversial topic, this issue is par-
ticularly well suited to induce epistemic emotions.
The findings of a think-aloud study using the same
texts confirmed that this procedure elicited cognitive
conflict, and that the vast majority of the emotions
that occurred during reading the texts were epistemic
(Muis et al., 2015).
Participants first responded to a task value ques-
tionnaire. Next, they were confronted with four texts,
 4 R. PEKRUN ET AL.
which were adopted from Bråten and Strømsø (2009).
The first pair of texts presented conflicting information
on the causes of climate change (man-made vs.
natural), whereas the second pair presented conflict-
ing information on the consequences of climate
change (positive vs. negative). Prior to reading the
texts, participants were alerted to the conflicting infor-
mation contained in the texts (see Supplemental
Material). After each of the first three texts, partici-
pants completed the short 7-item version of the EES
to report the emotions they had experienced while
reading the texts. After the last text, participants com-
pleted the full 21-item version of the EES to report the
emotions they had experienced while reading all four
texts. Following this, they self-reported the frequency
with which they used cognitive and metacognitive
Downloaded by [93.204.107.184] at 23:20 22 July 2016
strategies to learn the material across the four texts.
Measures
Emotions
The long and short versions of the EES were used to
assess students’ epistemically-related emotions (see
Appendix).
Perceived task value and learning strategies
Perceived value of the task was assessed with seven
items adapted from Wigfield (1994; e.g. “In general, I
find learning about climate change very interesting”).
Four scales from the Motivated Strategies for Learn-
ing Questionnaire (MSLQ; Pintrich, Smith, Garcia, &
McKeachie, 1993) assessed participants’ use of
strategies during learning, including rehearsal,
critical thinking, elaboration, and metacognitive self--
regulation (4, 5, 6, and 12 items, respectively). For
both measures, participants responded on a 1 (not
at all true of me) to 7 (very true of me) scale, and
the scores were summed to form the task value
and strategy indexes (αs = .86, .59, .83, .72, and .69,
respectively).
Scale versions for German participants
The original English-language versions of the EES, the
Task Value Scale, and the MSLQ scales were translated
into the German language by two German emotion
experts and one bilingual translator.
Rationale for CFA
CFA using Mplus 7.3 (Muthén & Muthén, 2012) was
employed to test the internal validity of the 21-item
EES. To estimate parameters, the robust maximum like-
lihood estimator (MLR) was used which is robust to non-
normality of the observed variables. Emotions were
modelled as latent variables, and the scale items
served as manifest indicators. First, CFA was used to
examine the structural validity of the EES, including
competitive testing of different models representing
the seven emotions (Figure 1). Second, multi-group
CFA was employed to investigate measurement invar-
iance in a series of sequentially more restrictive
models fixing parameters to be equal across participant
groups from different countries. The fit of each model
was compared to the fit of the previous model. Specifi-
cally, we tested (a) a configural invariance model; (b) a
metric invariance model that restricted factor loadings
to be equal across groups; and (c) a scalar invariance
model that additionally restricted intercepts to be
equal (Byrne, 2008).
Results and discussion
Item and scale statistics
Item and scale statistics document that there was suf-
ficient variance in scores on all items and scales,
despite positive skewness for some of the scales (see
Supplemental Material, Tables S1 and S2). Further-
more, item-total correlations (part-whole corrected)
were above .53 for all items, and reliabilities were in
the α = .76 to .88 range (Table 1). These findings indi-
cate that the EES scales show sufficient variation and
that scale reliabilities range from good to excellent.1
Internal validity
Correlational analysis
Correlations between scores for the long EES scales
and scores for the short EES scales (averaged across
assessments) ranged from r = .65 to .83 for the same
emotions, suggesting convergent validity
(Supplemental Material, Table S5). Correlations
between scales for different emotions indicate that
the emotions measured by the EES are distinct from
each other, with most of the correlations ranging
from r = .10 to .40 (see Table 1 for the long scales
and Supplemental Material, Table S3, for the short
scales). However, the pattern of correlations diverges
from the relations found for other types of emotions.
Typically, positive emotions correlate positively, nega-
tive emotions correlate positively, and correlations
between positive and negative emotions tend to be

Downloaded by [93.204.107.184] at 23:20 22 July 2016
Figure 1. Structure of CFA models for epistemic emotions. Notes: Upper part: Model 1 (one-factor model); middle part: Model 2 (two-factor
model); lower part: Model 3 (seven-factor model). Item labels are su = surprise; cu = curiosity; jo = enjoyment; co = confusion; ax = anxiety; fr
= frustration; and bo = boredom.
negative (e.g. achievement emotions; Pekrun & Ste-
phens, 2012). Herein, all emotions except boredom
showed positive correlations, including surprise, posi-
tively valenced emotions (curiosity, enjoyment), as
well as negatively valenced emotions (confusion,
anxiety, frustration). As such, these correlations
suggest that emotions during epistemic activities are
primarily linked along the arousal dimension of
emotion rather than the valence dimension. In contrast
to deactivating boredom, epistemic surprise, curiosity,
enjoyment, confusion, anxiety, and frustration can be
considered activating emotions, and these emotions
correlated positively, regardless of their valence.
Confirmatory factor analysis
Preliminary support for the internal validity of the EES
was provided by exploratory factor analysis (see
Supplemental Material). To further examine the
COGNITION AND EMOTION 5
homogeneity and distinctiveness of the scales (long
versions), we used CFA and compared three models
that varied in their degree of differentiation between
emotions (Figure 1). Model 1 was a one-factor model
that contained one bipolar factor representing all
emotions. Model 2 was a two-factor model that differ-
entiated between positive and negative epistemic
affect. The items for curiosity and enjoyment were
used as indicators for positive affect, and the items
for confusion, anxiety, frustration, and boredom were
used as indicators for negative affect. In two variants
of this model, the items for surprise were used as indi-
cators of either positive affect (Model 2a) or negative
affect (Model 2b). Finally, Model 3 was a seven-factor
model that differentiated between all seven emotions.
The one-factor model did not fit the data;
Χ 2(210) = 2724.12, p < .01, comparative fit index
(CFI) = .383, Tucker-Lewis index (TLI) = .322, root
 6 R. PEKRUN ET AL.

Table 1. Correlations for Epistemic Emotion Scales, task value, and learning strategies.
1 2 3 4 5 6 7
Manifest and latent correlations between the emotion scalesa
1. Surprise .84** .52*** .37*** .44*** .39*** .21*** −.26***
2. Curiosity .58*** .88** .30*** .16** .32*** .09 −.62**
3. Enjoyment .44*** .32*** .78** .19*** .13** .06 −.12***
4. Confusion .53*** .17** .24*** .78** .46*** .45*** .14**
5. Anxiety .43*** .30*** .15* .58*** .76** .60*** −.08
6. Frustration .26*** .09 .07 .56*** .77*** .77** .16**
7. Boredom −.32*** −.72*** −.13* .15* −.05 .19** .86**
Bivariate correlations with task value and learning strategies
Task value .08 .25*** .18*** .01 .05 .06 −.15**
Rehearsal .20*** .24*** .19*** .03 .04 −.06 −.24***
Critical thinking .11* .34*** .17*** .11* .24*** .17*** −.19***
Elaboration .17*** .33*** .21*** .09 .19*** .12*** −.23***
Metacognitive self-regulation .27*** .34*** .29*** .22*** .20*** .07 −.20***
Partial correlations with task value and learning strategiesb
Task value −.06 .11* −.04 −.03 .04
Rehearsal .12* .14** .03 −.02 −.05
Critical thinking −.09 .08 .06 .15** .15**
Downloaded by [93.204.107.184] at 23:20 22 July 2016

Elaboration .00 .12* .05 .10* .11*

Metacognitive self-regulation .11* .21*** .17*** .10* .04
a
Latent correlations in lower-left diagonal matrix; manifest correlations in upper right diagonal matrix; alpha coefficients in the diagonal.
b
Partial correlations controlling for curiosity and boredom. These correlations were calculated to meet a concern that the bivariate correlations of
surprise, enjoyment, confusion, anxiety, and frustration with task value and learning strategies are driven by curiosity and boredom. The cor-
relations show that most of these relations are preserved when controlling for curiosity and boredom although reduced in size, which suggests
that these emotions have relations with value and strategies that are independent from the influence of curiosity and boredom.
*p < .05, **p < .01, ***p < .001.

mean square error of approximation (RMSEA) = .165,
and standardized root mean square residual
(SRMR) = .176. Fit for the two variants of the two-fac-
tor model was better but also not acceptable. Inter-
estingly, Model 2a (surprise as part of positive
affect) showed a slightly better fit than Model 2b
(surprise as part of negative affect), with Χ 2(188) =
2086.80, p < .01, CFI = .515, TLI = .458, RMSEA = .152,
SRMR = .164, and Χ 2(188) = 2252.39, CFI = .473, TLI
= .411, RMSEA = .158, and TLI = .171, respectively.
This is in line with the correlations for surprise
which indicated that surprise is primarily related to
curiosity and enjoyment (Table 1), and suggests
that surprise is positively valenced in an epistemic
context.
In contrast to the one-factor and two-factor
models, the seven-factor model showed a good fit;
Χ 2(168)= 418.38, CFI = .936, TLI = .920, RMSEA = .058,
and SRMR = .056. The factor loadings for the model
were above .60 for all items, and above .70 for 18 of
the 21 items (Supplemental Material, Table S1). This
finding confirms that the EES emotion factors can be
considered sufficiently homogenous.
Furthermore, the analysis provides estimates for
the latent correlations between the emotions (Table
1). The pattern of correlations is the same as the
manifest correlations, with positive correlations
between the activating emotions and a more differen-
tiated pattern of relations for boredom. Importantly,
although some of these correlations were relatively
high, they clearly indicate that all of the emotion con-
structs are separable, given that latent correlations are
corrected for unreliability and represent the highest
possible estimates of these relations. As such, the
results of CFA show that the seven emotions included
in the EES are distinct. These emotions would not be
adequately represented by summary factors of posi-
tive and negative affect; rather, they need to be
assessed in differentiated ways.
Measurement equivalence across countries
Multi-group CFA with the long version of the EES was
used to examine measurement equivalence of the
instrument across countries. The type of measurement
invariance needed to infer equivalence depends on
the purpose of measurement (Byrne, 2008). Metric
and scalar invariance are required if the goal is to
compare score distributions (e.g. means), whereas
metric invariance is sufficient to establish that the
measured constructs can be conceptualised in the
same way across groups. Accordingly, we sought to
establish metric invariance to ensure that the

emotions are interpreted in the same way across the
three countries.
To compare model fit for the configural, metric, and
scalar invariance models, we followed recommen-
dations by Chen (2007). To test metric invariance, a
change of ≥−.010 in CFI, supplemented by a change
of ≥.015 in RMSEA or a change of ≥.030 in SRMR
would indicate non-invariance. To test scalar invar-
iance, a change of ≥−.010 in CFI, supplemented by a
change of ≥.015 in RMSEA or a change of ≥.010 in
SRMR would indicate non-invariance. The configural
invariance and metric invariance models showed a
similar fit to the data (Supplemental Material, Table
S6). The differences between these models were
ΔCFI = −.003, ΔRMSEA = .001, and ΔSRMR = .019, indi-
cating that metric invariance could be accepted. The
Downloaded by [93.204.107.184] at 23:20 22 July 2016
differences between the scalar and metric invariance
models were ΔCFI = −.024, ΔRMSEA = .006, and
ΔSRMR = .001; as such, the loss in CFI suggests that
scalar invariance cannot be assumed, whereas the
differences in RMSEA and SRMR would imply that
scalar invariance can be accepted.
These findings confirm metric measurement
invariance of the EES across groups. For scalar invar-
iance, the evidence is mixed; this suggests that com-
parisons across groups that require scalar invariance,
such as comparisons of mean scores, should be used
cautiously. In the present research, we were inter-
ested in establishing structural measurement prop-
erties of the EES as well links with external
variables, rather than comparing emotion scores
across countries. The finding that the EES showed
metric invariance supports this approach and
suggests that the emotion constructs used in the
EES are conceptually equivalent across the three
countries.
External validity
Impact of contradictory texts on emotions
We expected participants’ emotions assessed by the
short scales to change across texts because the texts
contained conflicting information, with Text 1 and
Text 2 directly contradicting each other. Based on
the hypothesis that cognitive incongruity can
trigger surprise, curiosity, and confusion, we
expected these emotions to increase from Text 1
to Text 2. For Text 3, we expected these emotions
to return back to levels experienced during Text 1
given that no conflicting information was presented
in Text 3 compared to the first two texts. As
COGNITION AND EMOTION 7
predicted, surprise and confusion significantly
increased from Text 1 to Text 2, and then signifi-
cantly decreased from Text 2 to Text 3 (for details,
see Supplemental Material, Table S7 and Figure S1;
and Muis et al., 2015). Curiosity, enjoyment, and
anxiety decreased from Text 1 to Text 2, and sub-
sequently increased from Text 2 to Text 3. Frustration
and boredom significantly decreased from Text 2 to
Text 3.
These findings corroborate that reading conflict-
ing materials promotes surprise and confusion. Unex-
pectedly, curiosity was reduced during Text 2;
however, curiosity recovered during Text 3. Similarly,
Text 2 reduced participants’ enjoyment, which was
restored during Text 3; conversely, the negative
emotions confusion, anxiety, frustration, and
boredom decreased from Text 2 to Text 3. In sum,
the results confirm that emotions during epistemic
activities show a temporal dynamic that is aligned
with the amount of text-induced cognitive incongru-
ity, except for the decrease of curiosity from Text 1
to Text 2.
Relations with task value and self-reported
learning strategies
As expected, curiosity and enjoyment correlated
positively and boredom negatively with the per-
ceived value of the task (Table 1; Muis et al.,
2015). Furthermore, surprise, curiosity, and enjoy-
ment correlated positively with all four self-reported
learning strategies; anxiety correlated positively with
all of the strategies except rehearsal; confusion cor-
related positively with critical thinking and metacog-
nitive self-regulation; and frustration correlated
positively with critical thinking and elaboration.
These findings are in line with our expectation
that activating emotions can promote strategy use.
They are also consistent with recent evidence
suggesting that not only pleasant epistemic
emotions such as enjoyment, but also unpleasant
emotions such as confusion can promote learning
(D’Mello et al., 2014). By contrast, in line with evi-
dence that boredom undermines any systematic
effort at learning (e.g. Pekrun, Götz, Daniels, Stu-
pinsky, & Perry, 2010), boredom related negatively
to all four strategies. The relations were most pro-
nounced for curiosity and boredom, respectively,
thus highlighting the important role of these two
emotions in epistemic contexts. In sum, the findings
are consistent with our hypothesis that emotions
during epistemic activities are linked to value
 8 R. PEKRUN ET AL.
appraisals and strategy use during learning, thus
also attesting to the external validity of the EES.
Limitations
Due to the use of adjectives to measure emotion, it
cannot fully be ruled out that respondents use the
EES items to also report non-epistemic emotions
that can occur during epistemic activity, such as
achievement or social emotions. In a think-aloud
study with 56 Canadian undergraduate students
which used the same materials and procedure as
the current study, the nature of the emotions occur-
ring during reading the texts was examined (Muis
et al., 2015). The findings show that the vast
majority of the emotions (83%) were epistemic.
Downloaded by [93.204.107.184] at 23:20 22 July 2016
Nonetheless, scores derived from the EES should
be interpreted cautiously and with reference to
the specific instructions and epistemic context pro-
vided for answering the instrument. Also, the instru-
ment measures major epistemically-related
emotions but does not cover all types of these
emotions. Future research could expand upon the
present research by constructing additional scales
for emotions that occur less frequently in epistemic
contexts and are not included in the EES (e.g. awe
and admiration).
Conclusion
The present research shows that the EES are reliable
and can be used to examine a range of major
emotions that occur during epistemic activities. The
findings also contribute support for the internal
and external validity of the measure. Future research
should further validate the EES by including more
male participants, samples from other age groups
and cultures not considered here, and multiple
types of cognitive tasks, such as problem-solving
tasks or contradictory texts on less familiar topics
about which participants do not have existing
opinions. As compared with alternative methods
such as think-aloud protocols, physiological
measures, or behavioural observation, the instrument
is easy to administer, cost-effective, and minimally
task-invasive in terms of keeping interruption of
ongoing activities to a minimum. As noted at the
outset, epistemic emotions are critically important
for cognitive learning, problem solving, and the gen-
eration of knowledge, which suggests that research-
ers across disciplines should attend to these
emotions. The present findings proffer the EES as
one promising method that can be used to explore
these emotions.
Note
1. There were no gender differences in mean emotions
scores (long version of the EES) except for enjoyment,
Ms = 4.92 and 5.45, SDs = 2.04 and 2.25, for female and
male students, respectively; d = −.025, t = −2.04, p = .042,
Disclosure statement
No potential conflict of interest was reported by the authors.
Funding
This work was supported by the University of Munich [grant
number VII.1-H172.10], by the Social Sciences and Humanities
Research Council of Canada [grant number SSHRC 410-2011-
0182], and by a Visiting Fellowship Award from the Center for
Advanced Study, University of Munich, awarded to Krista Muis.
References
Bråten, I., & Strømsø, H. I. (2009). Effects of task instruction and
personal epistemology on the understanding of multiple
texts about climate change. Discourse Processes, 47, 1–31.
doi:10.1080/01638530902959646
Brun, G., Doğuoğlu, U., & Kuenzle, D. (Eds.). (2008). Epistemology
and emotions. Aldershot, UK: Ashgate.
Byrne, B. M. (2008). Testing for multigroup equivalence of a
measuring instrument: A walk through the process.
Psicothema, 20, 872–882.
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of
measurement invariance. Structural Equation Modeling: A
Multidisciplinary Journal, 14, 464–504. doi:10.1080/
10705510701301834
D’Mello, S. K. (2013). A selective meta-analysis on the relative inci-
dence of discrete affective states during learning with tech-
nology. Journal of Educational Psychology, 105, 1082–1099.
doi:10.1037/a00326749
D’Mello, S. K., & Graesser, A. C. (2012). Dynamics of affective states
during complex learning. Learning and Instruction, 22, 145–
157. doi:10.1016/j.learninstruc.2011.10.001
D’Mello, S. K., Lehman, B., Pekrun, R., & Graesser, A. C. (2014).
Confusion can be beneficial for learning. Learning and
Instruction, 29, 153–170. doi:10.1016/j.learninstruc.2012.05.003
Hookway, C. (2008). Epistemic immediacy, doubts and anxiety:
On a role for affective states in epistemic evaluation. In G.
Brun, U. Doğuoğlu, & D. Kuenzle (Eds.), Epistemology and
emotions (pp. 51–65). Aldershot, UK: Ashgate.
Jirout, J., & Klahr, D. (2012). Children’s scientific curiosity: In search of
an operational definition of an elusive concept. Developmental
Review, 32, 125–160. doi:10.1016/j.dr.2012.04.002
Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure, S. M.,
Wang, J. T. Y., & Camerer, C. F. (2009). The wick in the candle of
learning: Epistemic curiosity activates reward circuitry and
 COGNITION AND EMOTION 9

enhances memory. Psychological Science, 20, 963–973. doi:10.
1111/j.1467-9280.2009.02402.x
Markey, A., & Loewenstein, G. (2014). Curiosity. In R. Pekrun & L.
Linnenbrink-Garcia (Eds.), International handbook of emotions
in education (pp. 228–245). New York, NY: Taylor & Francis.
Morton, A. (2010). Epistemic emotions. In P. Goldie (Ed.), The
Oxford handbook of philosophy of emotion (pp. 385–399).
Oxford, UK: Oxford University Press.
Muis, K. R., & Duffy, M. C. (2013). Epistemic climate and epistemic
change: Instruction designed to change students’ epistemic
beliefs and learning strategies and improve achievement.
Journal of Educational Psychology, 105, 213–225. doi:10.
1037/a0029690
Muis, K. R., Pekrun, R., Azevedo, R., Sinatra, G., Trevors, G., Meier,
E., & Heddy, B. C. (2015). The curious case of climate change:
Epistemic emotions mediate relations between epistemic
beliefs, learning strategies and learning outcomes. Learning
and Instruction, 39, 168–183. doi:10.1016/j.learninstruc.2015.
06.003
Downloaded by [93.204.107.184] at 23:20 22 July 2016
Scheffler, I. (Ed.). (1991). In praise of the cognitive emotions and
other essays in the philosophy of education. New York, NY:
Routledge.
Scherer, K. R. (2009). The dynamic architecture of emotion:
Evidence for the component process model. Cognition and
Emotion, 23, 1307–1351. doi:10.1080/02699930902928969
Silvia, P. J. (2010). Confusion and interest: The role of knowledge
emotions in aesthetic experience. Psychology of Aesthetics,
Creativity, and the Arts, 4, 75–80. doi:10.1037/a0017081
Wigfield, A. (1994). Expectancy-value theory of achievement
motivation: A developmental perspective. Educational
Psychology Review, 6, 49–78. doi:10.1006/ceps.1999.1015
Appendix
Epistemically-Related Emotion Scales
Instruction

Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide: Seventh We are interested in the emotions you experienced when
edition. Los Angeles, CA: Muthén & Muthén. reading the texts about climate change. For each emotion,
Pekrun, R., Götz, T., Daniels, L. M., Stupinsky, R. H., & Perry, R. P. please indicate the strength of that emotion by circling the
(2010). Boredom in achievement settings: Exploring control- number that best describes the intensity of your emotional
value antecedents and performance outcomes of a neglected response during learning of the texts.
emotion. Journal of Educational Psychology, 102, 531–549.
doi:10.1037/a0019243 Not at all Very little Moderate Strong Very strong
Pekrun, R., & Linnenbrink-Garcia, L. (Eds.). (2014). International Curious 1 2 3 4 5
handbook of emotions in education. New York, NY: Taylor & Bored 1 2 3 4 5
Francis. Confused 1 2 3 4 5
Pekrun, R., & Perry, R. P. (2014). Control-value theory of achieve- Surprised 1 2 3 4 5
Interested 1 2 3 4 5
ment emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.),
Anxious 1 2 3 4 5
International handbook of emotions in education (pp. 120– Frustrated 1 2 3 4 5
141). New York: Taylor & Francis. Inquisitive 1 2 3 4 5
Pekrun, R., & Stephens, E. J. (2012). Academic emotions. In K. R. Dull 1 2 3 4 5
Harris, S. Graham, & T. Urdan (Eds.), APA educational psychol- Amazed 1 2 3 4 5
ogy handbook (Vol. 2, pp. 3–31). Washington, DC: American Worried 1 2 3 4 5
Psychological Association. Happy 1 2 3 4 5
Pintrich, P., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Muddled 1 2 3 4 5
Reliability and predictive validity of the Motivated Strategies Irritated 1 2 3 4 5
Monotonous 1 2 3 4 5
for Learning Questionnaire (MSLQ). Educational and
Excited 1 2 3 4 5
Psychological Measurement, 53, 801–813. doi:10.1177/ Astonished 1 2 3 4 5
0013164493053003024 Dissatisfied 1 2 3 4 5
Reeve, J. (1993). The face of interest. Motivation and Emotion, 17, Nervous 1 2 3 4 5
353–375. Joyful 1 2 3 4 5
Rozin, P., & Cohen, A. B. (2003). High frequency of facial Puzzled 1 2 3 4 5
expressions corresponding to confusion, concentration, and Not at all Very little Moderate Strong Very strong
worry in an analysis of naturally occurring facial expressions Note: Items for the seven-item short version of the EES are surprised,
of Americans. Emotion, 3, 68–75. doi:10.1037/1528-3542.3.1.68 curious, excited, confused, anxious, frustrated, and bored.

View publication stats