Isre 2022 1123

This article was downloaded by: [103.179.8.
80] On: 04 September 2022, At: 11:52

Publisher: Institute for Operations Research and the Management Sciences (INFORMS)
INFORMS is located in Maryland, USA
Information Systems Research

Publication details, including instructions for authors and subscription information:
http://pubsonline.informs.org
Could Gamification Designs Enhance Online Learning

Through Personalization? Lessons from a Field Experiment
Alvin Chung Man Leung, Radhika Santhanam, Ron Chi-Wai Kwok, Wei Thoo Yue
To cite this article:

Alvin Chung Man Leung, Radhika Santhanam, Ron Chi-Wai Kwok, Wei Thoo Yue (2022) Could Gamification Designs Enhance
Online Learning Through Personalization? Lessons from a Field Experiment. Information Systems Research
Published online in Articles in Advance 31 Mar 2022
. https://doi.org/10.1287/isre.2022.1123
Full terms and conditions of use: https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and-

Conditions
This article may be used only for the purposes of research, teaching, and/or private study. Commercial use
or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher
approval, unless otherwise noted. For more information, contact permissions@informs.org.
The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness
for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or
inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or
support of claims made of that product, publication, or service.
Copyright © 2022 The Author(s)
Please scroll down for article—it is on subsequent pages
With 12,500 members from nearly 90 countries, INFORMS is the largest international association of operations research (O.R.)
and analytics professionals and students. INFORMS provides unique networking and learning opportunities for individual
professionals, and organizations of all types and sizes, to better understand and use O.R. and analytics tools and methods to
transform strategic visions and achieve better outcomes.
For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org
INFORMATION SYSTEMS RESEARCH
Articles in Advance, pp. 1–23
http://pubsonline.informs.org/journal/isre ISSN 1047-7047 (print), ISSN 1526-5536 (online)
Could Gamification Designs Enhance Online Learning Through

Personalization? Lessons from a Field Experiment
Downloaded from informs.org by [103.179.8.80] on 04 September 2022, at 11:52 . For personal use only, all rights reserved.
Alvin Chung Man Leung,a Radhika Santhanam,b,* Ron Chi-Wai Kwok,a Wei Thoo Yuea
a
Department of Information Systems, College of Business, City University of Hong Kong, Hong Kong; b Division of MIS, Price College of
Business, University of Oklahoma, Norman, Oklahoma 73019
*Corresponding author
Contact: acmleung@cityu.edu.hk, https://orcid.org/0000-0001-8961-8357 (ACML); radhika@ou.edu,
https://orcid.org/0000-0002-1365-6071 (RS); isron@cityu.edu.hk (RC-WK); wei.t.yue@cityu.edu.hk,
https://orcid.org/0000-0002-1344-153X (WTY)
Received: July 3, 2020 Abstract. The COVID-19 pandemic has highlighted the importance of online learning. As
Revised: February 21, 2021; August 8, 2021; learner autonomy is relatively high in online environments, learners must engage in self-
December 12, 2021 regulated learning (SRL) to achieve optimal learning outcomes. Because most learners are
Accepted: January 15, 2022 unable to consistently engage in SRL, gamification interventions are being implemented to
Published Online in Articles in Advance: improve SRL engagement; however, mixed results cast doubt on the efficacy of this
March 31, 2022
approach. Massively open online courses (MOOCs), a type of online learning environment,
https://doi.org/10.1287/isre.2022.1123 are currently experiencing rapid growth due to widespread adoption by many institutions.
In MOOCs, there is no instructor intervention; hence, students have difficulty regulating
Copyright: © 2022 The Author(s) their own learning and are easily distracted. Therefore, this study investigates whether
mixed-research results regarding the efficacy of gamification can be attributed to lack of
attention to individual learner traits during design. For this purpose, the study analyzes
MOOCs as an instance of online learning by applying SRL theory and gamification princi-
ples. We altered a traditional MOOC platform to provide different types of gamified per-
formance feedback to facilitate learners’ SRL engagement. We then examined whether this
matched with goal orientation, an individual learner trait to influence SRL and learning
outcomes. Using learning-analytics tools, we tracked 760 college students’ SRL engage-
ment on a MOOC platform over five weeks. As theorized, SRL engagement and learning
outcomes of participants who had a strong performance-avoidance goal orientation
increased with positively framed performance feedback that involved no social compari-
sons; however, the same feedback had a negative impact on participants with a strong mas-
tery goal orientation. Our findings add to SRL theory by demonstrating that gamification
designs can enhance SRL engagement and learning outcomes in online learning, but with a
caveat—this occurs only when there is a match with learner traits—confirming the gamifi-
cation principle stating that task improvements and meaningful engagement can only
occur through thoughtful gamification design.
History: Raghu Santanam, Senior Editor; J. J. Hsieh, Associate Editor.

Open Access Statement: This work is licensed under a Creative Commons Attribution 4.0 International
License. You are free to copy, distribute, transmit and adapt this work, but you must attribute this
work as “Information Systems Research. Copyright © 2022 The Author(s). https://doi.org/10.1287/
isre.2022.1123, used under a Creative Commons Attribution License: https://creativecommons.org/
licenses/by/4.0/.”
Funding: Financial supported from the University Grants Committee’s (UGC’s) Special Grant for the Stra-
tegic Development of Virtual Teaching and Learning [Project Number 6430900], the UGC Teaching
and Learning Funding [Project Numbers 6391211 and 6391001], and the Digital Innovation Laboratory
of Department of Information Systems, City University of Hong Kong is gratefully acknowledged.
Supplemental Material: The online appendix is available at https://doi.org/10.1287/isre.2022.1123.
Keywords: self-regulated learning • gamification • learning goal orientation • MOOC • online learning
1. Introduction et al. 2016). Massively open online courses (MOOCs)

Online learning—also referred to as technology- are a popular, widely used example of online learn-
mediated learning (TML) and e-learning—is of deep ing. It was recently estimated that more than 101 mil-
interest to information systems (IS) researchers due to lion students across the globe learn through such
its extensive use for training employees and students platforms (Shah 2019). Furthermore, MOOCs are cur-
on information technology (IT) platforms (Alavi and rently being integrated into regular university degree
Leidner 2001, Gupta and Bostrom 2009, Santhanam programs in innovative ways (Littenberg-Tobias and
1
Leung et al.: Gamification and Self-Regulated Learning
2 Information Systems Research, Articles in Advance, pp. 1–23, © 2022 The Author(s)
Reich 2020). Because of the recent growth in the use facilitating these goal-directed activities (Butler and
of MOOCs and other online platforms among stu- Winne 1995, Zimmerman 2000). Therefore, gamifica-
dents, researchers are investing substantial efforts tion designs that incorporate objects like badges and
into identifying designs that can maximize learning leaderboards are appropriate interventions for improv-
success. Results from these studies, however, are ing SRL engagement, as they can draw learners’
mixed, and gamification designs do not always dem- attention to learning goals and provide performance
onstrate enhanced learning outcomes (e.g., Henderikx feedback in a fun manner (Santhanam et al. 2016, Anto-
et al. 2017, Reich and Ruipérez-Valiente 2019, and naci et al. 2019, Sailer and Homner 2019). However,
Huang et al. 2021). Using self-regulated learning such gamified designs have failed to deliver the
theory and gamification principles, we investigate expected improvements in learning outcomes, prompt-
whether differences in individual learner traits ing calls for additional investigation (Ding et al. 2017,
impact gamification outcomes and could offer explan- Kyewski and Krämer 2018). With the expectation of
ations for these mixed findings. seeing learning improvements, gamification designs
Although learners participating in online learning have been most frequently applied in the education sec-
environments may hold different motivations, person- tor (Majuri et al. 2018, Koivisto and Hamari 2019). Yet,
ality traits, learning experiences, and strategies, multi- research findings do not indicate that gamification has
ple studies have demonstrated that despite these a consistent, positive impact on learning. This inconsis-
differences, learners must engage in self-regulated tency must be addressed by scrutinizing each gamifica-
learning (SRL) strategies to achieve learning success tion design element to identify how it can improve
(e.g., Santhanam et al. 2008, Kizilcec et al. 2017, Kim learner engagement and outcomes (Cheong et al. 2014,
et al. 2018, and Jansen et al. 2020). In online learning Park et al. 2019).
environments, learners have substantial autonomy We focus on a key gamification design element—
over their learning process. As a result, they may not performance feedback—which studies have identified
be able to obtain similar levels of instructor-provided as integral to enhancing SRL, as it facilitates goal-
regulation and feedback or the encouragement and directed actions (Butler and Winne 1995, Mory 2004,
social learning affordances that are more readily avail- Hsieh and Cho 2011). However, the gamification per-
able in face-to-face environments (Kennedy 2014, sonalization principle states that a good match1
Alraimi et al. 2015). Therefore, many researchers find between gamification design elements and user traits
that SRL—which refers to the practice of learners is necessary to achieve meaningful engagement and
intentionally following and monitoring their goal- improve task outcomes (Liu et al. 2017). In other
directed learning activities—holds added significance words, the extent of match of gamified performance
in online learning and is critical to its success (e.g., feedback to individual learner traits is likely to impact
Zimmerman 2000, Santhanam et al. 2008, Eom 2015, SRL and learning outcomes and could explain the
and Wong et al. 2019). However, existing empirical inconsistent research results regarding gamified designs
studies have found that most learners are unable to in online learning. To better understand this effect, we
engage in and sustain SRL strategies in online learn- adapted a MOOC platform to enhance SRL via different
ing environments, resulting in subpar learning out- types of gamified performance feedback and examined
comes and high dropout rates (e.g., Santhanam et al. its impact on goal orientation, the learner trait that is
2008, Wan et al. 2012, Chang et al. 2013, and Jansen most relevant to SRL. Learner goal orientation is key to
et al. 2020). To tackle this problem, a variety of inter- SRL engagement because learners must set learning
ventions, including gamification, have been advanced goals, monitor their progress, and regulate their behav-
to promote the SRL engagement of learners in online iors to achieve their goals (Pintrich 2000, Schunk and
environments. However, the studies on gamification Zimmerman 2012). Furthermore, learner goal orientation
effectiveness have generated conflicting results that influences learners’ interpretations of performance feed-
currently have no clear explanation (Hamari et al. back and their subsequent cognitive resource allocation;
2014, Antonaci et al. 2019). Hence, we investigate the some studies have shown that goal orientation may
impact of the individual learner trait of goal orienta- influence how learners respond to gamification designs
tion on SRL engagement in gamified designs to deter- (VandeWalle and Cummings 1997, Hakulinen and
mine whether it could provide an explanation for Auvinen 2014).
these conflicting results. Per goal-orientation theory, learner goal orienta-
According to social cognitive theory, SRL engage- tions can be classified as mastery (i.e., developing
ment requires learners to intentionally pursue goal- competence by mastering challenging situations) or
directed activities and monitor their own learning per- performance goal orientation (i.e., validating one’s
formance (Bandura 1991, Pintrich 2000, Zimmerman competence by seeking positive judgments or avoid-
2000). It is described in terms of distinct phases, with ing negative judgments) (Dweck and Leggett 1988).
performance feedback playing an important role in The latter category can be further divided into
Information Systems Research, Articles in Advance, pp. 1–23, © 2022 The Author(s) 3
performance-approach and performance-avoidance goal environments are characterized by relatively high levels
orientations (Elliot 1999). These learner traits may influ- of learner autonomy and lower instructor or peer inter-
ence how learners respond to gamified performance actions, it is essential that learners engage in SRL in such
feedback, which can be provided in several ways. For environments. This is particularly true for MOOC online
example, social comparison uses leaderboards that show platforms, where there are no instructor interventions.
a learner’s performance relative to their peers, whereas In such contexts, learners feel isolated and face substan-
personal comparison shows a learner’s performance rel- tial challenges when practicing SRL (Kizilcec et al. 2017,
ative to their goals (Domı́nguez et al. 2013, Christy and Wong et al. 2019, Huang et al. 2021). Hence, it is an
Fox 2014). Another method for providing gamified per- appropriate context for investigating the efficacy of
formance feedback is message goal framing; that is, gamification designs in online learning.
framing a message either positively or negatively rela- We recruited participants from students enrolled in
tive to the goal (Levin et al. 1998, Van de Velde et al. an introductory course on management information
2010). A positively framed message emphasizes gains systems (MIS) at a university in Hong Kong. Students
(e.g., steps to obtain an award or badge), whereas a neg- were randomly assigned to groups that received dif-
atively framed message highlights losses (e.g., penalties ferent types of gamified performance feedback: per-
for not attaining a goal). According to goal-orientation sonal versus social comparison and positive versus
theory, learners with high performance-avoidance goal negative goal framing. For comparison, a control
orientation will likely perform better in response to experiment was conducted with participants of simi-
gamified performance feedback that uses positive fram- lar backgrounds who were studying on the same
ing and personal comparison, rather than negative fram- online platform, but did not receive gamified perform-
ing and social comparison, because these learners do not ance feedback. We made use of new learning-
like their poor performance to be revealed to peers. analytics methods (Pardo et al. 2017, Kim et al. 2018)
However, the strengths of these effects must be tested to capture participants’ online learning activities using
empirically. their digital traces on the platform to measure their
Building on theoretical understandings of SRL, goal SRL engagement and learning efficiency. As expected,
orientation, and gamification design principles, we our findings indicate that positively framed social
tested the match between learners’ goal-orientation comparison feedback matches well with participants
trait and variations in gamified performance feedback,
with a strong performance-approach goal orientation
such as the type of comparison (personal or social)
and enhances these participants’ SRL engagement and
and message framing (positive or negative), in an
learning efficiency. Positively framed personal com-
online learning environment. We then assessed the
parison feedback works effectively for participants with
impact of matching on SRL engagement, learning effi-
a strong performance-avoidance goal orientation.
ciency, and learning outcomes. Learning efficiency is
However, the same gamified feedback negatively impacts
particularly important in online contexts, as online
participants with a strong mastery goal orientation.
learners can be easily distracted; moreover, because of
As expected, we found that SRL is a significant media-
work and other commitments, today’s learners cannot
ting factor leading to better learning outcomes. Our
devote substantial amounts of time to learning (Cook
findings advance theoretical understandings of SRL in
et al. 2010, Khan and Madden 2016, Huang et al.
2021). Therefore, to comprehensively assess learning online learning and gamification designs and offer
success, we also evaluated the impact of the match explanations for the inconsistent results observed in
between learners’ goal orientations and gamified per- prior studies on gamification designs in MOOCs and
formance feedback on learning efficiency (measured other online platforms (e.g., Ding et al. 2017 and
by the pace of learning with the SRL process), along Kyewski and Krämer 2018). We demonstrate that
with SRL engagement and learning outcomes. gamification designs must consider the trait of learner
We developed the gamified performance feedback goal orientation to avoid negative impacts on task out-
on a MOOC platform using it as an instance of online comes and personalize to achieve the best online
learning. We chose a MOOC platform because it is learning outcomes.
growing in popularity; there is also a noted lack of stu-
dent engagement on MOOC platforms, and researchers 2. Research Background and Hypotheses
have called for more studies to improve SRL processes 2.1. SRL and Gamification in Online Learning
in online learning contexts (Antonaci et al. 2017, Reich SRL engagement is a series of intentional goal-directed
and Ruipérez-Valiente 2019, Wong et al. 2019). It should activities, strategies, and behaviors by learners that can
be noted that considerable research indicates that SRL facilitate their learning success (Zimmerman 2000). In
engagement by learners enhances learning outcomes in social cognitive theory, SRL engagement is described in
all learning environments (Zimmerman and Schunk terms of activities that occur in the following three
2013, Winne 2019); however, because online learning phases: forethought (in which learners set their learning
task goals and plans); performance and volitional control with some studies indicating no effect or even negative
(in which learners pay attention to their tasks using effects of the use of gamification (e.g., Landers et al.
learning strategies such as note-taking, recording per- 2017, Mekler et al. 2017, and Schöbel et al. 2017). Most
formance, and self-observing their learning progress to of these gamification studies bundle multiple gamifica-
optimize their efforts); and self-reflection (in which learn- tion design elements together, treating gamification as
ers reflect on and evaluate their learning progress a generic unified construct and providing a conclusion
against their goals and may adapt their learning strat- as to whether it was successful. However, to advance
egies) (Zimmerman 2000). Self-reflection could lead to the understanding of gamification design effectiveness,
the revision of strategies that influence forethought in it is critical to systematically study the impact and
subsequent learning; therefore, SRL engagement is nuances of each element for specific tasks (Landers et al.
described as a cyclical process. Research studies pro- 2017, Liu et al. 2017, Schöbel et al. 2017, Kyewski and
vide a library of activities in each of these phases that Krämer 2018, Majuri et al. 2018, Schmidt-Kraepelin et al.
can be checked to measure the extent to which learners 2018). Given the mixed nature of previous findings, we
are engaging in SRL2 (e.g., Butler and Winne 1995, San- unpack and study one element—namely, gamified per-
thanam et al. 2008, and Schunk and Zimmerman 2012). formance feedback—to better understand how it sup-
Although learners may engage differently and put ports learners’ SRL engagement and learning efficiency
more or less effort into each activity or phase, the pres- to improve learning outcomes.
ence of more of these identified activities shows greater To understand the perspectives and gaps in gamifi-
SRL engagement of the learner, whereas the presence cation studies in IS, we briefly reviewed the research
of fewer activities indicates less SRL engagement. published in the past five years in the Senior Scholars’
In online learning, learners must exercise and main- Basket of Eight journals. Studies focusing on online
tain SRL engagement because they typically do not platforms such as MOOCs are practically nonexistent,
have autonomous control over their learning and gen- excepting for one recent paper (see Table A1 of Online
erally receive little to no prompting from instructors Appendix A). A recent study highlights that procrasti-
(e.g., Santhanam et al. 2008, Wan et al. 2012, Kuo et al. nation by learners is high in MOOC online environ-
2014, and Eom 2015). Research suggests that learners ments and proposes interventions to reduce these
may engage in various activities—such as goal setting, occurrences (Huang et al. 2021). Please note that pro-
time or effort management, and cognitive strategies— crastinations result in low SRL engagement (Wolters
that are related to the three phases of SRL, even 2003, Cheng and Xie 2021). Another study demon-
though they are not aware of SRL (Hu and Gramling strates that it is particularly important to design
2009). However, because all learners do not consis- timely and adequate performance feedback using
tently and fully engage in SRL, major research efforts technology artifacts, so as to address requirements for
have been undertaken to increase the extent of SRL massified education (Piccoli et al. 2020). A study on
engagement in online learning (Santhanam et al. 2008, gamified e-learning concluded that a one-size-fits-all
Bol and Garner 2011, Jansen et al. 2020). Interestingly, approach is not appropriate and set forth possible
although learners’ SRL engagement is low in online study designs to address how the impact of each
environments, it is these IT platforms that provide game element might be tested (Santhanam et al. 2016).
affordances such as gamified performance feedback A few other studies on online learning are summar-
through which learners can become aware of their ized in Table A1 of Online Appendix A. Outside the
progress and thus potentially revise their strategies to domain of online learning, gamification research in IS
achieve learning goals. In fact, online learning plat- has been applied in several areas, including knowl-
forms could be more effective than traditional class- edge sharing, training in phishing prevention, and
rooms for stimulating SRL because the technology health and exercise management, indicating that
artifacts can provide regular feedback, support all gamification is a useful approach to inducing user
three SRL phases, and prompt learners to fine-tune engagement (see Table A2 of Online Appendix A). A
their learning strategies (Narciss et al. 2007, Chang commentary on gamification lists several principles
et al. 2013). Through the affordances of the technology for achieving gamification design success and high-
artifacts, gamification design elements that support lights the importance of matching gamification design
performance feedback (e.g., points, badges, and lead- elements to user traits (Liu et al. 2017). In other related
erboards) are often used to boost learners’ activities disciplines of gamified learning (see Table A3 of
and enhance their SRL and learning outcomes Online Appendix A), some studies have documented
(Dicheva et al. 2015, Landers et al. 2015, Majuri et al. the positive effect of gamification on learning per-
2018). formance, but some have shown mixed or even nega-
Some gamification designs to improve SRL engage- tive effects. In general, researchers have concluded
ment have demonstrated a positive influence on engag- that gamification cannot prompt all learners to deeply
ing learners; however, such effects are not universal, engage in learning unless gamification design is
person-specific (van Roy and Zaman 2018). Taken not like social comparisons. Game elements that show
together, although recent research speaks to the value personal comparisons (e.g., posting the percentage of
of improving gamification designs in online learning, modules that the learner has completed) provide pro-
no studies have attempted to address the role of gressive information to users, with which they can
learner traits and individual gamification design ele- evaluate their learning strategies, whereas social com-
ments in improving SRL engagement, which is a key parisons through game elements (e.g., leaderboards
aspect of online learning (Pintrich 2000, Santhanam and loserboards) offer observational or vicarious rein-
et al. 2008, Wan et al. 2012, Liu et al. 2017). We address forcements that indicate the affirmation or disaffirma-
this issue by studying whether the interaction tion of behaviors exhibited by peers (Christy and Fox
between gamified performance feedback and individ- 2014, Landers et al. 2017).
ual learner traits of goal orientation affects SRL Whether gamified performance feedback is given in
engagement and learning performance in MOOCs. the form of personal or social comparisons, the fram-
ing of the message could be changed to affect how
2.2. Gamified Performance Feedback and Goal learners respond. Message-framing techniques— that
Orientation Consideration is, the presentation of logically equivalent options in
On online learning platforms, gamified performance semantically different ways—can be used to persuade
feedback in the form of badges and leaderboards is users to act on a particular message by framing it in
quite popular. We built and extended these feedback either a positive or negative format (Levin et al. 1998,
options in an SRL theory-driven manner by altering Krishnamurthy et al. 2001). One method of accom-
the comparison type and framing of gamified plishing this, known as goal framing, is to present
performance-feedback messages to study how they messages relative to goals (Van de Velde et al. 2010).
influenced learners with different traits of goal orien- Positively goal-framed messages emphasize obtaining
tation. Performance feedback can be delivered in dif- gains, whereas negatively goal-framed messages high-
ferent ways and in line with the gamification princi- light losses that might accrue as a consequence of not
ple, which states that comparison type and message achieving goals (Lindenberg and Steg 2013, Schmidt-
format could be tailored to users to make the feedback Kraepelin et al. 2018). Therefore, in positive goal fram-
more persuasive (Schmidt-Kraepelin et al. 2018, Zhou ing, learners’ progress and achievements are high-
et al. 2020). One type of gamified performance feed- lighted, and positive consequences (e.g., providing a
back is social comparison. Through leaderboards, this master badge for completing a learning module and
type of gamified performance feedback highlights a highlighting the steps required to reach the next mile-
learner’s progress relative to that of other learners stone) are emphasized with the expectation that they
(Dicheva et al. 2015, Sailer and Homner 2019). Indi- will generate positive actions among users (Roney
vidual learners’ goal orientations may impact how and Lehman 2008). Conversely, negative goal framing
they respond. Per goal-orientation theory (Elliot and emphasizes negative consequences (e.g., earning a
Church 1997), learners with a strong performance- loser badge and highlighting an incomplete learning
approach goal orientation may prefer to see social module), with the expectation that users will want to
comparisons on leaderboards. However, those with a avoid incurring such consequences in their learning.
strong performance-avoidance goal orientation are Whereas positive goal framing reminds users of the
anxious about showing ignorance in front of their rewards associated with status and affirmation (Antin
peers and, thus, may prefer not to see their progress and Churchill 2011), negative goal framing uses infor-
on the leaderboards. In particular, learners with a mational feedback to prompt users to develop better
strong mastery goal orientation do not pay attention learning strategies by highlighting penalties that
to scores. Studies indicate that some learners favor could occur as a result of inaction (Krishnamurthy
such social comparison on leaderboards, but others et al. 2001). The effectiveness of goal framing depends
do not (Christy and Fox 2014, Lim and Yang 2015). If on the match between learners’ goal orientations and
learners do not like or pay attention to leaderboards, situationally induced goals (Sagiv and Schwartz
the very purpose of gamified performance feedback is 2000). As described above, our study addresses two
defeated. dimensions of performance feedback (see Figure 1):
Another type of gamified performance feedback is type of comparison (social versus personal) and mes-
personal feedback that shows learners’ performance sage framing (positive versus negative goal framing).
on the learning task as measured against a goal According to goal-orientation theory, the individual
(Cheong et al. 2014, Mekler et al. 2017). This type trait of goal orientation indicates that learners pos-
of personal comparison feedback, which highlights sess highly different learning motivations that are
personal accomplishments and progress without mak- influential in their learning performance, particularly
ing peer comparisons (e.g., meeting a deadline and in online contexts (e.g., Ames 1992, Elliot et al. 2002,
earning a badge), may be suitable for learners who do Zweig and Webster 2004, and Santhanam et al. 2008).
Figure 1. Experimental Treatments—Gamified Performance Feedback

The trichotomous goal-orientation framework classifies requires that learners pay complete attention to learn-
individual goal orientations as mastery,3 performance- ing goals by screening out distractions and applying
approach, or performance-avoidance (Elliot and Har- optimal learning strategies, their learning efficiency
ackiewicz 1996, Elliot and Church 1997, Zweig and and effectiveness should be high (Zimmerman 2000).
Webster 2004). Learners with a strong mastery goal ori- Learning efficiency as measured by the pace of mod-
entation care primarily about competency and place ule completion does not imply shallow learning, as
less emphasis on scores and performance relative to learners who engage in SRL undergo the cyclical stra-
others (Ames 1992). In contrast, learners with a per- tegic thinking process before, during, and after taking
formance goal orientation take a normative perspective. part in various learning activities, and they are consis-
Those with a strong performance-approach goal orien- tent for maintaining pace with SRL (Järvelä et al.
tation focus on attaining a favorable judgment of com- 2016). This is particularly important in online environ-
petency (Midgley et al. 2001), whereas those with a ments (Kolfschoten et al. 2010).
strong performance-avoidance goal orientation focus Learners with a strong performance-approach goal
on avoiding being embarrassed in front of others (Dar- orientation care about whether they are perceived as
non et al. 2007). Goal orientation is an important indi- competent and tend to demonstrate the ability to earn
vidual trait that is relevant to user interactions with favorable judgment from others (Elliot et al. 2002).
technology, such as the use of web-based IS (Yi and Previous research shows that those learners strive for
Hwang 2003) and acceptance of e-learning systems public approval by demonstrating their competence
(Santhanam et al. 2008, Wan et al. 2012). through achievements (Lee et al. 2003). To maintain
their high achievements, they are likely to engage in
rigorous and persistent study behaviors (Elliot and
2.3. Hypotheses Development
Church 1997). Therefore, gamified and positively framed
With SRL engagement, learners can actively seek to
social comparison performance feedback that displays
reduce a discrepancy between a goal state and their
their achievements in front of others may motivate them
current level of performance by planning and moni-
to be more engaged. We thus posit:
toring their progress; therefore, gamified performance
feedback can be highly useful in this process (Cleary
and Zimmerman 2012). To facilitate SRL engagement, Hypothesis 1 (a). Positively framed (positive) social com-
gamified performance feedback can be given using a parison feedback encourages learners with a strong perform-
social or personal comparison approach and positive ance-approach goal orientation to engage in SRL;
or negative goal framing. However, because learners and
possess different goal orientations, their responses to
Hypothesis 1 (b). Positively framed (positive) social com-
gamified performance feedback will vary. We
parison feedback helps learners with a strong performance-
expected that a match between gamified performance
approach goal orientation develop higher learning efficiency.
feedback and learners’ traits would improve learners’
SRL engagement and increase their learning efficiency Learners with a strong performance-avoidance goal
(Henderikx et al. 2017). Because SRL engagement orientation worry about negative judgments and tend
to avoid situations that show normative incompetence Hypothesis 4. SRL mediates the influence of the match
(Elliot and Harackiewicz 1996). With low self- between gamified performance feedback and goal orientation
perceived abilities and a strong fear of failure, these on learning outcomes.
learners tend to avoid challenging or difficult tasks
We present our hypotheses and research model in
(Dweck and Leggett 1988). Because they place exces-
Figure 2.
sive weight on the possibility of failure, uncertainty in
their learning may make them even more concerned
about their performance (Darnon et al. 2007). To alle-

3. Research Method
3.1. Overview
viate such concerns, personal feedback with positive
To examine whether the match between learner goal ori-
goal framing can help learners focus on positive
entations and gamified performance feedback impacts
aspects and provide them with clear guidance in
SRL and learning outcomes in TML, we implemented a
learning. Therefore, we posit:
longitudinal field experiment to track participants’ SRL
Hypothesis 2 (a). Positively framed (positive) personal engagement and learning efficiency over time. We chose
comparison feedback encourages learners with a strong per- a MOOC platform4 as our test environment and meas-
formance-avoidance goal orientation to engage in SRL; ured learners’ activities over the course of five weeks.
and The development of the design was theory-driven so
that learners could cycle through the stages of fore-
Hypothesis 2 (b). Positively framed (positive) personal
thought, performance and volitional control, and reflec-
comparison feedback helps learners with a strong performance-
tion during the SRL process (see Online Appendix B for
avoidance goal orientation develop higher learning efficiency.
screenshots). Our research participants were university
Learners with a strong mastery goal orientation are students enrolled in an introductory MIS course, and the
intrinsically motivated to improve their knowledge experiment was integrated as a class activity. As pro-
and skills and care less about scores and performance posed in recent learning-analytics research, the digital
relative to others (Ames 1992). To achieve their mas- traces of learning activities in the three SRL phases were
tery goals, learners are active in monitoring their captured and used to derive students’ SRL engagement
learning progress and engage in cognitive learning scores. Self-regulated learning engagement, learning effi-
strategies (Pintrich 2000). Feedback on their learning ciency, and learning outcomes served as dependent
performance motivates them to find ways to further measures, whereas the gamified treatments were consid-
improve themselves. Negative feedback is valuable to ered independent measures. For hypothesis testing, we
these learners because it can help them identify inef- ran regressions and identified the impact of the interac-
fective learning and substandard learning perform- tion terms of learner traits and gamified treatment on
ance (VandeWalle and Cummings 1997). Personal the dependent measures. A significantly positive (nega-
feedback with negative goal framing can pinpoint tive) coefficient estimate of the interaction term indicates
these individuals’ shortcomings. Such feedback can that there is a match (mismatch) between a learner’s
trigger individuals with strong mastery goal orienta- goal orientation and type of gamified performance feed-
tions to pay attention to the inadequacy of their learn- back. We chose regression because it is suitable for test-
ing strategies so that they can rectify them and ing moderators and accounts for heterogeneity among
improve their learning skills. Thus, we posit: experimental subjects (Becker et al. 2013).
Hypothesis 3 (a). Negatively framed (negative) personal
3.2. Learning Course
comparison feedback encourages learners with a strong
The online course comprised five learning modules,
mastery goal orientation to engage in SRL;
each of which provided video training and lecture
and slides covering the core knowledge and skills related
to social analytics and business intelligence. As part of
Hypothesis 3 (b). Negatively framed (negative) personal
the learning outcomes, participants were expected to
comparison feedback helps learners with a strong mastery
know how to use two software programs—Facepager
goal orientation develop higher learning efficiency.
and SAS Enterprise Miner—to download Facebook
Learners’ engagement in SRL leads to better learning posts and conduct a simple text topic analysis, respec-
outcomes because users set learning goals; adopt strat- tively. Participants were told at the beginning of the
egies to achieve these goals; and go through the phases experiment that they would not be directly tested on
of forethought, performance and volitional control, and the skills learned in each module on the midterm or
reflection in a motivated manner (Zimmerman 2000, final exam. However, they were required to apply
Santhanam et al. 2008, Landers et al. 2015). The media- what they had learned in a group project. They were
ting role of SRL is important in the learning process. informed that, after five weeks of self-learning, there
Therefore, we posit as a confirmation: would be a knowledge and performance test, and top
Figure 2. (Color online) Conceptual Model

Note. H, hypothesis.
performers would be rewarded.5 As part of a flipped goals of the majority of students (Maehr and Zusho
classroom pedagogy (Mok 2014), the materials were 2009) and are important elements of the trichotomous
not discussed in lectures during the experiment achievement-motivation framework that is used widely
period, but were revisited in lectures after the test. in IS research on TML (e.g., Santhanam et al. 2008, Wan
Because it was an introductory course, the majority of et al. 2012, and Santhanam et al. 2016).
participants were first-year students (68% freshmen
and 32% transfer students). As demonstrated by a 3.4. Background and Demographics
pre-experiment survey and test, participants did not Our pre-experiment survey (Online Appendix D) also
have a significant amount of prior knowledge on collected participants’ IT expertise, demographic
related topics or the software, with a mean pre- information, and knowledge on the course topic via a
experiment test score of 2.74 out of 7. knowledge test. Based on the survey and test, partici-
3.3. Goal-Orientation Classification pants7 were placed into different treatment groups,
To identify participants’ goal orientation, we obtained such that their averages on the pre-experiment knowl-
the mastery, performance-approach, and performance- edge test, IT expertise, goal orientations, and other
avoidance goal orientations6 of each participant using demographic variables were similar. Furthermore, we
the measures proposed by Midgley et al. (2000). Online recruited participants for our control group who
Appendix F shows internal consistencies along with con- received no gamified performance feedback.8 The
vergent and divergent validity testing results for the demographic information of the treatment and control
three goal orientations. We focused on these three goal groups and pre-experiment survey results are shown
orientations because they represent the primary learning in Table G2 of Online Appendix G.
3.5. Self-Regulated Learning Engagement and completed) in each module. They also received tips
Learning Outcomes for obtaining master badges (avoiding loser badges).12
According to Zimmerman (2000), SRL comprises three Table 2 lists all the badges that participants could
major phases: forethought, performance and voli- potentially receive. For the positive social comparison
tional control, and self-reflection. We recorded and (negative social comparison) treatment group, we con-
analyzed actual learners’ activities on the MOOC plat- structed a leaderboard (loserboard) and helpfulness
form to measure their SRL engagement, rather than board (miser board). The leaderboard (loserboard)
asking them to reflect on their activities through a sur- showed the names and scores of the highest (lowest)
vey, as has been done in previous studies (e.g., San- 20 daily cumulative scores.13 We used a randomization
thanam et al. 2008 and Wan et al. 2012). From the strategy and repeated the procedure multiple times
recorded user log, we identified standard activities until the F-tests showed that the participants across the
used in prior research to describe the three SRL four treatment groups were similar. Details of the
phases, as shown in Table 1. experimental procedure are shown in Figure 3. We
For the forethought phase, we focused on goal set- assumed that stable unit treatment value assumption
ting and planning, where we identified the extent to (SUTVA) was valid where potential social interactions
which participants developed study plans (see Figure among participants had little effect on the treatments
B2 in Online Appendix B). For performance and voli- assigned to participants (Mithas et al. 2014).
tional control, indicators included proactive efforts Each treatment group received a specific gamified
related to self-control and self-observation (Zimmer- email, as shown in Online Appendix C. Both treat-
man 2000, Santhanam et al. 2008). Hence, activities ment and control groups received the following stand-
that showed attention and self-revision—such as ard elements through the MOOC system: (a) a dead-
reading lecture slides, watching videos, taking notes, line for completing a module, (b) cumulative scores
seeking help in forums, and self-evaluation through based on learning activities, (c) helpfulness scores
volitional quiz-taking—were all included. At the end based on the number of “up-votes” and “down-votes”
of each module, quizzes were available, through received, and (d) master and loser badges obtained.
which participants could conduct self-evaluation on
their learning performance.9 For the reflection phase, 3.7. Propensity Score Matching
participants were asked to write a self-reflection essay To alleviate sampling bias in the treatment and control
to evaluate their learning strategies and provide sug- groups (Dehejia and Wahba 2002), we conducted pro-
gestions to improve their study plans. Engagement in pensity score matching (PSM) to match each participant
these activities in each phase was calibrated to indi- in the treatment group to another participant in the con-
cate learners’ SRL engagement.10 trol group, such that the participants across groups were
Table 1 shows the details of SRL, which provided similar in terms of goal orientations, demographics, and
the measure of SRL engagement. Learning efficiency, pre-experiment knowledge test scores (see Online
as captured by the pace of module completion with Appendix G). In conducting the PSM, sampling with
the SRL process, was measured by the time a participant replacement or without replacement could be applied.
took to complete a module and all three SRL phases. To We chose the former primarily because it ensures sam-
determine learning outcomes, participants took a postex- ple independence (Raj and Khamis 1958). Previous IS
periment test11 (Online Appendix E) at the end of the research use the same approach to approximate the true
self-study period. The test comprised knowledge and population distribution (Bapna and Umyarov 2015).
performance sections and was created in accordance After PSM, the treatment and control participant
with prior IS research on learning and training, which ratio was 1:1. The distribution of the propensity score
evaluated learners’ knowledge and skills in using spe- as shown in the box and smooth histogram plots (Fig-
cific technologies (Yi and Davis 2003, Santhanam et al. ures G1 and G2 in Online Appendix G) confirms that
2008). the matched control group closely resembles the treat-
ment group. The Kolmogorov-Smirnov test also
3.6. Gamified Performance-Feedback Treatments shows that the distributions of the propensity score of
Our two-by-two treatments included personal and the treatment and matched control groups are similar,
social comparison feedback and positive and negative with a p-value of 0.756. We reported the summary sta-
goal-framing feedback. Figure 1 shows the four types tistics of all participants in both treatment and control
of gamified feedback treatments: positive personal groups before and after PSM in Table G2 of Online
comparison, negative personal comparison, positive Appendix G.
social comparison, and negative social comparison. In
the positive personal (negative personal) feedback 3.8. Manipulation Check
treatment group, participants were reminded of the To ensure that participants correctly perceived our
percentage of tasks they had completed (or not yet gamified treatments, we asked all participants several
Table 1. Measurements of Self-Regulated Learning Engagement 10
Positive Negative Negative

personal Positive social personal social
comparison comparison comparison comparison ANOVA F-test
treatment treatment treatment treatment Control statistic
SRL phase Activities and brief explanationa (N 97) (N 106) (N 110) (N 106) (N 419) (p-value)
Forethought Set up a study plan (please see Figure B2 in 0.83 (0.32) 0.76 (0.37) 0.78 (0.35) 0.76 (0.37) 0.77 (0.33) 0.78 (0.54)
Online Appendix B) for each learning module
out of all modules attempted (to better
understand learners’ engagement in task
analysis)
Performance Ratio of learning modules that meet scheduled 0.09 (0.16) 0.07 (0.19) 0.11 (0.21) 0.11 (0.23) 0.22 (0.30) 12.37 (0.00)
and study plans (to understand whether there is a
volitional gap between desired and actual performance)
control Number of revised study plans per learning 1.09 (0.59) 1.00 (0.53) 1.15 (0.62) 1.02 (0.53) 1.01 (0.41) 2.07 (0.08)
module attempted (to measure how learners
monitor their progress and self-control and
adjust their learning activities if there is a gap)
Number of questions asked in class forums 0.01 (0.10) 0.04 (0.19) 0.05 (0.21) 0.11 (0.59) 0.15 (0.62) 2.75 (0.03)
(please see Figure B3 in Online Appendix B) (to
measure help-seeking activities as a part of self-
observation and self-control)
Number of discussions initiated in class forums 4.68 (4.70) 3.75 (4.59) 4.44 (4.78) 3.70 (4.12) 1.40 (2.44) 30.84 (0.00)
(please see Figure B4 in Online Appendix B) (to
understand whether learners secure necessary
help from external sources to continue their
learning process)
Average number of quizzes attempted in each 2.42 (1.06) 2.41 (1.12) 2.51 (1.03) 2.41 (1.10) 2.31 (1.25) 0.81 (0.52)
module (please see Figure B7 in Online
Appendix B) (to measure whether learners
engage in cognitive monitoring by taking
quizzes)
Number of notes taken (please see Figure B8 in 1.09 (4.11) 0.73 (3.59) 1.12 (4.54) 1.68 (8.78) 0.76 (4.14) 0.83 (0.51)
Online Appendix B) (to measure whether
learners monitor their learning progress
cognitively)
Number of notes edited (please see Figure B9 in 0.01 (0.10) 0.04 (0.27) 0.10 (0.96) 0.04 (0.31) 0.01 (0.11) 1.35 (0.25)
Online Appendix B) (to measure the self-control
process of learners)
Reflection Submitting self-evaluation and plans for further 2.25 (1.69) 2.08 (1.72) 2.14 (1.70) 1.89 (1.62) 1.40 (1.33) 11.4 (0.00)
improvement (please see Figure B10 in Online
Appendix B) out of all modules attempted (to
measure metacognitive monitoring of learners’
learning progress and how learners propose
new ways to adapt their learning strategies)
Information Systems Research, Articles in Advance, pp. 1–23, © 2022 The Author(s)
difference across groups. One possible reason is the mismatch of gamified performance feedback and learners’ goal orientation, which leads to significantly negative change in SRL. Resultantly,
SRL explanations are based on Butler (2002), Butler and Winne (1995), Ryan et al. (2001), Santhanam et al. (2008), Schunk and Zimmerman (2012), and Zimmerman and Schunk (2008), and test
Note that the means of knowledge test and performance test in the treatment groups are smaller than those in the control group. The ANOVA test also shows the existence of significant
manipulation check questions after the experiment.
ANOVA F-test
4.21 (0.00)
8.29 (0.00)
1.72 (0.14)
(p-value)
As shown in Online Appendix H, the results indicate
statistic that our treatments were correctly perceived.
3.9. Empirical Model

To test Hypothesis 1(a), Hypothesis 2(a), and Hypoth-
esis 3(a), we developed a model with the SRL engage-
26.09 (20.61)
−0.11 (0.74)
14.11 (6.67)
(N 419)
Control
ment measure as a dependent variable. As shown in

Equation (1), we modeled the impact of the gamified
performance-feedback treatment, individual goal ori-
entation, interaction between the treatment and goal
orientation, and other control variables (Control) on
SRL.14 Pooled regression performed better than sub-
22.01 (18.75)
comparison
0.04 (0.89)
11.84 (5.83)
treatment
(N 106)
Negative
social
sample analysis here because the latter involves

dichotomizing continuous variables (e.g., goal orienta-
Note. Numbers without parentheses are averages, and numbers with average are standard deviations in the four treatment and control groups.
tion), which may lead to problems like data truncation

(Maxwell and Delaney 1993, MacCallum et al. 2002).
Our tests of hypotheses were determined by the
26.09 (19.11)
comparison
explanatory variable Gamified Feedback × Goal. Table 3

0.17 (0.94)
12.18 (5.65)
treatment
(N 110)
Negative
personal
describes the notations used in Equation (1).

SRL β0 + β1 Gamified Feedback + β2 Goal
+ β3 Gamified Feedback × Goal + β4 Control + ε:
(1)
Positive social
21.72 (17.41)
comparison
0.03 (0.91)
10.71 (5.95)
treatment
(N 106)
Hypothesis 1(b), Hypothesis 2(b), and Hypothesis 3(b)

evaluated learning efficiency, which is measured by
the pace of module completion with SRL. We tested
this dynamic process using a Cox proportional-
hazards model15 (Austin 2017), which is a survival
24.61 (21.20)
comparison
model examining how specific factors influence the

0.18 (0.82)
12.63 (5.82)
treatment
personal
(N 97)
Positive
rate at which a particular event occurs over time. In

our research, the event in question is the completion
of a learning module with three SRL phases. We
studied the activities of 838 participants (treatment
and matched control) over the course of 35 days, as in
principal component of seven performance and
volitional control activities, and one reflection
Equation (2). A positive and significant coefficient

Performance test (to measure skill acquisition)
SRL engagement is measured by the principal
results are based on Santhanam et al. (2008) and Yi and Davis (2003).
component of one forethought activity, the
knowledge and conceptual understanding)
estimate implies a hazard ratio greater than one—in

Activities and brief explanationa
other words, a faster pace of module completion with

Knowledge test (to measure declarative
SRL and higher learning efficiency.

h(t) h0 (t) exp(β1 Gamified Feedback + β2 Goal
+ β3 Gamfied Feedback × Goal + β4 Control): (2)
the lower SRL leads to poorer test performance.
4. Empirical Results
We first investigated the impact on SRL engagement
of a match between goal orientation and gamified per-
activity
formance feedback combinations. These results are

shown in Table 4, with key results highlighted.16 We
Table 1. (Continued)
found that the coefficient estimate of Positive Social

Comparison × Performance-Approach was positive and
significant (β 0.4540, p < 0.001), whereas those of
Test resultsb
other interaction terms and Performance-Approach were

SRL phase
not at all significant. This result provides support for

Hypothesis 1(a), which hypothesized that gamified
SRL
positive social comparison feedback would encourage

a
Table 2. Master and Loser Badges Available
Badge Titles Explanation
Master badges
Leader of the Week Top 20 who score the highest in the class
Module Expert Complete a module with 100% accuracy in an end-of-module quiz
Fast Module Learner Top 100 students who are the fastest to complete a module with at least 70% accuracy in an end-of-module
quiz
Module Explorer Complete Modules 4 and 5 with at least 50% accuracy
Course Master Complete all compulsory modules with at least 70% accuracy in end-of-module quizzes
Course Master Plus Complete all compulsory and voluntary modules with at least 70% accuracy in end-of-module quizzes
Course Expert Complete all compulsory modules with 100% accuracy in end-of-module quizzes
Course Expert Plus Complete all compulsory and voluntary modules with 100% accuracy in end-of-module quizzes
First Post Create first post on class forum
First Vote Receive first “vote” on class forum
Loser badges
Module Laggard Fail to complete a compulsory module in 10 days
Module Inferior Complete a module with less than 50% accuracy in an end-of-module quiz
Loser of the Week Bottom 20 who score the least in the class
Miser of the Week Never post on class forum in Weeks 1–5
learners with a strong performance-approach goal ori- hypothesized that positive personal comparison
entation to engage in SRL. Similarly, the coefficient feedback would increase the learning efficiency of
estimate of Positive Personal Comparison × Performance- learners with a strong performance-avoidance goal
Avoidance was positive and significant (β 0.4112, p < orientation. However, Hypothesis 3(b) was not sup-
0.01), supporting Hypothesis 2(a), which hypothe- ported, as Positive Personal Comparison × Mastery had a
sized that gamified positive personal comparison negative and significant impact (β −0.9004, p < 0.01).
feedback would encourage learners with a strong Hence, our hypotheses are supported only for gami-
performance-avoidance goal orientation to engage in fied positive social comparison feedback provided to
SRL. Contrary to expectations, the coefficient estimate learners with a strong performance-approach goal ori-
of Negative Personal Comparison × Mastery was not sig- entation and gamified positive personal comparison
nificant. Hypothesis 3(a), which hypothesized that feedback to learners with a strong performance-
negative personal comparison feedback would avoidance goal orientation. For mastery-oriented par-
encourage learners with a strong mastery goal orien- ticipants, positive personal comparison feedback had a
tation to engage in SRL was thus not supported. negative impact on both SRL engagement and learning
Moreover, to our surprise, we observed that Positive efficiency, suggesting that a mismatch could debilitate
Personal Comparison × Mastery was negatively significant SRL.
(β −0.4237, p < 0.01), indicating that the treatment of A visual description of the effects of gamified per-
positively framed personal comparison feedback actually formance feedback designs and goal orientations on
reduced SRL engagement for learners with a strong mas- SRL engagement and learning efficiency is shown in
tery goal orientation. Figure 4. Detailed empirical analyses are available in
Hypothesis 1(b), Hypothesis 2(b), and Hypothesis Online Appendix I. We also examined the normality
3(b) addressed learning efficiency, which we measured and homoscedasticity of Equation (1) and propor-
using the pace of module completion with the SRL tional hazard assumption and Cox-Snell model fit of
process. We considered a module to be completed if Equation (2) in Online Appendix J.
the participant completed all three SRL phases, read all We found that the impact of SRL on learning out-
teaching materials, watched demonstration videos, and comes17 was positive and significant for both knowl-
completed end-of-module quizzes. Column (2) of Table edge (β 1.6124, p < 0.001) and performance (β
4 shows that the coefficient estimates of these combina- 5.3394, p < 0.001) test scores, as shown in Table 5.18 To
tions, Positive Social Comparison × Performance-Approach test Hypothesis 4, we used Baron and Kenny’s (1986)
(β 0.6910, p < 0.05) and Positive Personal Comparison × steps and Preacher and Hayes’ (2008) bootstrapping
Performance-Avoid (β 0.4938, p < 0.05), were positive method. We found that the mediating results were
and significant. The results provide support for significant for both knowledge and performance test
Hypothesis 1(b), which hypothesized that positive scores (see Online Appendix K), providing strong
social comparison feedback would increase the learning support for Hypothesis 4 that SRL mediates the influ-
efficiency of participants with a strong performance- ence of the match between gamified feedback and
approach goal orientation, and Hypothesis 2(b), which goal orientation on learning outcomes.
Figure 3. Flowchart of Experimental Procedures

5. Discussion learner goal orientations in an online learning plat-

Online learning platforms have been gamified with form. We found that when there was a match, gamifi-
the expectation of enhancing SRL engagement and cation design was indeed effective at enhancing learn-
improving learning outcomes. However, study results ers’ SRL engagement. In other cases, however, it led
on gamification design implementations are often to the unintended consequence of a deterioration in
mixed; hence, research must be undertaken to under- SRL, defeating the very purpose of gamification. These
stand the impact of individual gamification design findings highlight the need to closely consider learners’
elements (Dicheva et al. 2015, Seaborn and Fels 2015, goal orientations when developing gamification designs
Koivisto and Hamari 2019). Moreover, principles of in online learning environments and the importance of
gamification suggest that a good match between incorporating the personalization gamification principle
design elements and individual traits is essential for in designing performance feedback.
fostering meaningful engagement and achieving task The unexpected finding that some types of gamified
outcomes (Liu et al. 2017). Therefore, to offer one performance feedback (e.g., positive personal compar-
explanation for the mixed results from gamification of ison) negatively impacted the SRL engagement of
online platforms, we tested the match between gami- learners with a mastery orientation could perhaps be
fied performance feedback and the individual traits of explained by the phenomenon of overjustification,
Table 3. Explanation of Variables Used in Research Models
Variable Explanation References
SRL Self-regulated learning engagement indicator that Santhanam et al. (2008), Schunk
reflects a learner’s effort spent on each phase of and Zimmerman (2012), and
self-regulated learning, namely, forethought, Zimmerman and Schunk (2008)
performance and volitional control, and
reflection.
SRL is measured by the principal component of
one forethought activity, the principal
component of seven performance and volitional
control activities, and one reflection activity in
Table 1. SRL is a continuous number; high value
indicates more engagement in self-regulated
learning activities, and low value indicates less
engagement.
Test Knowledge Test Score; integer in the range of [0, 25] Yi and Hwang (2003)
Performance Test Score; integer in the range of [0,
75]
Gamified Performance Feedback Four gamification treatments include Positive Positive and negative goal framing
Personal Comparison, Positive Social Comparison, (Van de Velde et al. 2010); social
Negative Personal Comparison, and Negative Social (Dicheva et al. 2015, Sailer and
Comparison, as described below. Homner 2019); and personal
comparisons (Cheong et al.
2014, Mekler et al. 2017)
Positive Personal Comparison Indicator variable: positive personal gamification Cheong et al. (2014), Mekler et al.
treatment (1); otherwise (0) (2017), and Van de Velde et al.
Examples include percentage of modules (2010)
completed and tips to get master badges (please
see Figure C1 of Online Appendix C)
Positive Social Comparison Indicator variable: positive social gamification Dicheva et al. (2015), Sailer and
treatment (1); otherwise (0) Homner (2019), and Van de
Examples include leader board and helpfulness Velde et al. (2010)
board (please see Figure C2 of Online Appendix C)
Negative Personal Comparison Indicator variable: negative personal gamification Cheong et al. (2014), Mekler et al.
treatment (1); otherwise (0) (2017), and Van de Velde et al.
Examples include percentage of modules not yet (2010)
completed and tips to avoid loser badges (please
see Figure C3 of Online Appendix C)
Negative Social Comparison Indicator variable: negative social gamification Dicheva et al. (2015), Sailer and
treatment (1); otherwise (0) Homner (2019), and Van de
Examples include loser board and miser board Velde et al. (2010)
(please see Figure C4 of Online Appendix C)
Goal Three goal orientations that include Mastery, Midgley et al. (2000)
Performance-Approach, and Performance-Avoidance,
as described below.
Mastery Mastery goal orientation; average of five Midgley et al. (2000)
measurement items M1–M5 in Online Appendix
F; continuous number in the range of [1, 5],
where one indicates low mastery goal
orientation and five indicates high mastery goal
orientation.
Performance-Approach Performance-approach goal orientation; average of Midgley et al. (2000)
three measurement items P1–P3 in Online
Appendix F; continuous number in the range of
[1, 5], where one indicates low performance-
approach goal orientation and five indicates high
performance-approach goal orientation.
Performance-Avoidance Performance-avoidance goal orientation; average of Midgley et al. (2000)
two measurement items A1 and A2 in Online
Appendix F; continuous number in the range of
[1, 5], where one indicates low performance-
Variable Explanation References
avoidance goal orientation and five indicates

high performance-avoidance goal orientation.
Control Control variablesa include Male, Age, Transfer, Pajares (2002), Zimmerman (1990),
IT_Skill, TechUnivMajor, Pre_Test, CGPA, Veronneau et al. (2014), and Yi
HighSchoolMajorj, and TestVersionk and Davis (2003)

a
Male is an indicator variable; Age is an ordinal variable in the range of one to four; Transfer is an indicator variable of whether a participant is a
transfer student or nonfreshman; IT_Skill is self-perceived skills in the range of zero to five; TechUnivMajor is an indicator variable of whether a
participant is major in IS or CS; Pre_Test is the pre-experimental test scores on social analytics and business intelligence in the range of one to
seven; CGPA is cumulative grade point average in the range of 0 to 4.3; HighSchoolMajorj ae dummy variables that include Arts, Humanities, and
Business ( j 1); Science and Mathematics (j 2); and Others (j 0). In the test performance analysis, we further controlled for test version (i.e.,
versions A–E). TestVersionk are dummy variables that include versions A (k 0), B (k 1), C (k 2), D (k 3), and E (k 4). Based on previous
studies, gender (Pajares 2002) and age (Zimmerman 1990) may influence SRL. Therefore, we controlled for gender (Male) and age. We also
accounted for experience in tertiary education (Transfer) in our research model. Because the context of the study was IT-related and prior
knowledge/experience on IT might confound the results, we controlled for self-evaluated IT skills (IT_Skill) and related IT knowledge/
experience (TechUnivMajor) obtained in other courses. To better control for prior knowledge on our experiment subject (i.e., social analytics and
business intelligence), we controlled for the pre-experiment knowledge test scores on the related subject (Pre_Test) (Yi and Davis 2003). We also
controlled for the general intellectual ability of participants, proxied by past cumulative grade point average (CGPA) (Veronneau et al. 2014).
Because most of the students were freshmen, we controlled for their high school backgrounds (HighSchoolMajorj).
which occurs because a person’s intrinsic interest in a the statements by learners with a strong mastery goal
particular activity may decrease when they are given orientation who received positively framed personal
explicit incentives, such as rewards, for completing comparison feedback support our expectations: “Give
that activity (Lepper and Greene 2015). Research sug- details of my learning process and results rather than
gests that extrinsic motivators may crowd out intrinsic those badges as I don't really know how I perform
motivation in a task (Bénabou and Tirole 2006, Hanus through those badges”; “Stop using badges. Motivate
and Fox 2015). Mastery goal-oriented learners are students in ways other than marks”; “Statistics emails
intrinsically motivated, learn out of curiosity, and pur- don't say much. They just encouraged others to post
sue courses that are personally meaningful and inter- senseless posts to get meaningless badges and points,
esting to them by engaging in deep learning strategies but no real learning.”
(Ames and Archer 1988, Elliot 1999, Senko and Miles On the other hand, the very same type of feedback,
2008). In fact, they follow their own learning agenda; when provided to learners with a strong performance-
place effort into learning material they find personally approach goal orientation, elicited the following
interesting, rather than that suggested by their teach- responses: “I like the colorful badges. This motivated
ers; evaluate their performance using self-referential me to get high marks on KEEP”; “It’s good if I can
standards; and may not receive the highest grades in know my ranking”; “Show the highest score among my
the course—a pattern dissimilar from that of classmates to inform me and thus motivate me”; “Give
performance-approach goal-oriented learners (Senko some information about the distance between top
and Miles 2008). Hence, for the mastery goal-oriented students”; “It would be more encouraging if I can com-
learners, even though the gamified performance feed- pare my scores and rankings with my peers.” These
back was personal and there was no comparison with qualitative statements support our premise that gami-
peer performance, reminders about external rewards fied performance feedback may be viewed differently
(e.g., badges and indicators of how far they were from by different learners, depending on their goal orienta-
obtaining the next badge) may have frustrated them tions and highlight the observed statistical differences
and caused them to lose engagement in the task. Prior in our findings on the impact of gamified performance
research has found that praise and rewards may have feedback on SRL engagement, learning outcomes and
no effect on, or even undermine, intrinsic motivation learning efficiency.
(Henderlong and Lepper 2002). We found that for
learners with mastery orientation, the gamified per- 5.1. Theoretical Implications
formance feedback did not enhance their SRL engage- Self-regulated learning theory is among the most fre-
ment. In fact, from a relatively significant positive SRL quently applied theories in learning and training in IS
engagement, the gamified feedback led to a significant and education because it describes the learning proc-
negative change in SRL engagement. In a postexperi- ess well and explains learning success in many differ-
ment survey, we invited learners to share their com- ent contexts and at various educational levels (e.g.,
ments on the gamified performance feedback. Some of Gravill and Compeau 2008, Santhanam et al. 2008,
Table 4. The Impact of Goal Orientations and Gamified Performance Feedback on SRL
Independent variables: Goal orientations, Dependent variables

gamified performance feedback, and
interaction terms SRL Engagement Pace to Complete a Module with SRL
Positive Personal Comparison 0.2668** 0.5425**

[0.1078, 0.4258] [0.1691, 0.9160]
(0.0810) (0.1905)
Positive Social Comparison 0.1865* 0.3625
[0.0067, 0.3663] [−0.0141, 0.7391]
(0.0916) (0.1922)
Negative Personal Comparison 0.2921** 0.6096**
[0.1124, 0.4718] [0.2397, 0.9796]
(0.0915) (0.1888)
Negative Social Comparison 0.1903* 0.2637
[0.0207, 0.3599] [−0.1145, 0.6420]
(0.0864) (0.1930)
Mastery 0.3475*** 0.4028*
[0.1528, 0.5423] [0.0841, 0.7215]
(0.0992) (0.1626)
Performance-Approach −0.1135 −0.0787
[−0.2359, 0.0089] [−0.3493, 0.1920]
(0.0623) (0.1381)
Performance-Avoidance −0.1877* −0.1487
[−0.3340, −0.0414] [−0.4587, 0.1613]
(0.0746) (0.1582)
Positive Personal Comparison 3 Mastery 20.3797** 20.4722*
[20.6454, 20.1140] [20.9236, −0.0207]
(0.1354) (0.2303)
Positive Social Comparison × Mastery −0.1924 −0.1641
[−0.4925, 0.1077] [−0.6947, 0.3664]
(0.1529) (0.2707)
Negative Personal Comparison × Mastery −0.1534 −0.0391
[−0.4504, 0.1437] [−0.5205, 0.4422]
(0.1513) (0.2456)
Negative Social Comparison × Mastery −0.0555 −0.2586
[−0.3198, 0.2087] [−0.7514, 0.2342]
(0.1346) (0.2514)
Positive Personal Comparison × Performance- −0.0671 −0.1131
Approach [−0.3296, 0.1953] [−0.5611, 0.3350]
(0.1337) (0.2286)
Positive Social Comparison 3 0.4264** 0.5395*
Performance-Approach [0.1826, 0.6702] [0.0568, 1.0221]
(0.1242) (0.2462)
Negative Personal Comparison × Performance- 0.2061 0.1528
Approach [−0.1024, 0.5147] [−0.2920, 0.5976]
(0.1572) (0.2270)
Negative Social Comparison × Performance- 0.1498 0.0655
Approach [−0.0708, 0.3704] [−0.3832, 0.5141]
(0.1124) (0.2289)
Positive Personal Comparison 3 0.5436*** 0.5247*
Performance-Avoidance [0.2945, 0.7926] [0.0191, 1.0303]
(0.1269) (0.2580)
Positive Social Comparison × Performance- 0.0450 0.0160
Avoidance [−0.2547, 0.3447] [−0.5248, 0.5568]
(0.1527) (0.2759)
Negative Personal Comparison × Performance- 0.1468 0.1784
Avoidance [−0.0988, 0.3925] [−0.2875, 0.6444]
(0.1252) (0.2377)
Negative Social Comparison × Performance- 0.0726 0.1169
Avoidance [−0.2499, 0.3950] [−0.3882, 0.6220]
(0.1643) (0.2577)
Constant/Theta Constant: -1.9463*** Theta: 0.1036
[−2.8908, −1.0017] (0.0485)
(0.4812)
Independent variables: Goal orientations, Dependent variables

gamified performance feedback, and
interaction terms SRL Engagement Pace to Complete a Module with SRL
With control variablesa

N 838 838
Model Fit Adjusted R2 0.16 Chi2 94.37

Max VIF 5.10 4.55
a
Some control variables are significant. In the model of “SRL engagement,” Male (coefficient −0.2086, p < 0.001), Pre-experiment Knowledge
Test (coefficient 0.0522, p < 0.01), and CGPA (coefficient 0.4425, p < 0.001) are significant. In the model of “time to complete a module,” Male
(coefficient −0.2033, p < 0.05) and CGPA (coefficient 0.7364, p < 0.001) are significant. Other variables are not significant.
*p < 0.05; **p < 0.01; ***p < 0.001.
Wan et al. 2012, Zimmerman and Schunk 2013, Wong explain the mixed results of gamification designs in
et al. 2019, and Jansen et al. 2020). Gamification inter- online learning contexts, demonstrating that the over-
ventions are being applied to enhance SRL with vary- all gamification design is not likely to lead to success-
ing degrees of success because higher SRL leads to ful learning outcomes unless it is aligned with learner
increased learning outcomes. However, to the best of traits. Hence, unlike a one-size-fits-all approach for
our knowledge, no study has yet examined learners’ quickly developing gamified systems, the use of gami-
detailed learning activities over a period of time to fication principles, enhanced understanding of con-
examine the impact of gamification interventions. By text, and careful design will lead to meaningful
closely tracking SRL engagement through a MOOC engagement and continued use of gamified IS (Liu
learning platform with digital tracing over a five- et al. 2017). Personalization in online learning will be
week time period, this study contributed to theoretical beneficial and add to the trend in designs of online
understandings of SRL by demonstrating how gamifi- platforms that call for more personalization and adap-
cation designs can either enhance or diminish SRL tation (Ho et al. 2011, Zhang et al. 2011, Ho and Bodoff
engagement among different types of learners. We 2014).
provide empirical evidence that SRL is an important Online learning continues to be of considerable
mediator of personalized gamification and learning interest to IS researchers because it is used extensively
performance outcomes. Unlike previous research on in corporate training and education (e.g., George 2014,
SRL in IS (e.g., Santhanam et al. 2008 and Wan et al. Panigrahi et al. 2018, and Huang et al. 2021). Because
2012), we used more granular methods—such as ana- digital platform designs are increasingly used for cor-
lyzing learners’ digital trace data captured by our sys- porate activities, research that takes a sociotechnical
tem and learning analytics—rather than applying a perspective on technology design is needed (Tiwana
cross-sectional survey design to measure SRL. Finally, et al. 2010, Sarker et al. 2019). From a sociotechnical per-
in this study, we captured the pace of SRL engage- spective, the interplay of social and technical elements
ment in learning and found that a match between closely conforms to the nomological net proposed by
game elements and learner goal orientations had a Benbasat and Zmud (2003), which describes how IT arti-
positive impact on learning efficiency. facts (e.g., gamification) impact the context (i.e., online
From a gamification perspective, recent reviews platforms) in which they are embedded. We identified
show that there are many scholarly papers and the boundary conditions (i.e., a match between goal ori-
reports on gamification in practice; however, few entation and gamification feature) under which online
studies have carefully examined the individual game learning platforms can promote participants’ SRL behav-
elements and their impact on the psychological proc- iors and learning performance.
esses of individual users (Koivisto and Hamari 2019,
Park et al. 2019). Moreover, few studies are theoreti- 5.2. Implications for Practice
cally anchored, and among those that are, social deter- Although the adoption of gamified IS has been grow-
mination theory is the most applied theory in almost ing in popularity, failed gamification projects are esti-
every context (Seaborn and Fels 2015). We applied mated to cost U.S. businesses U.S. $700 million over
SRL theory because it is most pertinent to online the last decade, casting concerns on gamification
learning tasks, and we closely examined the impact of designs and their use for heterogeneous work scenar-
gamified performance feedback on individual learn- ios (Maddox 2018). Rather than concluding that a
ers’ psychological processes over a period of time. We gamification design failed, it should be considered
found that gamification design enhanced SRL engage- that it may have resulted in unmet goals for certain
ment and significantly improved learning outcomes types of users; hence, a nuanced and thoughtful
for some learners, but not for others. This result helps design could lead to successful outcomes. Because
Figure 4. (Color online) Match Between Learner Goal Orientations and Gamified Performance Feedback
design has to match individual traits, one approach to who pursue learning that they find personally mean-
gamification is to use an adaptive design, wherein ingful and interesting, curiosity could be evoked
game elements can be tailored to individual users at through game elements that give a preview of coming
the initial stage based on their demographic and topics, rather than gamified performance feedback
learning-orientation data or based on individual based on goals. In many universities, information on
behaviors learned from their use of the system (Liu students, including their personalities and goal orien-
et al. 2017, Park et al. 2019). For example, curiosity is tations, is collected when they enroll (e.g., through the
an important motivational driver that can be used in Learning and Study Strategies Inventory test).
gamification designs. For mastery-oriented learners Machine learning and artificial intelligence techniques
Table 5. The Impact of SRL on Test Performance
Variable (1) Knowledge Test (2) Performance Test
SRL 1.5905*** 5.7157***

[1.1108, 2.0701] [4.4092, 7.0222]
(0.2444) (0.6656)
Constant 0.0371 −1.9279
[−3.4793, 3.5534] [−12.9692, 9.1134]

(1.7915) (5.6251)
With control variablesa and test versions
N 838 838
Adjusted R2 0.22 0.25
Max VIF 2.67 2.67
a
Some control variables are significant. In the model of “Knowledge Test,” Male (coefficient −1.4951, p < 0.01), IT Skill (coefficient 1.0278, p
< 0.001), Age (coefficient 0.8570, p < 0.01), Pre-experiment Knowledge Test (coefficient 0.5162, p < 0.01), Technical Major (coefficient 3.8021, p <
0.001), and CGPA (coefficient 2.6786, p < 0.001). In the model of “Performance Test,” Male (coefficient 2.6290, p < 0.05), IT Skill (coefficient
2.1092, p < 0.01), Age (coefficient −1.9136, p < 0.05), Pre-experiment Knowledge Test (coefficient 1.4135, p < 0.01), Technical Major (coefficient
8.2663, p < 0.01), CGPA (coefficient 10.0842, p < 0.001), and Arts, Business, and Humanities Background in Secondary School (coefficient −8.3611,
p < 0.001).
***p < 0.001.
could be used to infer preferences and traits of learn- individual trait—goal orientation—because of its
ers. This might allow designs to be adapted to individ- importance in SRL, future studies should address the
ual learner traits and preferences. Overall, a move impact of other individual traits. We tested our ideas
away from a general to a more nuanced and personal- in one instance of online learning—a MOOC, with
ized approach will help develop effective gamified studies indicating that observed behaviors general-
online learning. ize to other online environments (Almeda 2018).
While highlighting the importance of careful design, However, online learning can occur in different for-
our research also demonstrates that gamification mats, ranging with some to no instructor interven-
designs are useful and provide value by motivating tion. Our study results could be tested in other
some unenthusiastic learners to learn. Learners with a online environments, with other game elements and
strong approach-avoidance goal orientation have typi- other individual traits.
cally been ignored in SRL research because of their Despite the growing popularity of gamification, its
strong desire to avoid actions and face difficulties in failure rate is astonishingly high, and we find that one
applying SRL (Wan et al. 2012). Here, we showed that way to alleviate this is to consider the match between
these learners respond to and engage with positive personalized gamification elements and user traits.
personal comparison gamification feedback showing COVID-19 has increased the reliance on and adoption
significantly increased SRL engagement. of online education for many higher educational uni-
versities, and this study makes a timely contribution
6. Limitations, Future Research, and to make online learning more effective.
Concluding Remarks
As with any field experiment, our study has several Acknowledgments
limitations. First, we did not directly observe all the The authors appreciate the support of KEEP in helping us
conduct the field experiment on their platform and thank
learning activities of individual participants, but
the senior editor, associate editor, and anonymous
relied on system-usage data for interpretation of SRL. reviewers for their constructive feedback throughout the
Any activities outside the online platform, such as review process. The authors thank Yulin Fang, Choon
offline discussions with peers, were not captured. Sec- Ling Sia, and participants at the CityU IS Summer
ond, SUTVA may not be valid due to social interac- Research Workshop for their suggestions; and Ken Lee
tions among participants. However, as our treatments and Wilson Weixun Li for helping as research assistants.
contained personalized feedback, it is not likely that
participants shared such private information with
their friends. Moreover, some treatments were specific Endnotes
to individuals (e.g., module progress) and may not
1
The term “match” refers to the alignment between gamification
design elements and user traits. The noun “match” is used here in
have had any impact on others, even if they were accordance with Gamification Principle B by Liu et al. (2017).
shared. Third, we collected data from college stu- 2
Although earlier studies asked participants to reflect on their SRL
dents, but these findings may not be easily generaliz- activities, new learning analytics like those used in this study trace
able to all learners, some of whom are working adults. learners’ activities on a digital platform to evaluate SRL engage-
In addition, whereas this study focused on one ment and learning efficiency (Pardo et al. 2017, Kim et al. 2018).
3
It has also been termed mastery-approach goal orientation (Elliot could receive an up-vote from their peers (see Figure C6 of Online
1999), learning orientation (Zweig and Webster 2004), and learning Appendix C). They could also receive a down-vote for unhelpful
goal orientation (Wan et al. 2012). behaviors. We summarized these votes to construct the helpfulness
4
The MOOC platform Knowledge & Education Exchange Platform and miser boards, which showed the top and bottom 20 performers,
(KEEP) (https://keep.edu.hk/en/about) is an open platform where respectively. Figures C2 and C4 of Online Appendix C show exam-
universities can organize free online courses for the general public. ples of leader and helpfulness boards and loserboards and miser
We created a private course (only available to the experimental sub- boards, respectively.
14
jects) called Social Analytics and Business Intelligence. The details of control variables are shown in the last row of Table 3.
5 15
Participants were informed that by the end of the self-learning In Equation (2), t represents the time to adopt SRL, h(t) represents
period, the top 20 participants with the highest cumulative scores the hazard function, and represents a baseline hazard function
and the top 20 participants with the highest helpfulness scores where all covariates are zero. We incorporated cluster-specific ran-
would receive a Hong Kong (HK) $50 (U.S. $6.50) ice cream coupon. dom effects to account for within-cluster heterogeneity (Austin
The cumulative and helpfulness scores were determined by partici- 2017). Each class a participant attended was considered a specific
pants’ learning activities completed on KEEP and the total number cluster; 25 classes were considered.
of up-votes and down-votes they received on KEEP forums, 16
As a robustness check, we run regressions without control varia-
respectively. bles. The results are qualitatively similar. We also try including one
6
It is possible that a participant may possess more than one goal interaction variable at a time, the results remain qualitatively simi-
orientations. For details, please refer to Online Appendix F. lar. Please see Tables L1, L2, L3, and L4 of Online Appendix L for
7
We invited 440 students to participate in the experiment between details.
17
January and February 2016. However, some failed to complete the Apart from learning outcomes, we also examine the relationship
pre-experiment survey or post-experiment test, and some failed to between SRL and course examination scores and that between SRL
accurately answer two validity-check questions in the pre- and course GPA with the same set of control variables. Please refer
experiment survey and were eliminated. The final subject pool com- to Table L5 of Online Appendix L for details.
prised 419 participants. The two validity-check questions are Q8 18
As a robustness check, we also run regressions without control
and Q27 in Online Appendix D. variables. The results are qualitatively similar. Please see Table L2
8
To alleviate the concern that the experimental results were driven of Online Appendix L for details.
by MOOC features (e.g., self-scheduling of learning modules,
quizzes, and forums), daily reminder emails (e.g., personal accom-
References
plishments), or standard elements in gamified treatment emails
Alavi M, Leidner DE (2001) Technology-mediated learning—a call
(e.g., cumulative scores and badges) rather than our gamified treat-
for greater depth and breadth of research. Inform. Systems Res.
ments, we conducted a control experiment between September and
12(1):1–10.
October 2016. A total of 370 participants were recruited. A sample
Almeda M (2018) Comparing the factors that predict completion
of 341 participants was retained after filtering out participants who
and grades among for-credit and open/MOOC students in
did not participate in either the pre-experiment survey or post-
online learning. Online Learning 22(1):1–18.
experiment test and those who failed to correctly answer the two
Alraimi KM, Zo H, Ciganek AP (2015) Understanding the MOOCs
validity-check questions (see Online Appendix D for details).
continuance: The role of openness and reputation. Comput. Ed.
9
One point was given for each correctly answered quiz question. A 80:28–38.
participant could attempt each end-of-module quiz at most four Ames C (1992) Classrooms: Goals, structures, and student motiva-
times. If a participant attempted a quiz multiple times, we com- tion. J. Ed. Psych. 84(3):261–271.
puted the average score on each quiz question. Ames C, Archer J (1988) Achievement goals in the classroom: Stu-
10
We treated SRL as a formative construct that consists of three dents’ learning strategies and motivation processes. J. Ed. Psych.
phases: forethought, performance and volitional control, and reflec- 80(3):260–267.
tion. We measured this construct according to the principal compo- Antin J, Churchill EF (2011) Badges in social media: A social psy-
nent of the three phases (see Table 1 for details). chological perspective. ACM CHI 2011 Gamification Workshop
11
Participants took the test individually in a standalone cubicle. Proc. (Association for Computing Machinery, New York), 1–4.
They were informed that the test questions were different from Antonaci A, Klemke R, Specht M (2019) The effects of gamification
those on the end-of-module quizzes and that the topics were from in online learning environments: A systematic literature review.
modules 1–3 rather than 4 and 5. The purpose was to determine Informatics 6(3):32.
whether the treatments could motivate participants to perform Antonaci A, Klemke R, Stracke CM, Specht M (2017) Gamification in
additional work (modules 4 and 5), even if it was not covered on MOOCs to enhance users’ goal achievement. 2017 IEEE Global
the test. Top performers on the test were awarded a HK $50 (U.S. Engrg. Ed. Conf. EDUCON (IEEE, Piscataway, NJ), 1654–1662.
$6.50) ice cream coupon. All participants received a 1% bonus on Austin PC (2017) A tutorial on multilevel survival analysis: Methods,
their coursework for participating in the pre-experiment survey and models and applications. Internat. Statist. Rev. 85(2):185–203.
another 1% bonus for taking part in the post-experiment test. Bandura A (1991) Social cognitive theory of self-regulation. Organ.
12
Behav. Human Decision Processes 50(2):248–287.
All participants received a master badge if they accomplished a Bapna R, Umyarov A (2015) Do your online friends make you pay?
milestone. They also received a loser badge for failing within a A randomized field experiment on peer influence in online
given time period. The badge information was visible to partici- social networks. Management Sci. 61(8):1902–1920.
pants on KEEP. Each master (loser) badge carried an additional Baron RM, Kenny DA (1986) The moderator–mediator variable dis-
score of +1 (−1) when tallying a participant’s cumulative scores. tinction in social psychological research: Conceptual, strategic,
Figures C1 and C3 of Online Appendix C show examples of treat- and statistical considerations. J. Personality Soc. Psych. 51(6):
ments with positive and negative personal comparison game ele- 1173–1182.
ments, respectively. Becker J-M, Rai A, Ringle CM, Volckner F (2013) Discovering unob-
13
The helpfulness board encouraged participants to provide peer served heterogeneity in structural equation models to avert val-
assistance. Participants who helped others by answering questions idity threats. MIS Q. 37(3):665–694.
Bénabou R, Tirole J (2006) Incentives and prosocial behavior. Amer. Gravill J, Compeau D (2008) Self-regulated learning strategies and
Econom. Rev. 96(5):1652–1678. software training. Inform. Management 45(5):288–296.
Benbasat I, Zmud RW (2003) The identity crisis within the IS disci- Gupta S, Bostrom RP (2009) Technology-mediated learning: A compre-
pline: Defining and communicating the discipline’s core prop- hensive theoretical model. J. Assoc. Inform. Systems 10(9):686–714.
erties. MIS Quart. 27(2):183–194. Hakulinen L, Auvinen T (2014) The effect of gamification on students
Bol L, Garner JK (2011) Challenges in supporting self-regulation in with different achievement goal orientations. 2014 Internat. Conf.
distance education environments. J. Comput. Higher Ed. 23(2): Teaching Learning Comput. Engrg. (IEEE, Piscataway, NJ), 9–16.
104–123. Hamari J, Koivisto J, Sarsa H (2014) Does gamification work? A litera-
Butler DL (2002) Individualizing instruction in self-regulated learn- ture review of empirical studies on gamification. 2014 47th Hawaii
ing. Theory Practice 41(2):81–92. Internat. Conf. System Sci. (IEEE, Piscataway, NJ), 3025–3034.
Butler DL, Winne PH (1995) Feedback and self-regulated learning: Hanus MD, Fox J (2015) Assessing the effects of gamification in the
A theoretical synthesis. Rev. Ed. Res. 65(3):245–281. classroom: A longitudinal study on intrinsic motivation, social
Chang C-C, Tseng K-H, Liang C, Liao Y-M (2013) Constructing and comparison, satisfaction, effort, and academic performance.
evaluating online goal-setting mechanisms in web-based portfo- Comput. Ed. 80:152–161.
lio assessment system for facilitating self-regulated learning. Henderikx MA, Kreijns K, Kalz M (2017) Refining success and drop-
Comput. Ed. 69:237–249. out in massive open online courses based on the
Cheng S-L, Xie K (2021) Why college students procrastinate in intention–behavior gap. Distance Ed. 38(3):353–368.
online courses: A self-regulated learning perspective. Internet Henderlong J, Lepper MR (2002) The effects of praise on children’s
Higher Ed. 50:100807. intrinsic motivation: A review and synthesis. Psych. Bull. 128(5):
Cheong C, Filippou J, Cheong F (2014) Toward the gamification of 774–795.
learning: Investigating student perceptions of game elements. J. Ho SY, Bodoff D (2014) The effects of web personalization on user
Inform. Systems Ed. 25(3):233–244. attitude and behavior: An integration of the elaboration likeli-
Christy KR, Fox J (2014) Leaderboards in a virtual classroom: A test hood model and consumer search theory. MIS Quart. 38(2):497–
of stereotype threat and social comparison explanations for 520.
women’s math performance. Comput. Educ. 78:66–77. Ho SY, Bodoff D, Tam KY (2011) Timing of adaptive web personali-
Cleary TJ, Zimmerman BJ (2012) A cyclical self-regulatory account zation and its effects on online consumer behavior. Inform. Sys-
of student engagement: Theoretical foundations and applica- tems Res. 22(3):660–679.
tions. Christenson SL, Reschly AL, Wylie C, eds. Handbook of Hsieh P-AJ, Cho V (2011) Comparing e-learning tools’ success: The
Research on Student Engagement (Springer US, Boston), 237–257. case of instructor–student interactive vs. self-paced tools. Com-
Cook DA, Levinson AJ, Garside S (2010) Time and learning effi- put. Ed. 57(3):2025–2038.
ciency in Internet-based learning: A systematic review and Hu H, Gramling J (2009) Learning strategies for success in a web-
meta-analysis. Adv. Health Sci. Ed. 15(5):755–770. based course: A descriptive exploration. Quart. Rev. Distance Ed.
Darnon C, Harackiewicz JM, Butera F, Mugny G, Quiamzade A 10(2):123–134.
(2007) Performance-approach and performance-avoidance Huang N, Zhang J, Burtch G, Li X, Chen P (2021) Combating pro-
goals: When uncertainty makes a difference. Perspect. Soc. crastination on massive online open courses via optimal calls to
Psych. Bull. 33(6):813–827. action. Inform. Systems Res. 32(2):301–317.
Dehejia RH, Wahba S (2002) Propensity score-matching methods for non- Jansen RS, van Leeuwen A, Janssen J, Conijn R, Kester L (2020) Sup-
experimental causal studies. Rev. Econom. Statist. 84(1):151–161. porting learners’ self-regulated learning in massive open online
Dicheva D, Dichev C, Agre G, Angelova G (2015) Gamification in edu- courses. Comput. Ed. 146:1–17.
cation: A systematic mapping study. J. Ed. Tech. Soc. 18(3):75–88. Järvelä S, Järvenoja H, Malmberg J, Isohätälä J, Sobocinski M (2016)
Ding L, Kim C, Orey M (2017) Studies of student engagement in How do types of interaction and phases of self-regulated learn-
gamified online discussions. Comput. Ed. 115:126–142. ing set a stage for collaborative engagement? Learning Instruc-
Domı́nguez A, Saenz-de-Navarrete J, de-Marcos L, Fernández-Sanz tion 43:39–51.
L, Pagés C, Martı́nez-Herráiz J-J (2013) Gamifying learning Kennedy J (2014) Characteristics of massive open online courses
experiences: Practical implications and outcomes. Comput. Ed. (MOOCs): A research review, 2009-2012. J. Interactive Online
63:380–392. Learning 13(1):1–16.
Dweck CS, Leggett EL (1988) A social-cognitive approach to motiva- Khan AA, Madden J (2016) Speed learning: Maximizing student
tion and personality. Psych. Rev. 95(2):256–273. learning and engagement in a limited amount of time. Internat.
Elliot AJ (1999) Approach and avoidance motivation and achieve- J. Modern Ed. Comput. Sci. 8(7):22–30.
ment goals. Ed. Psych. 34(3):169–189. Kim D, Yoon M, Jo I-H, Branch RM (2018) Learning analytics to
Elliot AJ, Church MA (1997) A hierarchical model of approach and support self-regulated learning in asynchronous online courses:
avoidance achievement motivation. J. Personality Soc. Psych. A case study at a women’s university in South Korea. Comput.
72(1):218–232. Ed. 127:233–251.
Elliot AJ, Harackiewicz JM (1996) Approach and avoidance achieve- Kizilcec RF, Pérez-Sanagustı́n M, Maldonado JJ (2017) Self-regulated
ment goals and intrinsic motivation: A mediational analysis. J. learning strategies predict learner behavior and goal attainment
Personality Soc. Psych. 70(3):461–475. in massive open online courses. Comput. Ed. 104:18–33.
Elliot AJ, McGregor HA, Thrash TM (2002) The need for compe- Koivisto J, Hamari J (2019) The rise of motivational information sys-
tence. Deci EL, Ryan RM, eds. Handbook of Self-Determination tems: A review of gamification research. Internat. J. Inform. Man-
Research (University of Rochester Press, Rochester, NY), 361–387. agement 45:191–210.
Eom S (2015) The effects of student motivation and self-regulated Kolfschoten G, Lukosch S, Verbraeck A, Valentin E, de Vreede
learning strategies on student’s perceived e-learning outcomes G-J (2010) Cognitive learning efficiency through the use of
and satisfaction. Proc. Assoc. Inform. Systems Special Interest design patterns in teaching. Comput. Ed. 54(3):652–660.
Group Ed. AIS SIGED IAIM Conf., (Association for Information Krishnamurthy P, Carter P, Blair E (2001) Attribute framing and
Systems, Atlanta), 1–15. goal framing effects in health decisions. Organ. Behav. Human
George JF (2014) The future of higher education in the U.S.: Decision Processes 85(2):382–399.
MOOCs, a case in point. Accessed December 1, 2021, https:// Kuo Y-C, Walker AE, Schroder KEE, Belland BR (2014) Interaction,
aisel.aisnet.org/success_strategies/8/. Internet self-efficacy, and self-regulated learning as predictors
of student satisfaction in online education courses. Internet and Electronic Commerce: Empirical Research (Routledge Taylor &
Higher Ed. 20:35–50. Francis Group, London and New York), 63–86.
Kyewski E, Krämer NC (2018) To gamify or not to gamify? An Mok HN (2014) Teaching tip: The flipped classroom. J. Inform. Sys-
experimental field study of the influence of badges on motiva- tems Ed. 25(1):7–11.
tion, activity, and performance in an online learning course. Mory EH (2004) Feedback research revisited. Jonassen DH, ed.
Comput. Ed. 118:25–37. Handbook of Research on Educational Communications and Technol-
Landers RN, Bauer KN, Callan RC (2017) Gamification of task per- ogy (Lawrence Erlbaum Associates Publishers, Mahwah, NJ),
formance with leaderboards: A goal setting experiment. Com- 745–783.
put. Human Behav. 71:508–515. Narciss S, Proske A, Koerndle H (2007) Promoting self-regulated
Landers RN, Bauer KN, Callan RC, Armstrong MB (2015) Psycho- learning in web-based learning environments. Comput. Human
logical theory and the gamification of learning. Reiners T, Behav. 23(3):1126–1144.
Wood CL, eds. Gamification in Education and Business (Springer Pajares F (2002) Gender and perceived self-efficacy in self-regulated
International Publishing, Cham, Switzerland), 165–186. learning. Theory Practice 41(2):116–125.
Lee FK, Sheldon KM, Turban DB (2003) Personality and the goal- Panigrahi R, Srivastava PR, Sharma D (2018) Online learning: Adop-
striving process: The influence of achievement goal patterns, tion, continuance, and learning outcome—a review of literature.
goal level, and mental focus on performance and enjoyment. J. Internat. J. Inform. Management 43:1–14.
Appl. Psych. 88(2):256–265. Pardo A, Han F, Ellis RA (2017) Combining university student self-
Lepper MR, Greene D (2015) Overjustification research and beyond: regulated learning indicators and engagement with online
Toward a means-ends analysis of intrinsic and extrinsic motiva- learning events to predict academic performance. IEEE Trans.
tion. Lepper MR, Greene D, eds. The Hidden Costs of Reward: Learning Tech. 10(1):82–92.
New Perspectives on the Psychology of Human Motivation (Psychol- Park J, Liu D, Yi MY, Santhanam R (2019) GAMESIT: A gamified
ogy Press, London), 109–148. system for information technology training. Comput. Ed. 142:
Levin IP, Schneider SL, Gaeth GJ (1998) All frames are not created 103643.
equal: A typology and critical analysis of framing effects. Piccoli G, Rodriguez J, Palese B, Bartosiak ML (2020) Feedback at
Organ. Behav. Human Decision Processes 76(2):149–188. scale: Designing for accurate and timely practical digital skills
Lim M, Yang Y (2015) Effects of users’ envy and shame on social evaluation. Eur. J. Inform. Systems 29(2):114–133.
comparison that occurs on social network services. Comput. Pintrich PR (2000) The role of goal orientation in self-regulated
Human Behav. 51:300–311. learning. Boekaerts M, Pintrich PR, Zeidner M, eds. Handbook of
Lindenberg S, Steg L (2013) Goal-framing theory and norm-guided Self-Regulation (Academic Press, San Diego), 451–502.
environmental behavior. Van Trijp HCM, ed. Encouraging Sus- Preacher KJ, Hayes AF (2008) Asymptotic and resampling strategies
tainable Behavior (Psychology Press, New York), 37–54. for assessing and comparing indirect effects in multiple media-
Littenberg-Tobias J, Reich J (2020) Evaluating access, quality, and tor models. Behav. Res. Methods 40(3):879–891.
equity in online learning: A case study of a MOOC-based blended Raj D, Khamis SH (1958) Some remarks on sampling with replace-
professional degree program. Internet Higher Ed. 47(100759):1–11. ment. Ann. Math. Statist. 29(2):550–557.
Liu D, Santhanam R, Webster J (2017) Toward meaningful engage- Reich J, Ruipérez-Valiente JA (2019) The MOOC pivot. Science
ment: A framework for design and research of gamified infor- 363(6423):130–131.
mation systems. MIS Quart. 41(4):1011–1034. Roney CJR, Lehman DR (2008) Self-regulation in goal striving:
MacCallum RC, Zhang S, Preacher KJ, Rucker DD (2002) On the Individual differences and situational moderators of the
practice of dichotomization of quantitative variables. Psych. goal-framing/performance link. J. Appl. Soc. Psych. 38(11):
Methods 7(1):19–40. 2691–2709.
Maddox WT (2018) Why enterprise gamification is broken (the psycho- Ryan AM, Pintrich PR, Midgley C (2001) Avoiding seeking help in
logical science of motivation and effort). Amalgam Insights (March 1), the classroom: Who and why? Ed. Psych. Rev. 13(2):93–114.
https://amalgaminsights.com/2018/03/01/why-enterprise- Sagiv L, Schwartz SH (2000) Value priorities and subjective well-
gamification-is-broken-the-psychological-science-of-motivation- being: Direct relations and congruity effects. Eur. J. Soc. Psych.
and-effort/#more-895. 30(2):177–198.
Maehr ML, Zusho A (2009) Achievement goal theory. Wentzel KR, Sailer M, Homner L (2019) The gamification of learning: A meta-
Miele DB, eds. Handbook of Motivation at School (Routledge, analysis. Ed. Psych. Rev. 32:77–112.
New York), 77–104. Santhanam R, Liu D, Shen W-CM (2016) Gamification of
Majuri J, Koivisto J, Hamari J (2018) Gamification of education and technology-mediated training: Not all competitions are the
learning: A review of empirical literature. Proc. 2nd Internat. same. Inform. Systems Res. 27(2):453–465.
GamiFIN Conf. GamiFIN 2018 (CEUR-WS.org), 11–19. Santhanam R, Sasidharan S, Webster J (2008) Using self-regulatory
Maxwell SE, Delaney HD (1993) Bivariate median splits and spuri- learning to enhance e-learning-based information technology
ous statistical significance. Psych. Bull. 113(1):181–190. training. Inform. Systems Res. 19(1):26–47.
Mekler ED, Brühlmann F, Tuch AN, Opwis K (2017) Toward under- Sarker S, Chatterjee S, Xiao X, Elbanna A (2019) The sociotechnical
standing the effects of individual gamification elements on intrinsic axis of cohesion for the is discipline: Its historical legacy and its
motivation and performance. Comput. Human Behav. 71:525–534. continued relevance. MIS Quart. 43(3):695–720.
Midgley C, Kaplan A, Middleton M (2001) Performance-approach Schmidt-Kraepelin M, Thiebes S, Tran MC, Sunyaev A (2018)
goals: Good for what, for whom, under what circumstances, What’s in the game? Developing a taxonomy of gamification
and at what cost? J. Ed. Psych. 93(1):77–86. concepts for health apps. Proc. 51st Hawaii Internat. Conf. System
Midgley C, Maehr ML, Hruda LZ, Anderman E, Anderman L, Free- Sci. (IEEE, Piscataway, NJ), 1217–1226.
man KE, Urdan T (2000) Manual for the patterns of adaptive Schöbel S, Söllner M, Mishra AN (2017) Does the winner take it all?
learning scales. Working paper, University of Michigan, Ann Toward an understanding of why there might be no one-size-
Arbor. fits-all gamification design. Eur. Conf. Inform. Systems (Associa-
Mithas S, Almirall D, Krishnan MS (2014) A potential outcomes tion for Information Systems, Atlanta), 1–14.
approach to assess causality in information systems research. Schunk DH, Zimmerman BJ (2012) Motivation and Self-Regulated Learn-
Kauffman RJ, Tallon PP, eds. Economics, Information Systems, ing: Theory, Research, and Applications (Routledge, New York).
Seaborn K, Fels DI (2015) Gamification in theory and action: A sur- Wong J, Baars M, Davis D, Van Der Zee T, Houben G-J, Paas F (2019)
vey. Internat. J. Human Comput. Stud. 74:14–31. Supporting self-regulated learning in online learning environ-
Senko C, Miles KM (2008) Pursuing their own learning agenda: ments and MOOCs: A systematic review. Internat. J. Human Com-
How mastery-oriented students jeopardize their class perform- put. Interaction 35(4-5):356–373.
ance. Contemporary Ed. Psych. 33(4):561–583. Yi MY, Davis FD (2003) Developing and validating an observational
Shah D (2019) Year of MOOC-based degrees: A review of MOOC learning model of computer software training and skill acquisi-
stats and trends in 2018. Rep. Class Central (January 6), https:// tion. Inform. Systems Res. 14(2):146–169.
www.classcentral.com/report/moocs-stats-and-trends-2018/. Yi MY, Hwang Y (2003) Predicting the use of web-based informa-
Tiwana A, Konsynski B, Bush AA (2010) Platform evolution: Coevo- tion systems: Self-efficacy, enjoyment, learning goal orientation,
lution of platform architecture, governance, and environmental and the technology acceptance model. Internat. J. Human Com-
dynamics. Inform. Systems Res. 21(4):675–687. put. Stud. 59(4):431–449.
Van de Velde L, Verbeke W, Popp M, Van Huylenbroeck G (2010) Zhang T, Agarwal R, Lucas HC (2011) The value of IT-enabled
The importance of message framing for providing information retailer learning: Personalized product recommendations and
about sustainability and environmental aspects of energy. customer store loyalty in electronic markets. MIS Quart. 35(4):
Energy Policy 38(10):5541–5549. 859–881.
van Roy R, Zaman B (2018) Need-supporting gamification in educa- Zhou X, Tang J, Zhao Y, Wang T (2020) Effects of feedback design
tion: An assessment of motivational effects over time. Comput. and dispositional goal orientations on volunteer performance in
Ed. 127:283–297. citizen science projects. Comput. Human Behav. 107:106266.
VandeWalle D, Cummings LL (1997) A test of the influence of goal Zimmerman BJ (1990) Self-regulated learning and academic achieve-
orientation on the feedback-seeking process. J. Appl. Psych. ment: An overview. Ed. Psych. 25(1):3–17.
82(3):390–400. Zimmerman BJ (2000) Attaining self-regulation: A social cognitive
Veronneau M-H, Hiatt Racer K, Fosco GM, Dishion TJ (2014) The perspective. Boekaerts M, Pintrich PR, Zeidner M, eds. Hand-
contribution of adolescent effortful control to early adult educa- book of Self-Regulation (Academic Press, San Diego), 13–39.
tional attainment. J. Ed. Psych. 106(3):730–743. Zimmerman BJ, Schunk DH (2008) An Essential Dimension of Self-
Wan Z, Compeau D, Haggerty N (2012) The effects of self- Regulated Learning (Lawrence Erlbaum Associates, New York).
regulated learning processes on e-learning outcomes in Zimmerman BJ, Schunk DH (2013) Reflections on theories of self-
organizational settings. J. Management Inform. Systems 29(1): regulated learning and academic achievement. Zimmerman BJ,
307–340. Schunk DH, eds. Self-Regulated Learning and Academic Achieve-
Winne PH (2019) Paradigmatic dimensions of instrumentation and ment: Theoretical Perspectives (Routledge, London), 282–301.
analytic methods in research on self-regulated learning. Comput. Zweig D, Webster J (2004) What are we measuring? An examination
Human Behav. 96:285–289. of the relationships between the big-five personality traits, goal
Wolters CA (2003) Understanding procrastination from a self- orientation, and performance intentions. Personality Individual
regulated learning perspective. J. Ed. Psych. 95(1):179–187. Differences 36(7):1693–1708.

Isre 2022 1123

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Isre 2022 1123

Uploaded by

Copyright:

Available Formats

This article was downloaded by: [103.179.8.

80] On: 04 September 2022, At: 11:52

Information Systems Research

Could Gamification Designs Enhance Online Learning

To cite this article:

Published online in Articles in Advance 31 Mar 2022

Full terms and conditions of use: https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and-

Copyright © 2022 The Author(s)

Please scroll down for article—it is on subsequent pages

Could Gamification Designs Enhance Online Learning Through

History: Raghu Santanam, Senior Editor; J. J. Hsieh, Associate Editor.

1. Introduction et al. 2016). Massively open online courses (MOOCs)

Figure 1. Experimental Treatments—Gamiﬁed Performance Feedback

about their performance (Darnon et al. 2007). To alle-

Figure 2. (Color online) Conceptual Model

Table 1. Measurements of Self-Regulated Learning Engagement 10

Positive Negative Negative

3.9. Empirical Model

ment measure as a dependent variable. As shown in

sample analysis here because the latter involves

tion), which may lead to problems like data truncation

explanatory variable Gamiﬁed Feedback × Goal. Table 3

describes the notations used in Equation (1).

Hypothesis 1(b), Hypothesis 2(b), and Hypothesis 3(b)

model examining how speciﬁc factors inﬂuence the

rate at which a particular event occurs over time. In

Equation (2). A positive and signiﬁcant coefﬁcient

knowledge and conceptual understanding)

estimate implies a hazard ratio greater than one—in

other words, a faster pace of module completion with

SRL and higher learning efﬁciency.

formance feedback combinations. These results are

found that the coefﬁcient estimate of Positive Social

other interaction terms and Performance-Approach were

not at all signiﬁcant. This result provides support for

positive social comparison feedback would encourage

Table 2. Master and Loser Badges Available

Badge Titles Explanation

Figure 3. Flowchart of Experimental Procedures

5. Discussion learner goal orientations in an online learning plat-

Table 3. Explanation of Variables Used in Research Models

Variable Explanation References

Variable Explanation References

avoidance goal orientation and ﬁve indicates

HighSchoolMajorj, and TestVersionk and Davis (2003)

Independent variables: Goal orientations, Dependent variables

Positive Personal Comparison 0.2668** 0.5425**

Independent variables: Goal orientations, Dependent variables

With control variablesa

Model Fit Adjusted R2 0.16 Chi2 94.37

Table 5. The Impact of SRL on Test Performance

Variable (1) Knowledge Test (2) Performance Test

SRL 1.5905*** 5.7157***

[−3.4793, 3.5534] [−12.9692, 9.1134]

You might also like

Positive Personal Comparison 0.2668 0.5425

SRL 1.5905* 5.7157*