sustainability
Article
Flipped Classroom and Gamification Approach: Its Impact on
Performance and Academic Commitment on Sustainable
Learning in Education
Lui-Kwan Ng * and Chung-Kwan Lo
Citation: Ng, L.-K.; Lo, C.-K. Flipped
Classroom and Gamification
Approach: Its Impact on Performance
and Academic Commitment on
Sustainable Learning in Education.
Sustainability 2022, 14, 5428. https://
doi.org/10.3390/su14095428
Academic Editors: Michael D.
Kickmeier-Rust, Danial Hooshyar
and Nour El Mawas
Received: 31 March 2022
Accepted: 28 April 2022
Published: 30 April 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Sustainability 2022, 14, 5428. https://doi.org/10.3390/su14095428
Department of Mathematics and Information Technology, The Education University of Hong Kong,
Hong Kong 999077, China; chungkwanlo@eduhk.hk
* Correspondence: s1122637@s.eduhk.hk
Abstract: The onset of the COVID-19 global pandemic has negatively impacted sustainable learning
in education (SLE). During city lockdowns, higher education institutes (HEIs) have transitioned from
adopting solely traditional didactic classroom teaching to including innovative, flexible learning
approaches such as flipped classrooms. Gamification is a new techno-pedagogy that has been in-
tegrated into flipped classrooms to promote learner achievement and engagement. Grounded in
self-determination theory, the objectives of this exploratory study were to analyse the influence of
the flipped classroom and gamification on SLE concerning learner achievement and engagement.
Participants were recruited from postgraduate business education programmes in China, and three
instructional interventions were applied for a semester of 10 weeks. The three instructional interven-
tions applied were: gamified flipped classroom (n = 25), non-gamified flipped classroom (n = 24) and
gamified traditional classroom (n = 19). A mixed-methods approach was used, and both quantitative
and qualitative data were analysed. The results indicated gamified traditional classrooms promote
learner achievement, and the gamified flipped classrooms promote learner engagement. Furthermore,
learning culture, such as teacher-dependency, also influence learner achievement and engagement.
The class observation reports and learner interviews suggested that both gamified flipped classrooms
and gamified traditional classrooms support SLE in the time of academic uncertainty during the
COVID-19 pandemic.
Keywords: COVID-19; flipped classroom; gamification; self-determination theory; business education;
adult learning
1. Introduction
The onset of the COVID-19 global pandemic has imposed a negative impact on the
Sustainable Development Goals (SDGs) of Education 2030 (SDG 4) declared by the United
Nations in 2015 [1,2]. The goal of SDG 4 is to equip adult learners with the knowledge and
skills necessary for the benefit of global sustainable development by providing flexible edu-
cation pathways in higher education [3]. Higher education institutes (HEIs) have to equip
adult learners with the knowledge and skills of sustainable learning in education (SLE) to
cope with challenging and complicated circumstances [4,5]. Thus, HEIs are responsible for
assisting adult learners in overcoming barriers and interruptions by providing sustainable
quality education, which is essential for the country’s SDGs [6,7]. HEIs were compelled
to rapidly redesign teaching approaches and classroom arrangements to offer flexible and
sustainable learning pathways for SLE in the times of COVID-19 lockdown [8,9].
Instructors in China innovatively planned their teaching by providing learners with
pre-class self-study materials during in-campus classroom lessons due to the uncertainty
of intermittent lockdowns [10]. Advancements in information communication technol-
ogy have made such learning arrangements that combine pre-class and in-classroom
https://www.mdpi.com/journal/sustainability
Sustainability 2022, 14, 5428 2 of 23

learning activities feasible. These arrangements are critical for the sustainability of educa-
tion programmes under enforced home confinements and campus lockdowns during the
pandemic [11,12]. Most importantly, these new pedagogical approaches can alleviate the
threats of the pandemic on the sustainability of the country’s education system [13] and
promote SLE amid COVID-19 [4].
Learners with flipped classroom approaches were provided with instructional ma-
terials for self-study at home before participating in practical learning activities in the
classroom, reducing the reliance on face-to-face teaching instead of a traditional classroom
arrangement that is totally reliant on it [14]. Learner achievement and engagement are two
important indicators of the success of flipped classrooms in HEIs [15,16]. Research also
indicates that the success of flipped classrooms depends on sustained learner engagement
in learning activities [17,18]. According to self-determination theory (SDT), gamification
has the potential to promote learner achievement and motivate learner engagement in the
learning activities of flipped classrooms [19]. SDT is an empirically derived theory in social
contexts that differentiates human motivation in terms of autonomous and controlled, in
which autonomously motivated learners thrive in educational settings [20,21]. Gamifica-
tion is the use of game-design elements in non-gaming contexts and has the potential to
motivate learning [22,23]. Based on SDT, gamification is hypothesised to foster intrinsic
motivation and satisfy the psychological needs for autonomy, competence, and social
relatedness in flipped classrooms [19].
Flipped classroom approaches have also been adopted in HEI programmes during
the COVID-19 pandemic for sustainable learning in higher education [24]. Instructors
provide prerecorded instructional videos to enable learners to study on their own dur-
ing home confinement periods. In addition to the flexibility, the flipped classroom ap-
proach also allows more in-class time for promoting SLE. The key components of SLE are
(1) active learning; (2) independent learning; (3) collaborative learning; (4) renewing and
relearning; (5) knowledge and skills transferability [4]. The key learning activities in flipped
classrooms are pre-class self-study (i.e., independent learning), knowledge application
(i.e., transferability), problem-solving (i.e., renewing and relearning), and peer-assisted
learning (i.e., collaborative and active learning) [8,19,25,26]; these are especially advanta-
geous in promoting SLE. HEIs in China flexibly adopted flipped classroom approaches in
response to the government’s policy of “suspension of classes without interrupting learn-
ing” during the COVID-19 pandemic [27,28]. However, additional research is required to
investigate the impact of the abrupt pedagogical change caused by flipped classrooms in the
long history of Chinese education, which is dominated by traditional didactic teaching [29].
This study investigated the impact of the flipped classroom and gamification ap-
proaches on learner achievement and engagement in postgraduate business programmes
for SLE in China during the COVID-19 pandemic. Three classes with different instructional
interventions were applied: gamified flipped classroom (GFC), non-gamified flipped class-
room (NFC) and gamified traditional classroom (GTC). Furthermore, the study was guided
by the following two research questions:
• RQ1. How does the adoption of the flipped classroom approach influence (a) learner
achievement and (b) learner engagement compared to non-flipped approaches?
• RQ2. How does the adoption of gamification influence (a) learner achievement and
(b) learner engagement compared to non-gamified approaches?

2. Literature Review
We have the following three sections of the literature review. First, we discuss the SLE
and flipped classroom approach during the COVID-19 pandemic. Second, we review how
the SDT framework supports and motivates SLE and flipped classrooms. Third, we explore
the integration of gamification with flipped and traditional classrooms for SLE.
Sustainability 2022, 14, 5428 3 of 23

2.1. SLE and Flipped Classroom during COVID-19 Pandemic

SLE is a philosophy of learning and teaching, which is not limited to education for or
about sustainability [30]. SLE supports SDGs in education for adult learners, whose past
knowledge and skills are viewed as flexible and receptive to modification [31]. SLE is an
emerging and timely concept designed to enable learners to keep pace with technological
and social changes [8]. HEIs with SLE initiatives provide learners with individual and
group learning [32]. SLE promotes learners’ willingness to participate and contribute to the
learning process, reflected in learners’ behavioural, emotional, and cognitive engagement
in learning activities [33,34]. Ben-Eliyahu [4] summarised the key components of SLE as:
1. Active learning: refers to seeking information actively and intentionally;
2. Independent learning: refers to self-sustained learning, which involves learning
by oneself, being autodidactic, identifying and finding sources for what needs to
be learned;
3. Collaborative learning is conducted in groups with peers of different levels of knowl-
edge and is a process of acquiring and creating knowledge;
4. Renewing and relearning is the self-reflection and awareness of our knowledge that
might have been forgotten or need to be relearned because of advancements in a field;
5. Knowledge and skills transferability: refers to using knowledge and skills learned in
one setting (e.g., in the classroom) to another setting (e.g., at work).
SLE could be integrated into education settings with information communication
technology (ICT), online courses (e.g., MOOC), and pre-class and in-class learning [8]. Such
flexible pedagogical approaches are especially beneficial to ensure learning sustainability
when classroom teaching is suspended during uncertain and complex situations [35].
The COVID-19 pandemic impedes learning sustainability by imposing academic
uncertainty on HEIs. Learners reported negative emotions and decreased satisfaction due
to interruptions to their sustainable learning [36,37]. HEIs in China needed to rapidly
adjust their instructional practices to anticipate the changes in the country’s confinement
policies [38]. In sync with this development, educators in China quickly adopted the flipped
classroom approach with the aid of information communication technologies [28,29]. The
flipped classroom approach is a technology-enhanced pedagogy that free ups class time by
providing pre-class instructional videos [39]. Flipped classrooms emphasise learner-centric
and problem-solving activities inside the classroom [40], which enable independent and
collaborative learning [41]. Peer-to-peer and peer-to-teacher interactions promote active
learning, and problem-solving activities encourage knowledge renewing and relearning
of SLE [4,29]. In addition, problem-solving activities for knowledge transferability also
enhance perceived learning and learner achievement [42,43].
Chinese learners are very concerned about the delays in their academic progress [44].
Therefore, they favoured flipped classroom approaches as they could benefit from the
self-paced independent learning enabled by pre-class instructional materials, such as pre-
recorded videos, even during confinement and closures of campus. They could then look
forward to increased engagement in collaborative learning once classroom teaching was
allowed [17]. However, the flipped classroom approach was not common in HEIs world-
wide or in China before the onset of the COVID-19 pandemic [45]. HEIs in China have long
adopted teacher-centric didactic pedagogy in which programmes are primarily conducted
face-to-face with scheduled timetables [46]. The COVID-19 pandemic precipitated an
abrupt transition in traditional classroom teaching approaches. Thus, additional research
with a solid theoretical foundation is required to investigate the impact of this abrupt
pedagogical change on learning, mainly learner achievement and engagement [24,47]. This
is especially important during the interruptions and frequent transitions between pre-class
and in-class modes of learning caused by the COVID-19 pandemic [48–50].

2.2. Self-Determination Theory

Motivation is the main catalyst for sustainable learning behaviour [51]. SDT posits
that humans inherently possess the propensity to be curious and interested in learning
Sustainability 2022, 14, 5428 4 of 23

and developing [52,53]. The desire for sustainable development of business executives and
entrepreneurs helps to attain the SDGs of the country [54]. SDT states that when the three
basic psychological needs of learners (i.e., autonomy, competence, and relatedness) are
fulfilled, they are motivated and are more likely to engage in education contexts [55,56].
Learner engagement is characterised by learners’ investment in learning, desire to exceed
standard requirements, and preference for challenges of varying difficulty levels [57].
Learner engagement includes behavioural, emotional, and cognitive dimensions [58].
Studies by Abeysekera and Dawson [40] and Sergis et al. [59] indicate that SDT could
be drawn upon to support learning in flipped classroom environments. As less class time
is consumed by lectures, learners have more opportunities for self-regulated, independent,
peer-to-peer collaborative learning and hands-on problem-solving activities for knowledge
transfer [59]. Hence, flipped classrooms facilitate autonomy, and supportive feedback from
teachers and peers promotes competence and relatedness [60].
A recent study of HEIs during the COVID-19 pandemic found that learning persistence
and sustainability were also directly impacted by learners’ needs for autonomy, related-
ness, and competence [61]. Therefore, higher engagement resulting from motivation, as
explained by SDT, is a prerequisite for the sustainability of education programmes during
the COVID-19 pandemic [62]. Supportive feedback from teachers is especially crucial for
engaging learners and can be conveyed via game-design elements such as points, badges,
and leaderboards. Learner engagement is positively related to perceived learning and
sustainable learning [5,63,64]. Thus, flipped and gamified classrooms based on the SDT
framework may promote learning through knowledge transferability in problem-solving
activities and collaborative and active learning [19,65]. Furthermore, the adoption of flipped
classrooms and game-design elements based on SDT has the potential to promote SLE in
HEIs during the COVID-19 pandemic [4,66].

2.3. Gamification
Gamification is the use of game-design elements in a non-game environment [67],
which can be a sustainable method to achieve the United Nations’ Sustainable Development
Goals for quality education (SDG 4) [68]. In the context of education, both gamification
and SDT aim to promote learner achievement and engagement [69]. Gamification is typi-
cally used in flipped classrooms to provide feedback to encourage learners to take on task
challenges with progressively increasing difficulty levels and to motivate independent and
collaborative learning [70]. Feedback and challenges are important to promote learning
achievement and engagement [71]. A progressive increase in the difficulty level intrinsi-
cally motivates learners to complete more challenging tasks, especially those tasks that
are relevant to their personal goals [54,72]. One major personal goal for learners in higher
education business programmes is to translate knowledge to practice (i.e., knowledge
transferability), as it enhances their employability [73]. Gamification promotes active learn-
ing and engagement in problem-solving activities that bridge the knowledge-to-practice
gap [74,75]. Game-design elements engage learners by helping them to build new knowl-
edge (i.e., renewing and relearning) and support learning achievement [70]. Therefore,
pedagogies with gamification promote SLE (i.e., renewing and relearning, collaborative
and active learning, and knowledge transferability) [4,76].
The most common game-design elements used to motivate learning in higher edu-
cation are points, badges, and leaderboards [22]. These game-design elements serve the
following specific purposes [77] (Table 1):
1. Points capture granular feedback directly related to learners’ specific actions, e.g., par-
ticipation in in-class learning exercises and pursuing optional and challenging tasks.
2. Badges capture sustained feedback to recognise learners’ progress and contributions
to the activity group tasks.
3. Leaderboards capture cumulative feedback on a series of actions performed by the
learner and their contributions to completing tasks by displaying the number of
badges earned in activity groups during the entire course.
 2.
2. Badges capture
Badges capture sustained
sustained feedback
feedback to
to recognise
recognise learners’
learners’ progress
progress and
and contributions
contributions
2. Badges capture sustained feedback to recognise learners’ progress and contributions
to the
to the activity
the activity group
activity group tasks.
group tasks.
tasks.
to
3.
3. Leaderboards capture
Leaderboards capture cumulative
cumulative feedback
feedback onon a series
series of
of actions
actions performed
performed byby the
the
3. Leaderboards capture cumulative feedback on aa series of actions performed by the
learner and
learner and their
their contributions
contributions to
to completing
completing tasks
tasks by
by displaying
displaying the
the number
number ofof
Sustainability 2022, 14, 5428 learner and their contributions to completing tasks by displaying the5 of number
23 of
badges earned
badges earned inin activity
activity groups
groups during
during the
the entire
entire course.
course.
badges earned in activity groups during the entire course.
Table 1.
Table 1. Game-design
Game-design elements
elements and
and feedback.
feedback.
Table 1. Game-design elements and feedback.
Table 1. Game-design elements and feedback.
Game-Design Ele-
Game-Design Ele-
Game-Design Ele- Picture
Picture Feedback
Feedback Description
Description
ment
Game-Design ment
Element Picture
Picture Feedback
Feedback Description
Description
ment
---- Rewards
Rewards
Rewards
Rewards
for
for
for for specific
specific
specific actions
actions
actions
specific
and participa-
and
and and
actions
participa-
participa-
tion in
tion in learning
in learning exercises.
learning exercises.
participation exercises.
in learning exercises.
Points
PointsPoints Granular feedback
Granular feedback
Granular feedback tion
Points Granular feedback --- Encouragement
Encouragement
Encouragement for pursuing optional
for pursuing
for pursuing and and
optional
optional and
- increasing
Encouragement
difficulty for pursuing
levels of tasks.optional and
increasing difficulty
increasing difficulty levels
levels of
of tasks.
tasks.
increasing difficulty levels of tasks.
--- Rewards
Rewards
Rewards
Rewards
and
and
andand recognition
recognition
recognition for progress
for progress
for progress
recognition for and and
progress
and
and
Badges
Badges
Badges Sustained
Sustained feedback
feedback
Sustainedfeedback
feedback contributions
contributions to problem-solving
to problem-solving
contributions problem-solving exercises
exercises
to problem-solvingexercises in the
in the
exercises in the
Badges Sustained contributions to
activity
activity groups
in the
groups (e.g.,
activity(e.g., case
groups studies).
(e.g.,
case case studies).
studies).
activity groups (e.g., case studies).

Leaderboards Cumulative feedback

feedback -- Ranking
--
Ranking of
Ranking
Ranking
of learners
learners in
of learners
of learners
in activity
activity groups
in activity
in activity
groups ac-
ac-
groups ac-
groups
Leaderboards
Leaderboards
Leaderboards Cumulative
Cumulative
Cumulative feedback
feedback cording to total badges earned.
cording
cording to total
according badges
to total
to total badges earned.
badges earned.
earned.

Gamification can
Gamification can be be adopted
adopted in in flipped
flipped classrooms
classrooms to to motivate
motivate learners
learners to to increase
increase
Gamification can be adopted in flipped classrooms to motivate learners to increase
Gamification
their efforts
their efforts can
and
efforts and be adopted
participation
and participation in flipped
during
participation during
during the classrooms
the COVID-19
the COVID-19 to motivate
pandemic
COVID-19 pandemic learners
pandemic [78].[78]. to
Recent
[78]. Recent increase
studies
Recent studies found
studies found
found
their
their efforts
that and participation
gamification
that gamification
gamification [79][79] during
and
[79] and
and flipped the
flipped COVID-19
classrooms
flipped classrooms
classrooms [70]pandemic
[70] [78].
support
[70] support Recent
positive
support positive studies
learning
positive learning found
achievement
learning achievement
achievement
that
that gamification
and [79] andFurther,
and engagement.
engagement. flippedSDT
Further, classrooms
SDT provides
provides [70]
thesupport
the positive
theoretical
theoretical learning
foundation
foundation onachievement
on which gamification
which gamification
and engagement. Further, SDT provides the theoretical foundation on which gamification
and engagement.
and flipped
flippedFurther,
classroomSDTpedagogies
provides thecan
pedagogies theoretical
can be groundedfoundation
grounded [59,80].on which gamification
Nevertheless, there isis a pau-
pau-
and flipped classroom pedagogies can be grounded [59,80]. Nevertheless, there is aa pau-
and classroom be [59,80]. Nevertheless, there
and flipped
city classroom
city of research
of research pedagogies
on can be grounded
on gamification
gamification applied in
applied [59,80].
in flipped
flipped Nevertheless,
classrooms there
classrooms is a paucity
for business
for business education
education
city of research on gamification applied in flipped classrooms for business education
of research
among
amongon gamification
[81]. applied in flipped classrooms for business education among
adults [81].
adults
among adults [81].
adults [81].
3. Research
3. Research Methods
Methods
3.
3. Research Research
Methods Methods
We adopted
We adopted an an explanatory
explanatory sequential
sequential design
design with
with a mixed-methods
mixed-methods approach approach in in
Wewhich We
adopted adopted an explanatory
anquantitative
explanatory sequentialsequential
design design
with with aa mixed-methods
a mixed-methods approach approach
in in
both
which both
both quantitative
quantitative and and qualitative
and qualitative research
qualitative research methods
research methods
methods were were used.
were used.
used. WhenWhen
When the the quantita-
the quantita-
quantita-
which
which both
tivequantitative
tive phase is
phase
and qualitative
is followed
followed by the
by
research methods
the qualitative
qualitative phase, it
phase, itwere
helpsused.
helps
When the
to explain
to explain the quantitative
the mechanism behind
mechanism behind
phase istive phase is
followed byfollowed by the phase,
the qualitative qualitative phase,
it helps it helpsthe
to explain to explain
mechanism the mechanism
behind the behind
the quantitative
the quantitative results
results [82].
[82]. This
This approach
approach can can also
also be
be adopted
adopted to to provide
provide an an insightful
insightful
the quantitative
quantitative results [82].results
This [82]. This approach
approach can also can also be adopted
be adopted to provideto provide an insightful
an insightful
comparison between
comparison between flipped
flipped classrooms
classrooms with with gamification
gamification to to traditional
traditional classrooms
classrooms in in the
the
comparison
comparison betweenbetween
flipped flipped
classrooms classrooms with gamification
with gamification to traditional
to traditional classrooms classrooms
in the in the
existing
existing study
study [60].
[60]. In this
In this section,
this section,
section, we
we first introduce
first introduce
introduce participants,
participants, followed
followed by the
by the research
the research
research
existingexisting study
study [60]. In [60]. In
this section, we firstwe first
introduce participants,
participants, followed followed by
by the research
design, data
design, data collection,
collection, andand analysis.
analysis.
design, design, data collection,
data collection, and analysis.
and analysis.
3.1. Participants
3.1. Participants
3.1. Participants
3.1. Participants
Participants
Participants
Participants were adult were
were adult business
adult
business business
executives executives
executives
(aged 25–42 (aged
(aged 25–42
25–42
years) years) studying
years)
studying studying
postgraduate postgradu-
postgradu-
Participants were adult business executives (aged 25–42 years) studying postgradu-
businessate
ate business
business programmes
programmes of
of HEI
HEI in
in eastern
eastern China.
China. Their
Their participation
participation was
was voluntary,
voluntary, and
and
ateprogrammes of HEI in of
business programmes eastern
HEI inChina.
easternTheir
China.participation was voluntary,
Their participation and
was voluntary, and
participants
participants
participants could
could withdraw withdraw
could withdraw
withdraw
from the from
fromstudythe study
thewithout without
study without
without
negative negative
negative consequences
consequences
consequences at any
at any
at any time. time.
any time.
time.
participants could from the study negative consequences at
They
They
They were were assigned
were
assigned assigned
to one of to
to
the one
one
threeofdifferent
of the three
the threeinstructional
different instructional
different instructional
approaches approaches
approaches
(GFC, n = 25; (GFC,
(GFC,
NFC, n == 25;
n 25;
They were assigned to one of the three different instructional approaches (GFC, n = 25;
NFC,
NFC,nn n = 24;
n===19)
24;for GTC,
GTC, n
n = 19) for the second module in weeks 6–10. For flipped classrooms
n = 24; GTC,
NFC, 24; the n
GTC, == 19)
19) module
second for the
for the second
second
in weeks module
6–10. For
module in weeks
in weeks 6–10.
flipped6–10. For flipped
classrooms
For flipped classrooms
(i.e., GFC
classrooms
(i.e.,the
(i.e.,
and NFC), GFC
GFC and NFC),
and
learners NFC),
were the
the learnerswith
learners
provided werepre-class
were providedinstructional
provided with pre-class
with pre-class instructional
instructional
videos recorded videos
videos
by rec-
rec-
(i.e., GFC and NFC), the learners were provided with pre-class instructional videos rec-
orded
orded
the teacher viaby
by
the the
the teacher via
teacher
institution’s via the institution’s
the
learning institution’s
management learning
learning
system management
management
(Moodle) four system
system
weeks (Moodle)
(Moodle)
before four
four
orded by the teacher via the institution’s learning management system (Moodle) four
weeks before
weeks
the scheduled before
in-class the
the scheduled
scheduled
lessons. in-class
in-class
Thein-class lessons.
lessons.
videos lessons. The videos
The
were intended videos were intended
were
to impart intended to impart
to
basic knowledge impart basic
basic
weeks before the scheduled The videos were intended to impart basic
knowledge
before the
knowledge before
in-class before
lessons.
before thethe
the in-class
After lessons.
watching
in-class lessons. theAfter watching
videos,
After the
the learners
watching videos,
could
the videos,
videos, the learners
thecomplete could
learners self- com-
could com-
com-
knowledge in-class lessons. After watching the the learners could
plete
evaluation
plete self-evaluation
multiple-choice
self-evaluation multiple-choice
quizzes during their
multiple-choice quizzes
free time.
quizzes during
Owing
during their freehigh
to the
their free time.
time. Owing
degree
Owing to the
of trust
to the high
high
plete self-evaluation multiple-choice quizzes during their free time. Owing to the high
degree of
and coordination
degree of trust
trust
between and the
and coordination
government
coordination between
and
between the government
government
Chinese
the higher and Chinese
Chinese
education
and higher[14],
institutes
higher education
education
degree of trust and coordination between the government and Chinese higher education
institutes
universities
institutes [14], universities
reopened
[14], universities
soon after localreopened
COVID-19
reopened soon cases
soon after local
after local COVID-19
wereCOVID-19 cases
contained.cases
COVID-19 werehygiene
Rigorous
cases were contained. Rig-
contained. Rig-
institutes [14], universities reopened soon after local were contained. Rig-
measuresorous hygiene
enabled
orous hygiene the measures
resumption
hygiene measures enabled
of
measures enabled regular
enabled the the resumption
face-to-face
the resumption
resumption of of
lessons regular
in face-to-face
classrooms.
of regular
regular face-to-face Teacherslessons
face-to-face lessons
lessons in in
in
orous
focused more on advanced topics during the in-class lessons once the institute was allowed
to reopen. In this study, the learners in the flipped classrooms spent more in-class time on
learning activities such as knowledge application for solving real and simulated business
problems and assignment discussions.
The learners in the gamified classrooms (i.e., GFC and GTC) were given an account
and access code to Qitoupiao. Guidelines and descriptions of expectations throughout the
 classrooms. Teachers focused more on advanced topics during the in-class lessons once
the institute was allowed to reopen. In this study, the learners in the flipped classrooms
spent more in-class time on learning activities such as knowledge application for solving
Sustainability 2022, 14, 5428 real and simulated business problems and assignment discussions. 6 of 23
The learners in the gamified classrooms (i.e., GFC and GTC) were given an account
and access code to Qitoupiao. Guidelines and descriptions of expectations throughout the
pre-class (GFC only) and in-class learning stages (GFC and GTC) were provided. These
pre-class (GFC only) and in-class learning stages (GFC and GTC) were provided. These
learners were not graded or provided marks for the points or badges they gained in the
learners were not graded or provided marks for the points or badges they gained in the
learning
learningactivities to prevent
activities them
to prevent from
them aiming
from at high
aiming scores
at high instead
scores of being
instead motivated
of being motivated
by by
thethe
game-design elements. A summary of the guidelines and necessary information
game-design elements. A summary of the guidelines and necessary information was
was made
made available
available onon Moodle,
Moodle, asas shown
shown inin Figure
Figure 1. 1.

In-class learning activities

- Attend case study lessons and attempt to solve the

problems in the cases (GFC, NFC & GTC)

- Attend knowledge application and assignment discussion

Pre-class learning activities (for GFC & NFC) lessons and attempt to solve the problems and challenges
in assignment questions (GFC & NFC only)
- Watch the lecture videos in Moodle
- Results can be presented in the following three ways (GFC,
- Review and study the instructional contents NFC & GTC):
- Do the self-check multiple-choice quizzes a) a summary report, or
b) a summary report with comments and real examples, or
c) a plan to solve the problems

- Obtain badges by innovative ideas and solutions, number

of badges accumulated is displayed on Qitoupiao
leaderboard (GFC & GTC)

(a) (b)
Figure 1. Summary of guidelines. (a) Guidelines for pre-class learning activities; (b) Guidelines for
Figure 1. Summary of guidelines. (a) Guidelines for pre-class learning activities; (b) Guidelines for
in-class learning activities.
in-class learning activities.
Participants in the
Participants in three classrooms
the three had tohad
classrooms submit a post-class
to submit assignment
a post-class in an essay
assignment in an
of around 2000 words after completing classroom lessons at the end of each module.
essay of around 2000 words after completing classroom lessons at the end of each module. As-
signment questions
Assignment mainly
questions focused
mainly on knowledge
focused application
on knowledge for for
application solving realreal
solving business
business
problems the learner s are facing at work. Furthermore, all assignment
problems the learners are facing at work. Furthermore, all assignment questions questions were
were
evaluated and approved by the programme academic team and external
evaluated and approved by the programme academic team and external examiners. examiners.

3.2.3.2.
Research Design
Research Design
TheThestudy
studywas wasconducted
conducted in in
thethecontext of of
context adult postgraduate
adult postgraduate business
business education
education
programmes
programmes in China
in China during thethe
during COVID-19
COVID-19 pandemic
pandemic from November
from November 2021 to January
2021 to January
2022.
2022. TheThe programmes
programmes consisted
consisted of two
of two modules.
modules. Each
Each module
module lasted
lasted forfor
fivefive weeks
weeks andand
consisted
consisted of of
1616h hofofin-class
in-classlessons
lessons (eight hours
hourseacheachday
dayfor Day-1
for Day-1and Day-2).
and Three
Day-2). kinds
Three
of intervention,
kinds of intervention,namely
namelythe the
gamified
gamified flipped classroom
flipped classroom(GFC, experimental
(GFC, experimental group-1),
group-the
1), non-gamified
the non-gamifiedflipped classroom
flipped (NFC,(NFC,
classroom experimental group-2),
experimental and theand
group-2), gamified traditional
the gamified
classroom
traditional (GTC, experimental
classroom group-3),
(GTC, experimental were introduced
group-3), in the second
were introduced module,
in the second which
mod-
ule,started
which at the sixth
started at the week.
sixth To evaluate
week. the influence
To evaluate of flipped
the influence classrooms
of flipped on learner
classrooms on
achievement
learner achievementand engagement
and engagement (i.e., (i.e.,
RQ 1), RQwe1),compared
we compared the GFC and and
the GFC GTCGTC experimental
exper-
groups,
imental and toand
groups, evaluate the influence
to evaluate the influenceof gamification on learner
of gamification achievement
on learner and learner
achievement and
engagement (i.e., RQ 2), we compared the GFC and NFC experimental groups. The
experimental design is summarised in Table 2.
 The experimental design is summarised in Table 2.

Table 2. Experimental design.

Experimental Experimental Group 1: Experimental Group 2: Experimental Group 3: Research Question

Sustainability 2022, 14, 5428 7 of 23
Groups and Ap- Gamified Flipped Class- Non-Gamified Flipped Gamified Traditional and Group Com-
proaches room (GFC) Classroom (NFC) Classroom (GTC) parisons
Flipped class-
YesTable 2. Experimental design.Yes No RQ1: GFC and GTC
room
GamifiedExperimental
class- Experimental Group 1: Experimental Group 2: Experimental Group 3: Research Question
Groups and Yes
Gamified Flipped No Flipped
Non-Gamified Yes
Gamified Traditional RQ2:
and GFC and
Group NFC
roomApproaches Classroom (GFC) Classroom (NFC) Classroom (GTC) Comparisons
Flipped classroom Yes Yes No RQ1: GFC and GTC
Gamified classroom 3.2.1. Class
Yes Rundown No Yes RQ2: GFC and NFC
All in-class lessons were conducted face-to-face after city lockdowns were lifted. In the
first3.2.1.
module
Class(weeks
Rundown 1–5), the students in all three experimental groups learned under the
traditional classroom approach
All in-class lessons (i.e., teacher-centric
were conducted face-to-facedidactic
after cityapproach
lockdownswith
werenolifted.
pre-class
In vid-
eos the
andfirst
gamification).
module (weeks The assignment
1–5), the students marks
in allfrom
three the first module
experimental were
groups recorded
learned underas pre-
the traditional
intervention classroom
references. approach
The (i.e., teacher-centric
three instructional didactic approach
interventions with no for
were adopted pre-class
the respec-
tive experimental groups in the second module during weeks 6–10 (Figure 2). as
videos and gamification). The assignment marks from the first module were recorded For the
pre-intervention references. The three instructional interventions were adopted
flipped classrooms (GFC and NFC), eight sessions of pre-recorded instructional videos with for the
respective experimental groups in the second module during weeks 6–10 (Figure 2). For
a duration of 30 min each were recorded by the module teacher and provided via the insti-
the flipped classrooms (GFC and NFC), eight sessions of pre-recorded instructional videos
tution’s
with alearning
durationmanagement
of 30 min eachsystem (Moodle).
were recorded Self-evaluation
by the module teacher multiple-choice
and provided via quizzes
withtheonly ten questions were offered to avoid heavy cognitive load in the
institution’s learning management system (Moodle). Self-evaluation multiple-choice pre-class stage of
learning
quizzes[40,83].
with only ten questions were offered to avoid heavy cognitive load in the pre-class
stage of learning [40,83].

Figure 2. Class
Figure rundown.
2. Class rundown.

3.2.2.
3.2.2. Lesson
Lesson Schedule
Schedule
The face-to-face lesson schedules were the same across the three experimental groups
The face-to-face lesson schedules were the same across the three experimental groups
for the first four morning hours of lecture lessons. This was followed by two hours of
for the
casefirst four
study morning
lessons in thehours of lecturefor
early afternoon lessons. This
all three was followed
classrooms by two
(GFC, NFC, andhours
GTC).of case
study
Thelessons in the
case study early
lessons afternoon
served for all
to expand thethree
scopeclassrooms (GFC,
of learning and NFC, and
knowledge GTC). The
through
casepeer
study lessonsand
interactive served to expand
collaborative the scope
learning. of learning
For the and knowledge
flipped classrooms (GFC andthrough
NFC), peer
there were
interactive two
and hours of knowledge
collaborative application
learning. For the and assignment
flipped discussion
classrooms (GFClessons in the there
and NFC),
late afternoon of Day-1 and Day-2, respectively. In contrast, for the non-flipped
were two hours of knowledge application and assignment discussion lessons in the late classroom
(GTC), there were lecture lessons in the late afternoon of both Day-1 and Day-2. Figure 3
afternoon of Day-1 and Day-2, respectively. In contrast, for the non-flipped classroom
provides details of the lesson schedule.
 provides details of the lesson schedule.
nability 2022, 14, x FOR PEER REVIEW 8 of 23

Sustainability 2022, 14, 5428 8 of 23

(GTC), there were lecture lessons in the late afternoon of both Day-1 and Day-2. Figure 3
provides details of the lesson schedule.

Figure 3. Lesson schedule.

The case study lessons format was the same for all three classrooms, which st
Figure 3. Lessonwith a business
schedule. case relevant to the morning lectures. Learners were grouped into act
Figure 3. Lesson schedule.
groups of five to seven peer members to discuss the case according to the guiding q
The case study Thelessons formatlessons
caseexample,
study was the samewas
format for all
thethree
sameclassrooms,
for which startedwhich started
tions. For the learners discussed theall three
case ofclassrooms,
emotional pricing and its imp
with a business case
with relevant to
a business therelevant
case morning tolectures. Learners
the morning wereLearners
lectures. groupedwereinto activity
grouped into activity
tions
groups of fivegroups
to sevenforpeer
marketing strategy
discussin Figure 4. Each toactivity group could then choose o
of five tomembers
seven peer to members the case according
to discuss the guiding
the case according to theques-
guiding questions.
three
tions. For example, levels of
the learners
For example, difficulty
discussed
the learners to present
the case the
discussed their results:
of emotional a summary
pricing and
case of emotional report
its implica-
pricing of the discussion (
and its implications
level),
tions for marketing
for real application
strategy
marketing in Figure
strategy in 4.with
Eachexamples
Figure activity
4. (Medium
group
Each activity could level),
groupthencould or
choose
then real
oneapplication with a bus
of one of three
choose
three levels oflevels
difficulty
plan of (Hardto present
level).
difficulty their results:
The learners’
to present a summary
choices
their results: report of the discussion
were recorded
a summary (Easy
report ofintheclass observation
discussion (Easy repor
level), real application
level), real
teaching with exampleswith
application
assistants. (Medium
examples level), or real level),
(Medium application with
or real a business
application with a business
plan (Hard level). The learners’
plan (Hard choices
level). The were choices
learners’ recordedwere in class observation
recorded in classreports by reports by
observation
teaching assistants.
teaching assistants.

Figure 4. Example of an in-class case study learning activity with three difficulty levels for presenting
Figure 4. Example of an in-class case study learning activity with three difficulty levels for presenting
the results.
Figure 4. Example of an in-class case study learning activity with three difficulty levels for prese
the results.
the results.
For the two flipped classrooms (GFC and NFC), there were two hours of knowledge
For the two flipped classrooms (GFC and NFC), there were two hours of knowledge
application (Figure 5) and two hours of assignment discussion (Figure 6) during the late
application (Figure 5)
Forand
thetwo hours
two of assignment
flipped classrooms discussion
(GFC (Figure
and 6) during
NFC), there the latetwo hours of knowl
were
afternoon of Day-1 and Day-2, respectively. In the knowledge application lessons, the
afternoon of Day-1 and
application Day-2, respectively.
(Figure 5) and In the knowledge
two hours application lessons, the
teachers introduced a topic or questions thatof assignment
focused discussion
on applying (Figurelearned.
the knowledge 6) during the
teachers introduced a topic or questions that focused on applying the knowledge learned.
For example, as shown in Figure 3, the learners learned different pricing strategies in
afternoon of Day-1 and Day-2, respectively. In the knowledge application lessons
the morning
teachers lessons and
introduced then attempted
a topic to apply
or questions them in real
that focused or simulated
on applying the business
knowledge lea
scenarios. These lessons were intended to strengthen their ability to apply their knowledge
 For example, as shown in Figure 3, the learners learned different pricing strategies in the
morning lessons For
andexample,
then attempted
as shown toinapply
Figurethem 3, thein learners
real or simulated business
learned different scenar-
pricing strategies in the
ios. These lessonsmorning
were intended
lessons and to strengthen
then attempted their to ability
applytothemapply in their
real or knowledge
simulated for business scenar
Sustainability 2022, 14, 5428 9 of 23
ios. Thesebusiness
solving real or simulated lessons were intended
problems, to strengthen
which was knowledgetheir ability to apply their
transferability andknowledge fo
solving
most relevant to their jobsreal
and oremployment.
simulated business Lastly,problems,
the assignment which discussion
was knowledge lessons transferability
on and
most relevant to their jobs and employment. Lastly,
Day-2 allowed the learners to discuss the individual assignment questions and learn with the assignment discussion lessons on
for solving
Day-2 allowedreal or simulated
the learners business problems,
to discussand which
the individual was knowledge
assignment transferability and
their peers collaboratively. Peer-to-peer interactive collaborative learningquestions
most relevant to their jobs and employment. Lastly, the assignment discussion lessons on
could in-and learn with
their
spire further thoughts, peers collaboratively.
ideas, and solutions Peer-to-peer
to thethe business interactive
problems and collaborative
listed in theand learning could in
assign-
Day-2 allowed the learners to discuss individual assignment questions learn with
spire further
ment questions. Learners
their peers could
thoughts,
renew and
collaboratively.
ideas, and solutions
relearn
Peer-to-peer
toand
in the process.
interactive
thecollaborative
business
Becauseproblems
the learners
learning
listed in the assign
couldininspire
ment questions.
further
the flipped classrooms thoughts,
(GFC Learners
andideas,
NFC)and could
solutions
were renew
aided toby and
the
the relearnproblems
business
knowledge in the process.
listed inBecause
application the
and as-the learners in
assignment
the flipped
questions.
signment discussion lessons, classrooms
Learners could (GFC
they subsequently renewandand NFC) were
relearn
required inaided
less the bytothe
process.
time knowledge
Becausethe
complete theapplication
learners in and as
same
the flipped classrooms (GFC and NFC) were aided by the knowledge application andthe same
signment discussion lessons, they subsequently required less time to complete
assignment writing required for all three experimental groups after the classroom lessons.
assignment
assignmentdiscussion lessons, they
writing required subsequently
for all required less
three experimental time to
groups complete
after the same lessons
the classroom
By contrast, those in GTC had
assignment writingto work onfortheir assignment writing from
after the beginning
By contrast, thoserequired
in GTC had all to
three
work experimental groups
on their assignment the classroom
writing fromlessons.
the beginning
and required moreBy time to complete it by themselves after classroom lessons.
and required more time to complete it by themselves after classroom lessons. and
contrast, those in GTC had to work on their assignment writing from the beginning
required more time to complete it by themselves after classroom lessons.

Figure 5. Example Figure

of a knowledge
Figure 5.5.Example
Exampleapplication
of of lesson.
a knowledge
a knowledge application
application lesson.
lesson.

Figure 6. Example of an assignment discussion lesson.

Figure 6. Example Figure
of an assignment
6. Example ofdiscussion lesson.
an assignment discussion lesson.
Learner surveys, interviews, class observation reports, and written feedback were
Learner surveys, Learner surveys,
interviews, interviews,
class class observation
observation reports, reports,
and and written
written feedback were
collected from all three classes at the end of Day-2 after the feedback
completion were
of all the in-class
collected
collected from alllessons from
three classes all three classes
at the end at the
of Day-2 end of Day-2 after the completion of all the in-class
of the second module (i.e., inafter
lessons of the second module (i.e., in week 10).
week the completion of all the in-class
10).
lessons of the second module (i.e., in week 10).
3.2.3.Application
3.2.3. Application ofof Game-Design
Game-Design Elements
Elements
3.2.3. Application of Game-Design
During
Duringthethecase Elements
case study
studylessons,
lessons, thethe
learners were
learners weredivided intointo
divided activity groups
activity of of 5–7
groups
5–7
During the case peers
peers perper
study group.
group. Game-design
Game-design
lessons, elements
the learnerselements were applied
were
were divided applied for
for the
into activity gamified
thegroups ofclassrooms
gamified classrooms (GFC
5–7
(GFC and GTC). To ensure that gamification was effective, points, badges, and leaderboards
peers per group. and GTC). To ensure
Game-design elementsthat gamification
were applied forwastheeffective,
gamified points, badges,(GFC
classrooms and leaderboards
were displayed through the Chinese classroom application Qitoupiao (‘Voting Together’).
were
and GTC). To ensure displayed through
that gamification the
was Chinese classroom application Qitoupiao (‘Voting Together’)
These game-design elements andeffective, points,
their use in the twobadges,
gamifiedand leaderboards
classrooms are described
These game-design
were displayed through
below andthe
inChinese
elements and their use in the two gamified classrooms are described
Table 3. classroom application Qitoupiao (‘Voting Together’).
below and in Table 3.
These game-design elements and their use in the two gamified classrooms are described
below and in Table 3.
 ers
ers were
ers were also
were also encouraged
also encouraged
encouraged to to compare
to compare their
compare their performance
their performance with
performance with those
with those of
those of other
of other activity
other activity
activity
groups
groups
groups inin the
in the same
the same classroom
same classroom [87].
classroom [87]. Thus,
[87]. Thus, the
Thus, the numbers
the numbers
numbers ofof accumulated
of accumulated badges
accumulated badges of
badges of each
of each
each
activity
activity group were displayed on the class leaderboard. The leaderboard
activity group were displayed on the class leaderboard. The leaderboard was in-
group were displayed on the class leaderboard. The leaderboard was
was in-
in-
tended
tended to
to inspire
inspire intragroup
intragroup peer
peer collaborative
collaborative learning
learning and
and a
a healthy
healthy
tended to inspire intragroup peer collaborative learning and a healthy intergroup intergroup
intergroup
competition
competition amongst
competition amongst
amongst the the groups
the groups to
groups to achieve
achieve aaa prominent
to achieve prominent position
prominent position on
position on the
on the leaderboard
the leaderboard
leaderboard
Sustainability 2022, 14, 5428 by
by contributing
by contributing
contributing toto learning
to learning activities
learning activities for
activities for more
for more badges
more badges [88,89].
badges [88,89].
[88,89]. 10 of 23

Table
Table 3.
Table 3. Application
3. Application of
Application of game-design
of game-design elements
game-design elements for
elements for gamified
for gamified classrooms
gamified classrooms (GFC
classrooms (GFC and
(GFC and GTC).
and GTC).
GTC).
Game-Design Ele- Table 3. Application of game-design elements for gamified classrooms (GFC and GTC).
Game-Design
Game-Design Ele-
Ele- Purpose
Purpose
Purpose How
How They
How They Were
They Were Awarded
Were Awarded
Awarded
ment
ment
ment
Game-Design Element Purpose How They Were Awarded
Point(s)
Point(s)
Point(s) for
for in-class
for in-class learning
in-class learning
learning and and presentation
and presentation activities
presentation activities based
activities based
based onon
on
Points
Points
Points Feedback
Feedback
Feedback toto encourage
to encourage
encourage tasktask
task level Point(s) for in-class learning and presentation activities based on
Points Feedback to encourage task level level of
of
of difficulty:
difficulty:
difficulty:
completion
completion
completion and
and
andandprogress
progress
progress to-  level
to-
to- of difficulty:
completion progress 
 •
Easy—one
Easy—one point
Easy—onepoint
Easy—one for
pointfor
point for a
for a summary
summary
aa summary
report
report
summaryreport
report
wards
wards
wards more
more
more
towards moreadvanced
advanced
advanced
advanced and
and
and
and 

 • Medium—two
Medium—two
Medium—two
Medium—two points
points
points
points for
for
for
for real
real
real
real application
application
application
application with
with
with
with examples
examples
examples
examples
challenging
challenging learning
learning
challenging
challenging activities.
learningactivities.
learning activities.
activities. 
 Hard—three
 • Hard—three
Hard—three points
Hard—threepointspointsfor
points for
for real
for real
real application
real application with
applicationwith
application with
with aa business
aa business
business plan
plan
business plan
plan
Badges
Badges
Badges
Badges Recognition
Recognition
Recognition of of learning
of learning efforts,
learning efforts,
efforts,
Recognition of
participation,
participation, learning
and
and efforts, Badges
contribu-
contribu- Badges were
were given
given to learners who presented good ideas, new
participation, and contribu- Badges
Badges were
were givento
given tolearners
to learners
learners whowho presented
whopresented
presented good
good
good ideas,
newnew
ideas,
ideas, new
participation,
tions; promoting and contributions;
social recogni- knowledge or innovative solutions to a problem.
tions; promoting
tions;promoting
promoting social recogni-
social recogni- knowledge
knowledge
knowledgeor or innovative
orinnovative solutions to
solutionstoto
innovative solutions a problem.
a problem.
a problem.
social recognition.
tion.
tion.
tion.
Leaderboards
Leaderboards
Leaderboards
Leaderboards Inspire
Inspire intragroup
intragroup peer learn-
Inspire
Inspire intragroup peer
intragroup peer learn-
learn-
peer learning The
The number
The number of
number of accumulated
of accumulated badges
accumulated badges obtained
badges obtained in
obtained in each
in each activity
each activity
activity
ing
ing and
ing and intergroup
intergroup
andintergroup
and intergroup competition
competition
competition
competition for The number of accumulated badges obtained in each activity group.
group.
group.
group.
for
for learning
for learning
learning advancement.
learningadvancement.
advancement.
advancement.

3.3.
3.3. Data
3.3. Data Collection
Data Collection
Collection
1. Points: Learners received points on completing tasks in the in-class learning activities.
Both
Both
Points quantitative
Both quantitative
quantitative and
and
serve as feedback qualitative
and qualitative
qualitative
to encourage data
data were
were collected
were collected
data learnerscollected during
during the
the in-class
the in-class
duringsubsequent
to complete and
and post-
and post-
in-classlearning
post-
class
class
class stages
stages (Table
(Table
stages (Table
activities 4).
4). Class
Class
4). Class
and achieve observation
observation
observation
more advanced reports
reports
reports were
were
challenges collected
collected
were collected by
by
together by teaching
teaching
in teaching assistants
assistants
assistants
a group [84,85]. dur-
dur-
dur-
One
ing
ing the
the
ingpoint in-class
the in-class
in-class stages.
stages.
was allotted Data
stages.toData
Data were
were from
from
from the
weregroups
activity the
the
thatpost-class
post-class stage
completestage
post-class and
theirand
stage and were
were collected
were collected
discussion collected based
based on
based on
and presented on
three
three
three sources:
thesources:
sources: learner
results learner
learner
as survey,
survey, learner
learner
survey,report,
a summary interview,
interview,
learnerwhich
interview, and
was and
and assignment
assignment marks.
level marks.
assignment
the lowest marks. The
The survey
The survey
of difficulty used
(Easy)used
survey used
to
aaa 5-point
5-point
5-point
complete Likert
Likert
Likert scale
theirscale
scale that
task. that
Two ranged
that ranged
ranged
and threefrom
from
from ‘Strongly
‘Strongly
‘Strongly
points Agree’
Agree’
were allotted (5)
Agree’to(5)
(5) through
through
through
activity ‘Neutral’
‘Neutral’
‘Neutral’
groups (3)
(3) to
(3) to
that chose to
the intermediate (Medium) or the most challenging (Hard) levels of difficulty to
present their discussion results in a real application with examples and a business
plan, respectively.
2. Badges: Learners received badges when they provided innovative ideas and solu-
tions to problems and questions during in-class learning activities. Badges serve as
recognitions of a learner’s contributions and encourage participation during learning
activities [60]. Badges also promote social validation as they provide opportunities
for learners to show their conformity and progress towards the expected learning
behaviour with their peers [86]. Learners in an activity group obtained one badge
when any member of the group presented a good idea, new knowledge, or an in-
novative solution to a problem or question. A ‘good idea’ refers to a new way to
apply the knowledge learned in the class, ‘new knowledge’ refers to a point that was
not mentioned or taught in the class, and an ‘innovative solution’ is a solution to a
problem-solving task that was not mentioned or taught in the class.
3. Leaderboards: Teamwork was encouraged within each activity group and the learners
were also encouraged to compare their performance with those of other activity
groups in the same classroom [87]. Thus, the numbers of accumulated badges of
each activity group were displayed on the class leaderboard. The leaderboard was
intended to inspire intragroup peer collaborative learning and a healthy intergroup
competition amongst the groups to achieve a prominent position on the leaderboard
by contributing to learning activities for more badges [88,89].

3.3. Data Collection

Both quantitative and qualitative data were collected during the in-class and post-class
stages (Table 4). Class observation reports were collected by teaching assistants during
the in-class stages. Data were from the post-class stage and were collected based on three
sources: learner survey, learner interview, and assignment marks. The survey used a
5-point Likert scale that ranged from ‘Strongly Agree’ (5) through ‘Neutral’ (3) to ‘Strongly
Disagree’ (1). The learner survey and interview focused on the learners’ perceptions and
their suggestions for pedagogical improvement.
Sustainability 2022, 14, 5428 11 of 23

Table 4. Data sources and collection.

Stage Data Purpose RQs Addressed

Class observation To evaluate learner engagement
report for levels of under the three instructional Learner engagement
In-class
difficulty and approaches (GFC, NFC, (RQ1 and RQ2)
participation and GTC).
Learner survey To evaluate learners’ perceived Learner perceived
achievement and engagement achievement and
Learner interview under the three engagement
instructional approaches. (RQ1 and RQ2)
Post-class
To evaluate the learner’s overall
Learner achievement
Assignment marks achievement under the three
(RQ1 and RQ2)
instructional approaches

The effects of classroom approaches (i.e., GFC, NFC, and GTC) on learner achievement
were evaluated based on the assignment marks. The teachers graded the assignment strictly
according to the marking scheme and rubrics provided by the institute. All in-class learning
activities, topics and assignment questions were approved by the school appointed external
examiner and the academic programme committee. Thirty per cent of the assignment
marks were countermarked and sent to the external examiner for final review to ensure
accurate assessment. Discrepancies in marks were discussed and were resolved in the
board of examiners’ meetings.
The learners were invited to complete a survey at the end of classroom lessons in
the second module. The survey consisted of questions on four themes: (i) perceived
learning (Items 1–3), (ii) behavioural engagement (Items 4–8), (iii) emotional engagement
(Items 9–13), and (iv) cognitive engagement (Items 14–17). It also contained one open-
ended question (Item 18). The learners were free to respond, and their responses were
tagged using anonymised labels, e.g., GFC-Learner 1, NFC-Learner 2, and GTC-Learner 3.
Sample items for each theme from the survey are presented in Table 5.

Table 5. Sample items of the learner survey.

Survey Items Supporting Citation

Perceived learning (Items 1–3)
Lo and Hew [60]
I learned more because of the classroom format. (Item 2)
Behavioural engagement (Items 4–8)
Skinner et al. [90]
I participated in in-class activities and discussions. (Item 6)
Emotional engagement (Items 9–13)
Skinner et al. [90]
The class was fun. (Item 11)
Cognitive Engagement (Items 14–17)
Rotgans et al. [91]
I was so involved that I forgot everything around me. (Item 17)

Qualitative data sources included Item 18 of the learner survey, class observation
reports, and learner interviews. Class observation was adopted for evaluating learner en-
gagement in in-class activities in accordance with the recommendation of a prior study [92].
Studies have found learners’ behavioural and cognitive engagement to be reflected in their
participation in learning activities [60,93]. Another study found that the willingness and
effort to exceed minimum requirements and adopt more challenging presentation methods
indicated a higher learning engagement [94]. Thus, teaching assistants recorded class
observation reports which focused on two aspects of the learners’ engagement: (1) the level
of difficulty that they chose for their case study results presentation and (2) their learning
participation. Learning participation ranges from passive receiving, active manipulating,
and constructive generating to the most engaging interactive dialoguing [92]. Learner
 Sustainability 2022, 14, x FOR PEER REVIEW 12 of 23

Sustainability 2022, 14, 5428 (2) their learning participation. Learning participation ranges from passive receiving, 12 ofac-
23
tive manipulating, and constructive generating to the most engaging interactive dialogu-
ing [92]. Learner engagement and their participation also reflected the different compo-
nents of SLEand
engagement [4]. Teaching assistants also
their participation recorded the learners’
reflected participation
the different components in in-class
of SLEactiv-
[4].
ities by checking
Teaching assistantsone box each
recorded the for the level
learners’ of difficulty
participation in and level
in-class of participation,
activities by checkingrespec-
one
tively,
box eachthat
for most closely
the level matchedand
of difficulty their observations
level of classes
of participation, during that
respectively, the most
in-class case
closely
matched their observations
study lessons (Figure 7). of classes during the in-class case study lessons (Figure 7).

Figure 7. Class observation report form with SLE components.

Figure 7. Class observation report form with SLE components.

Interviewshelp
Interviews helpto tounderstand
understandlearners’
learners’behaviour,
behaviour,feelings,
feelings,and andinterpretation
interpretationof ofthe
the
learning environment
learning environment [95].[95]. Therefore,
Therefore, we we conducted
conducted learner
learnerinterviews
interviews after
afterclassroom
classroom
lessons in the second module (Week 10). The interview adopted
lessons in the second module (Week 10). The interview adopted a semi-structured approach a semi-structured ap-
proach with a protocol (Table 6). The topics covered in the interview
with a protocol (Table 6). The topics covered in the interview were based on the engagement were based on the
engagement framework of Fredricks et al. [96] and McCallum et al.
framework of Fredricks et al. [96] and McCallum et al. [97]. Topics for exploring learner [97]. Topics for explor-
ing learner engagement
engagement included (1) included
behavioural(1) engagement
behavioural (e.g.,engagement (e.g., task participation
task participation and completion), and
completion),
(2) emotional (2) emotional(e.g.,
engagement engagement
enjoyment (e.g.,
or enjoyment
boredom), or and boredom), and engagement
(3) cognitive (3) cognitive
engagement
(e.g., investing(e.g.,
effortinvesting effort
in learning). Allininterviews
learning). wereAll interviews
conductedwere conducted
and recorded and rec-
in Chinese
orded
by in Chinese by
the researcher. Some thetranscripts
researcher.were Sometranslated
transcriptsinto
were translated
English into English
for reporting for re-
purposes.
porting
The purposes.were
interviewees Theinvited
interviewees
to check were invited to
all original check all original
transcriptions, transcriptions,were
and discrepancies and
discrepancies were corrected to ensure accuracy. Learners’ participation
corrected to ensure accuracy. Learners’ participation in the survey and interview was in the survey and
interview and
voluntary, wasnovoluntary,
incentiveandwasno incentive
offered. Afterwastheoffered. Afterofthe
completion thecompletion of thelessons,
second module second
module
we received lessons, we received
responses responses
for 49 surveys (GFC forn49 surveys
= 20, NFC n(GFC
= 17,nGTC= 20,nNFC
= 12)nand
= 17,
26GTC n=
learner
12) and 26 (GFC
interviews learner
n =interviews
7, NFC n = (GFC
11, GTCn = 7,nNFC
= 8). n = 11, GTC n = 8).

Table6.6.Sample
Table Samplequestions
questionsofofthe
thesemi-structured
semi-structuredlearner
learnerinterview
interviewprotocol.
protocol.

Dependent
Dependent Citation Refer-
Question
Question Citation Reference
Variable
Variable ence
Howdid
How did
thethe classroom
classroom approach
approach change
change the way
the way
Behavioural
Behavioural
you
Fredricks et al.
engagement youprepared
prepared forfor
studying this this
studying module differently
module Fredricks et al. [96]
differently
engagement from [96]
fromother
otheroror
previous classes
previous you have
classes attended?
you have attended?
Emotional
Emotional What
Whatdid you
did findfind
you most/least interesting
most/least in your
interesting in your
engagement studying of this module?
engagement studying of this module? MaCullum
MaCullum et al.
et al. [97]
Cognitive
Cognitive Did you do anything extra that helped your
Did you do anything extra that helped your learn- [97]
engagement learning when studying this module
engagement ing when studying this module
Sustainability 2022, 14, 5428 13 of 23

3.4. Data Analysis

3.4.1. Quantitative Data Analysis
To determine the most appropriate statistical test for quantitative data analysis, quan-
titative data were first tested for normality [98]. A Kolmogorov–Smirnov test showed a
significant deviation from normality for learner achievement (according to assignment
marks) across the three classes (first module p < 0.001; second module p = 0.017). Therefore,
non-parametric tests were used to analyse the quantitative data on learner achievement [98].
The Kruskal–Wallis non-parametric test for multiple group comparisons was per-
formed on the assignment marks from the first module, which was taught in traditional
classrooms, to check the initial equivalence of the three experimental groups (i.e., GFC,
NFC, and GTC). The results showed that the three experimental groups were statistically
equivalent (H = 0.256, p = 0.880) for the first module. The assignment marks from the second
module of the three experimental groups were then tested with the Kruskal–Wallis test
at a significance level of 0.05. Multiple Mann–Whitney tests were conducted for post hoc
pairwise comparisons when significant differences were found [98]. Bonferroni correction
was applied to avoid Type I errors [99]. Therefore, the post hoc analyses reported effects
at a significance level of 0.05/3 = 0.0167. The effect size (r) was then calculated using the
following formula [100]:
z
r= √
N0
where z is the z-score, and N 0 is the number of participants in the two experimental groups
in each pairwise comparison.

3.4.2. Qualitative Data Analysis

The qualitative data were transcribed in Chinese and analysed thematically by organ-
ising the data into categories. Direct quotations from participant interviews were used to
ensure data validity [101]. The qualitative analysis procedures followed the suggestion
of Creswell et al. [82]. Coding started with the shortest interview transcripts using some
exact wordings of the participants and concepts from the literature. All assigned codes
were reviewed and grouped with redundant codes to produce a preliminary list of codes,
which were then used to analyse the rest of the qualitative data. Exemplary quotes that
clearly illustrated new emerging themes were identified and added to the list of codes, and
similar codes were organised into subthemes.
Double-coding was adopted when data were descriptively and inferentially meaning-
ful but not in neat or isolated units [102,103]. Thirty percent of the qualitative data were
double-coded by an independent coder, and the intercoder agreement rate was checked.
Disagreements between the coders were resolved through discussion. The data were val-
idated by member checking to avoid any misunderstanding or misinterpretation [104].
Qualitative data were translated into English for reporting in this study.
Qualitative data from the written comments, feedback from learner surveys (Item 18),
teaching assistant interviews, and class observation reports and teacher interviews were
analysed with triangulation for comprehensive understanding [105] and to better explain
the quantitative results [82].

4. Results
4.1. Learner Achievement
The Kruskal–Wallis test indicated a significant difference between the three classrooms
in terms of the assignment marks obtained in the second module (H = 7.550, p = 0.023).
Figure 8 shows a boxplot of the results.

Sustainability 2022, 14, 5428
4. Results
4.1. Learner Achievement
The Kruskal–Wallis test indicated a significant difference between the
rooms in terms of the assignment marks obtained in the second14module
of 23 (H
0.023). Figure 8 shows a boxplot of the results.

Figure Boxplots
Figure 8.8. of assignment
Boxplots marks from
of assignment the second
marks frommodule by classroom.
the second module by classroom.
Mann–Whitney tests for pairwise comparisons showed no statistically significant
Mann–Whitney
differences between GFC tests for pairwise
and GTC (p = 0.199) comparisons
or between GFCshowed
and NFCno (p statistically
= 0.117). sig
However, the learners in GTC scored significantly higher assignment
ferences between GFC and GTC (p = 0.199) or between GFC and NFC (p = 0.117 marks than those
in NFC (U = 112.00, z = −2.875, p = 0.004) with an effect size r = 0.410. Table 7 shows the
the learners in GTC scored significantly higher assignment marks than those
results for the pairwise comparison of assignment marks between NFC and GTC.
112.00, z = −2.875, p = 0.004) with an effect size r = 0.410. Table 7 shows the re
pairwise comparison
Table 7. Pairwise ofassignment
comparison of assignmentmarks marks between
between NFC NFC
and GTC for theand GTC.
second module.

Assignment Marks n Mean SD Mean Rank Pairwise Comparison

Table 7.NFC
Pairwise comparison
24
of assignment
75.44 3.73
marks between NFC and GTC for the se
17.17
GTC > NFC *
GTC 19 78.79 3.05 28.11
Assignment Marks n
* p < 0.0167 (Bonferroni correction).Mean SD Mean Rank Pairwise Com
NFC 24 75.44 3.73 17.17
GTC > NF
Although the qualitative data from the interviews revealed several benefits of gami-
GTC 19 78.79 3.05 28.11
fication for learning achievement, the learners trusted knowledge delivered by teachers
*more
p < 0.0167 (Bonferroni
than knowledge correction).
received from their peers, regardless of whether their peers had
more firsthand experience in specific business scenarios. Interviews with both learners
and teaching
Although assistants indicated that data
the qualitative the knowledge
from the learned from teachers
interviews was reflected
revealed several bene
more frequently in the learners’ submissions of their assignments, which required them
fication
to apply theforknowledge
learninglearned
achievement, the to
in the module learners
provide trusted
a solutionknowledge delivered
to a real business
more
problemthan knowledge
in around 2000 words.received from
Table 8 shows the their peers,ofregardless
major benefits offrom
GTC identified whether
the the
learner firsthand
more interviews. These benefits included
experience the generation
in specific business of excitement
scenarios. andInterviews
curiosity and with b
the promotion of interactions and discussions in the classroom. In contrast, the learners
and teaching assistants indicated that the knowledge learned from teachers w
in NFC did not experience a sense of ‘belonging’ (NFC-Learner 8) to the class. Keywords
more frequently
that occurred in times
multiple the learners’ submissions
in the interviews offrom
of learners theirNFC
assignments, which re
were ‘boring/dry’
to apply the1knowledge
(NFC-Learners learned makes
and 7) and ‘no interaction in theme module to (NFC-Learners
sleepy/passive’ provide a solution
8 and 10). to a r
problem in around 2000 words. Table 8 shows the major benefits of GTC ide
the learner interviews. These benefits included the generation of excitement a
and the promotion of interactions and discussions in the classroom. In contra
ers in NFC did not experience a sense of ‘belonging’ (NFC-Learner 8) to the clas
that occurred multiple times in the interviews of learners from NFC were
(NFC-Learners 1 and 7) and ‘no interaction makes me sleepy/passive’ (NFC-Lea
10).
Sustainability 2022, 14, 5428 15 of 23

Table 8. Benefits of GTC.

Theme Example Quote

Excitement/curiosity ‘Very excited, and engaged’ (GTC-Learner 3)
‘Classmates encouraged each other to choose the difficult learning
Peer learning
option’ (GTC-Learner 8)
‘More feedback from the teacher, enabled by the rhythm of the
Exchanges/interactions
teaching process, made me more attentive’ (GTC-Learner 5)

4.2. Learner Engagement

The Kolmogorov–Smirnov test showed that the data from the learner surveys sig-
nificantly deviated from normality (p < 0.001); therefore, nonparametric Kruskal–Wallis
tests were performed. The difference between GFC and NFC was significant (p = 0.014)
for perceived learning. Post hoc pairwise Mann–Whitney tests showed that the learners in
GFC scored significantly higher than those in NFC (U = 102, z = −2.448, p = 0.0140), with
effect size r = −0.402. These results are shown in Table 9.

Table 9. Comparison between GFC and NFC for learner survey Item 2.

Mean Pairwise
Item Survey Question Class n Mean SD
Rank Comparison
I learned more because of GFC 20 4.80 0.41 22.40
2 GFC > NFC *
the classroom format. NFC 17 4.35 0.61 15.00
* p < 0.0167 (Bonferroni correction).

Class observation reports reflected that the learners in GFC engaged at the interactive
dialoguing level during learning activities [93]. This shows that GFC was the most engaging
of the three classrooms and performed all five SLE components. Those in NFC engaged
half their time at the active manipulation level and the other half at the constructive
generating level. This reveals that NFC was the least engaging among the three classrooms.
Gamification elevated the participation level of learners in GTC to constructive generating,
which showed three out of the five SLE components, indicating that GTC was more
engaging than NFC (Table 10).

Table 10. Level of difficulty and participation in in-class learning activities from class observation
reports with SLE components. The order of the classrooms in terms of learner engagement and the
number of SLE components that were performed was GFC > GTC > NFC.

Level of
Classroom Level of Difficulty SLE Components
Participation
Active learning
Independent learning
Interactive
GFC Medium Collaborative learning
dialoguing (100% *)
Renewing and relearning
Transferability
Active learning
Active manipulating (50% *)
NFC Medium Independent learning
Active learning
Constructive
Independent learning
generating (50% *)
Renewing and relearning
Active learning
Constructive
GTC Medium Independent learning
generating (100% *)
Renewing and relearning
* % of the time learners engaged at the level of participation.
Sustainability 2022, 14, 5428 16 of 23

Keywords that occurred multiple times in learner interviews from the gamified class-
rooms were ‘active participation’ (GFC-Learners 2 and 8), ‘focused and engaged’ (GFC-Learner 7,
GTC-Learner 3), and ‘the class was interesting, inspired my thoughts and stimulated deep learning’
(GFC-Learner 5). One point of contrast in the feedback between GFC and GTC related to
peer-dependent vs. teacher-dependent learning:
GFC-Learner 2: ‘The presence of such exceptional classmates made me more actively engaged
(in learning)’.
GTC-Learner 5: ‘In the classroom, I followed the rhythm and pace of the teacher, who enabled
me to focus during the interactive learning lesson’.
The feedback from the interviews with those in NFC, in contrast, reflected that the
learners faced certain obstacles to active participation:
NFC-Learner 1: ‘I wish the teacher could control classmates that engaged too much in
irrelevant discussions.’
NFC-Learner 5: ‘I am not very familiar (close) with the classmates, and the (learning)
interactions and bonding between us were not strong’.
In summary, there was a significant difference in learner achievement between GTC
and NFC (p = 0.004). The learners in GTC scored significantly higher than those in NFC for
learner achievement (U = 112.00, z = −2.875, p = 0.004), with effect size r = 0.410. Regarding
learner engagement, the results of class observation reports and learner surveys indicated
that the learners in GFC had the highest level of participation and perceived learning, and
performed all five components of SLE.

5. Discussion
In this study, we compared the influence of flipped classrooms and gamification on
learner achievement and engagement for SLE in the context of postgraduate business edu-
cation during the COVID-19 pandemic. The findings are discussed in the following three
subsections. First is the role of flipped classrooms and gamification during the COVID-19
pandemic. Second, teacher-dependency and learning interactions. Lastly, lessons learned
for the further improvement of flipped classrooms and gamification pedagogical design for
SLE in HEIs.

5.1. Role of Flipped Classroom and Gamification during COVID-19 Pandemic

Flipped classrooms were welcomed by learners [106] and helped to maintain the sus-
tainability of learning programmes in higher education during COVID-19 lockdowns [24].
However, learner motivation declines in the absence of learning interactions [48,107]. Gam-
ification promotes learning interactions and social networking [108]. The qualitative results
of class observation reports from the teaching assistants on learner engagement concur
with those of Lo and Hew [60] that learners in gamified classrooms (GFC and GTC) exhibit
a stronger sense of engagement, as evidenced by their level of participation during in-class
learning activities. In the context of adult postgraduate business education, the quantitative
results of the post-class assignment in our study showed that GTC offers advantages for
learner achievement and GFC for learner engagement, respectively. GFC improved learn-
ers’ perceived learning and engagement, but there was no significant impact on learner
achievement. Several other studies have found similar results, suggesting the need for
further investigation [109–111]. Our results confirm the observation of Bredow et al. [81]
that the benefit of a simple flipped classroom (i.e., NFC) may not be apparent. According
to McLean et al. [110], one reason for this might be that flipped classrooms challenge the
learners’ perception of the teacher’s role as a knowledge provider to playing the combined
role of knowledge application and synthesis facilitators.

5.2. Teacher-Dependency and Learning Interactions

A prior study found that, despite the autonomy that the flipped classroom pedagogy
affords to learners, in-class interactive learning was still preferred over pre-class self-
regulated study [39]. Peer-to-teacher interaction and learner preference for teachers to play
Sustainability 2022, 14, 5428 17 of 23

the role of authority for knowledge delivery are dominant themes in the Asian context [112].
In flipped classrooms, teacher roles change from being like those of sages to those of
facilitators, thereby conflicting with existing expectations [113]. Like previously published
findings [114], our results found better learning achievement in a traditional classroom
approach supplemented by gamification (i.e., GTC), as teacher recognition and feedback
in the form of game-design elements were valued more than responses from peers [115].
Such teacher dependency was also observed in previous studies [116–118]. Teachers should
introduce the importance and value of lesson content at the beginning of each lesson [119],
offer a brief review of pre-class materials [120], and facilitate peer-to-teacher and peer-to-
peer interactions [25].
In-class learning interactions played an important role in flipped classrooms [121], and
there is a need to explore this in further detail [60,122]. Learning interactions can be either
peer-to-peer or peer-to-teacher interactions. Our interviews with the teaching assistants
indicated that the learners were used to a teacher-dependent style of learning, which may
impede peer-to-peer learning. Learners needed the teacher to motivate the discussion when
the learners’ participation waned. The following is a transcript from an interview with our
experienced teaching assistant (JG):
‘Although the flipped classroom and gamified classroom may be the trends in future
education, our learners are relatively older adults, and the education model that they
experienced during their school ages was traditional. Many people are still inclined
towards the traditional methods of education. We have to use different teaching modes
according to the learners and their preferences and rely on teachers to facilitate changes’.
Therefore, to facilitate SLE in flipped classrooms, the cultural background and learning
styles of the learners, especially the type of schooling that the learners experienced, must be
considered. This is imperative if more instructional content is going to be imparted using
the flipped classroom and gamification approaches during the COVID-19 pandemic [24].

5.3. Lessons Learned

The flipped classroom and gamification designs adopted in this study were theoret-
ically grounded in SDT [56]. The three pedagogical interventions (GFC, NFC, and GTC)
provided valuable insight into the practical application of the theories in the context of
adult business education during the COVID-19 pandemic. Based on our findings, we
provide three observations and recommendations for the design of flipped classrooms and
gamification for SLE.
First, gamification plays an important role in motivating adult learners for learning
participation and engagement, which in turn promotes SLE [123]. From the perspective
of SDT, timely and evaluative feedback from teachers is vital to promoting learners’ sense
of engagement [55]. Second, teachers should facilitate learning interactions at the earliest
stage of flipped classrooms, such as at the beginning of the pre-class learning stage. Similar
inferences have also been made in other studies [112,124]. Knowledge learned from pre-
class materials must be recollected to set as the foundation of more advanced in-class
learning [125]. This is critical, especially in a highly teacher-dependent learning culture.
Teachers’ early participation and facilitation in both pre-class and in-class lessons would
trigger learners’ curiosity and interest and stimulate initial discussions, leading to more
learning participation [83]. Last, although we used the same game-design elements in both
GFC and GTC, learner achievement in the flipped GFC was lower than that of the traditional
GTC. Our results resonate with the findings of Jensen et al. [124] that the adoption of
flipped classrooms to create more time for in-class learning may not be effective even
with gamification. Teacher-dependency and learning culture must be considered. Adult
learners in our study benefitted more from teacher-led lectures (i.e., from the traditional
GTC) than the two flipped classroom approaches (GFC and NFC). This mirrors the study
of Magana et al. [126].
Nonetheless, the flipped classroom is an effective pedagogical approach to ensuring
the sustainability of education programmes in HEIs during COVID-19 lockdowns [24].
Sustainability 2022, 14, 5428 18 of 23

Pre-class self-study videos and materials ameliorate the anxiety of learners caused by
academic uncertainty and interruptions in learning [127]. For example, one learner from
GTC remarked, ‘if the school can provide pre-class videos, I can prepare in advance, integrate
my previous knowledge and develop clarity on the areas in which I need to learn more (in the
coming in-class lessons)’ (GTC-Learner 3). Therefore, a flipped classroom with gamification
is a valuable pedagogical approach for the sustainability of higher business education,
especially under the dynamic COVID-Zero strategy in China [128]. However, teachers
cannot rely solely on flipped classrooms or gamification pedagogies. Indeed, the early
presence of teachers along the entire learning journey, including in the pre-class stage, to
facilitate peer-to-teacher and peer-to-peer facilitations is deemed necessary.

6. Conclusions and Recommendations for Future Study

This study compared learner achievement and engagement across three pedagogical
approaches (i.e., GFC, NFC, and GTC). The learners in GTC performed the best in terms
of learner achievement, whereas those in GFC reported the highest perceived learning
and engagement for SLE. Qualitative findings from surveys and class observation reports
reflected that gamification made the lessons more interesting and exciting, and promoted
participation in in-class learning activities. Learners in GTC and GFC showed higher levels
of participation and performed more key components of SLE, which were the reasons for
higher learner achievement and engagement in comparison to NFC.
We provided three recommendations based on the study. First, flipped classrooms
can be leveraged to maintain the continuity and sustainability of education programmes,
especially under the uncertainty of lockdowns during the COVID-19 pandemic. Second,
gamification plays a key role in improving learner achievement, and should therefore be
integrated into flipped classrooms with the consideration of the learning culture and styles
of the learners. Third, teachers must facilitate and be involved in all stages of flipped
classrooms, especially in highly teacher-dependent learning cultures.
Notwithstanding the aforementioned contributions, there are certain limitations in
the study. First, the sample sizes of the three classrooms were small, which reduces the
generalisability of the results. Second, the study was conducted in a postgraduate adult
business education setting and may therefore not be equally applicable to other educational
contexts. Third, the duration of the study was limited to 10 weeks owing to restrictions
caused by the COVID-19 pandemic. Therefore, the findings must not be interpreted as
establishing that any one of the pedagogical approaches is better than the others. Further
studies are necessary to explore the differences between flipped classrooms with and
without gamification in larger samples and across a longer duration.

Author Contributions: Formal analysis, investigation, writing—original draft preparation, writing

—review and editing, project administration, and visualization, L.-K.N.; conceptualization, method-
ology, validation, data curation, and supervision, C.-K.L. All authors have read and agreed to the
published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement: The study was conducted according to the guidelines of the
Declaration of Helsinki and approved by the Human Research Ethics Committee of the Education
University of Hong Kong (protocol code: 2020-2021-0436; date of approval: 28 October 2021).
Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.
Data Availability Statement: The data samples and detailed coding procedures can be accessed by
contacting the author.
Acknowledgments: Thank you for the valuable and constructive feedback from the editor and
professional reviewers. This study is in partial fulfilment of the Research Project/Thesis Requirements
of the Education University of Hong Kong.
Conflicts of Interest: The authors declare no conflict of interest.
Sustainability 2022, 14, 5428 19 of 23

References
1. Chen, F.H.; Tsai, C.C.; Chung, P.Y.; Lo, W.S. Sustainability Learning in Education for Sustainable Development for 2030:
An Observational Study Regarding Environmental Psychology and Responsible Behavior through Rural Community Travel.
Sustainability 2022, 14, 2779. [CrossRef]
2. Leicht, A.; Heiss, J.; Byun, W.J. Issues and Trends in Education for Sustainable Development; UNESCO Publishing: Paris, France, 2018;
Volume 5.
3. Martin, M.; Godonoga, A. SDG 4-Policies for Flexible Learning Pathways in Higher Education: Taking Stock of Good Practices
Internationally; IIEP-UNESCO: Paris, France, 2020.
4. Ben-Eliyahu, A. Sustainable learning in education. Sustainability 2021, 13, 4250. [CrossRef]
5. Pilotti, M.A.; Al Ghazo, R. Sustainable education starts in the classroom. Sustainability 2020, 12, 9573. [CrossRef]
6. Rovio-Johansson, A. Students’ Knowledge Progression: Sustainable Learning in Higher Education. Int. J. Learn. High. 2016, 28,
427–439.
7. Yassin, A.A.; Razak, N.A.; Qasem, Y.A.; Mohammed, M.A.S. Intercultural learning challenges affecting international students’
sustainable learning in Malaysian higher education institutions. Sustainability 2020, 12, 7490. [CrossRef]
8. Hays, J.; Reinders, H. Sustainable learning and education: A curriculum for the future. Int. Rev. Educ. 2020, 66, 29–52. [CrossRef]
9. Pfeiffer, A.; Bezzina, S.; Denk, N.; Kriglstein, S.; Wimmer, S.; Thielen, C. Life-World Oriented Education in Times of the COVID-19
Lockdown. Twitch.tv as Pedagogical Tool. ICERI2020 Proc. 2020, 9, 10.
10. Yu, K.; Wu, L.; Zhou, L. Research on the Mixed Education Mode for the Safety Engineering Major during the Coronavirus
(COVID-19) Epidemic. Int. J. Environ. Res. Public Health 2022, 19, 1967. [CrossRef]
11. Pérez-Jorge, D.; Rodríguez-Jiménez, M.C.; Ariño-Mateo, E.; Barragán-Medero, F. The effect of COVID-19 in university tutoring
models. Sustainability 2020, 12, 8631. [CrossRef]
12. Murphy, M.P.A. COVID-19 and emergency eLearning: Consequences of the securitization of higher education for post-pandemic
pedagogy. Contemp. Secur. Policy 2020, 41, 492–505. [CrossRef]
13. Le, K. Pre-Recorded Lectures, Live Online Lectures, and Student Academic Achievement. Sustainability 2022, 14, 2910. [CrossRef]
14. Yang, R. China’s higher education during the COVID-19 pandemic: Some preliminary observations. High. Educ. Res. Dev. 2020,
39, 1317–1321. [CrossRef]
15. Ardhaoui, K.; Lemos, M.S.; Silva, S. Effects of new teaching approaches on motivation and achievement in higher education
applied chemistry courses: A case study in Tunisia. Educ. Chem. Eng. 2021, 36, 160–170. [CrossRef]
16. Bowden, J.L.H.; Tickle, L.; Naumann, K. The four pillars of tertiary student engagement and success: A holistic measurement
approach. Stud. High. Educ. 2021, 46, 1207–1224. [CrossRef]
17. Jang, H.Y.; Kim, H.J. A meta-analysis of the cognitive, affective, and interpersonal outcomes of flipped classrooms in higher
education. Educ. Sci. 2020, 10, 115. [CrossRef]
18. Huang, B.; Hew, K.F.; Lo, C.K. Investigating the effects of gamification-enhanced flipped learning on undergraduate students’
behavioral and cognitive engagement. Interact. Learn. Environ. 2019, 27, 1106–1126. [CrossRef]
19. Sailer, M.; Sailer, M. Gamification of in-class activities in flipped classroom lectures. Br. J. Educ. Technol. 2021, 52, 75–90. [CrossRef]
20. Deci, E.L.; Ryan, R.M. Self-determination theory. In Handbook of Theories of Social Psychology; Van Lange, P.A.M., Kruglanski, A.W.,
Higgins, E.T., Eds.; Sage Publications Ltd.: New York, NY, USA, 2012; pp. 416–436. [CrossRef]
21. Reeve, J. Self-determination theory applied to educational settings. In Handbook of Self-Determination Research: Theoretical and
Applied Issues; Deci, E.L., Ryan, R.M., Eds.; University of Rochester Press: Rochester, NY, USA, 2002; Volume 2, pp. 183–204.
22. Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L. From game design elements to gamefulness: Defining “gamification”. In
Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere,
Finland, 28–30 September 2011.
23. Dicheva, D.; Dichev, C.; Agre, G.; Angelova, G. Gamification in education: A systematic mapping study. J. Educ. Technol. Soc.
2015, 18, 75–88.
24. Collado-Valero, J.; Rodríguez-Infante, G.; Romero-González, M.; Gamboa-Ternero, S.; Navarro-Soria, I.; Lavigne-Cerván, R.
Flipped classroom: Active methodology for sustainable learning in higher education during social distancing due to COVID-19.
Sustainability 2021, 13, 5336. [CrossRef]
25. Akram, A.; Sattar, K.; Ahmad, T.; Abdulghani, H.M.; John, J.; Yusoff, M.S.B. Flipping the “Learning Skills” Course during
COVID-19: A Mixed-Modality Study. Educ. Med. J. 2021, 13, 31–41. [CrossRef]
26. French, H.; Arias-Shah, A.; Gisondo, C.; Gray, M.M. Perspectives: The flipped classroom in graduate medical education.
NeoReviews 2020, 21, e150–e156. [CrossRef] [PubMed]
27. Peters, M.A.; Wang, H.; Ogunniran, M.O.; Huang, Y.; Green, B.; Chunga, J.O.; Quianoo, E.A.; Ren, Z.; Hollings, S.; Hayes, S.; et al.
China’s internationalized higher education during COVID-19: Collective student autoethnography. Postdigit. Sci. Educ. 2020, 2,
968–988. [CrossRef]
28. Wu, Z. How a Top Chinese University Is Responding to Coronavirus. 2020. Available online: https://www.weforum.org/
agenda/2020/03/coronavirus-china-the-challenges-of-online-learning-for-universities/ (accessed on 28 March 2022).
29. Li, S.; Liao, X.; Burdick, W.; Tong, K. The Effectiveness of Flipped Classroom in Health Professions Education in China:
A Systematic Review. J. Med. Educ. Curric. 2020, 7, 2382120520962838. [CrossRef] [PubMed]
Sustainability 2022, 14, 5428 20 of 23

30. Filho, W.L.; Raath, S.; Lazzarini, B.; Vargas, V.R.; de Souza, L.; Anholon, R.; Quelhas, O.L.G.; Haddad, R.; Klavins, M.; Orlovic, V.L.
The role of transformation in learning and education for sustainability. J. Clean. Prod 2018, 199, 286–295. [CrossRef]
31. Willats, J.; Erlandsson, L.; Molthan-Hill, P.; Dharmasasmita, A.; Simmons, E. A university wide approach to embedding the
sustainable development goals in the curriculum—A case study from the Nottingham Trent University’s Green Academy. In
Implementing Sustainability in the Curriculum of Universities; Springer: Cham, Switzerland, 2018; pp. 63–78.
32. Hays, J. Chaos to Capability: Educating Professionals for the 21st Century; Unitec ePress Monograph Series (1); Unitec Institute of
Technology: Auckland, New Zealand, 2015. Available online: https://hdl.handle.net/10652/3053 (accessed on 28 March 2022).
33. Azevedo, R. Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical
issues. Educ. Psychol. 2015, 50, 84–94. [CrossRef]
34. Ben-Eliyahu, A.; Linnenbrink-Garcia, L. Integrating the regulation of affect, behavior, and cognition into self-regulated learning
paradigms among secondary and post-secondary students. Metacogn. Learn. 2015, 10, 15–42. [CrossRef]
35. Hettiarachchi, S.; Damayanthi, B.W.R.; Heenkenda, S.; Dissanayake, D.M.S.L.B.; Ranagalage, M.; Ananda, L. Student Satisfaction
with Online Learning during the COVID-19 Pandemic: A Study at State Universities in Sri Lanka. Sustainability 2021, 13, 11749.
[CrossRef]
36. Li, S.; Wang, Y.; Xue, J.; Zhao, N.; Zhu, T. The impact of COVID-10 epidemic declaration on psychological consequences: A study
on active weibo users. Int. J. Environ. Res. Public Health 2020, 17, 2032. [CrossRef]
37. Šakan, D.; Žuljević, D.; Rokvić, N. The role of basic psychological needs in well-being during the COVID-19 outbreak: A self-
determination theory perspective. Front. Public Health 2020, 8, 713. [CrossRef]
38. Carrillo, C.; Flores, M.A. COVID-19 and teacher education: A literature review of online teaching and learning practices. Eur. J.
Teach. Educ. 2020, 43, 466–487. [CrossRef]
39. Bishop, J.; Verleger, M.A. The flipped classroom: A survey of the research. In Proceedings of the 2013 ASEE Annual Conference &
Exposition, Atlanta, GA, USA, 23–26 June 2013; pp. 23–1200.
40. Abeysekera, L.; Dawson, P. Motivation and cognitive load in the flipped classroom: Definition, rationale and a call for research.
High. Educ. Res. Dev. 2015, 34, 1–14. [CrossRef]
41. Mahmud, S.N.D.; Husnin, H.; Tuan Soh, T.M. Teaching presence in online gamified education for sustainability learning.
Sustainability 2020, 12, 3801. [CrossRef]
42. Galindo-Dominguez, H. Flipped Classroom in the Educational System. J. Educ. Technol. Soc. 2021, 24, 44–60.
43. Hsia, L.H.; Lin, Y.N.; Hwang, G.J. A creative problem solving-based flipped learning strategy for promoting students’ performing
creativity, skills and tendencies of creative thinking and collaboration. Br. J. Educ. Technol. 2021, 52, 1771–1787. [CrossRef]
44. Cao, W.; Fang, Z.; Hou, G.; Han, M.; Xu, X.; Dong, J.; Zheng, J. The psychological impact of the COVID-19 epidemic on college
students in China. Psychiatr. Res. 2020, 287, 112934. [CrossRef] [PubMed]
45. Hernández, D.J.; Ortiz, J.J.G.; Abellán, M.T. Metodologías Activas en la Universidad y su relación con los Enfoques de Enseñanza.
Profr. Rev. De Currículum Y Form. Del Profr. 2020, 24, 76–94. [CrossRef]
46. Xiong, J.; Yan, J.; Fu, K.; Wang, K.; He, Y. Innovation in an authoritarian society: China during the pandemic crisis. J. Bus. Strategy
2021, 43, 79–86. [CrossRef]
47. Zainuddin, Z. Students’ learning performance and perceived motivation in gamified flipped-class instruction. Comput. Educ.
2018, 126, 75–88. [CrossRef]
48. Ahshan, R. A framework of implementing strategies for active student engagement in remote/online teaching and learning
during the COVID-19 pandemic. Educ. Sci. 2021, 11, 483. [CrossRef]
49. Heo, H.; Bonk, C.J.; Doo, M.Y. Enhancing learning engagement during COVID-19 pandemic: Self-efficacy in time management,
technology use, and online learning environments. J. Comput. Assist. Learn. 2021, 37, 1640–1652. [CrossRef]
50. Kuhfeld, M.; Soland, J.; Tarasawa, B.; Johnson, A.; Ruzek, E.; Liu, J. Projecting the potential impact of COVID-19 school closures
on academic achievement. Educ. Res. 2020, 49, 549–565. [CrossRef]
51. Luria, E.; Shalom, M.; Levy, D.A. Cognitive Neuroscience Perspectives on Motivation and Learning: Revisiting Self-Determination
Theory. Mind Brain Educ. 2021, 15, 5–17. [CrossRef]
52. Ryan, R.M. Psychological needs and the facilitation of integrative processes. J. Pers. 1995, 63, 397–427. [CrossRef] [PubMed]
53. Silvia, P.J. Interest—The curious emotion. Curr. Dir. Psychol. Sci. 2008, 17, 57–60. [CrossRef]
54. Ashari, H.; Abbas, I.; Abdul-Talib, A.N.; Zamani, S.N.M. Entrepreneurship and Sustainable Development Goals: A Multigroup
Analysis of the Moderating Effects of Entrepreneurship Education on Entrepreneurial Intention. Sustainability 2021, 14, 431.
[CrossRef]
55. Niemiec, C.P.; Ryan, R.M. Autonomy, competence, and relatedness in the classroom: Applying self-determination theory to
educational practice. Theory Res. Educ. 2009, 7, 133–144. [CrossRef]
56. Ryan, R.M.; Deci, E.L. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.
Am. Psychol. 2000, 55, 68–78. [CrossRef]
57. Connell, J.P.; Wellborn, J.G. Competence, autonomy, and relatedness. In A Motivational Analysis of Self-System Processes;
Gunnar, M.R., Sroufe, L.A., Eds.; Lawrence Erlbaum Associates, Inc.: Hillsdale, MI, USA, 1991; pp. 43–77.
58. Newmann, F.M. Student Engagement and Achievement in American Secondary Schools; Teachers College Press: New York, NY,
USA, 1992.
Sustainability 2022, 14, 5428 21 of 23

59. Sergis, S.; Sampson, D.G.; Pelliccione, L. Investigating the impact of Flipped Classroom on students’ learning experiences:
A Self-Determination Theory approach. Comput. Hum. Behav. 2018, 78, 368–378. [CrossRef]
60. Lo, C.K.; Hew, K.F. A comparison of flipped learning with gamification, traditional learning, and online independent study: The
effects on students’ mathematics achievement and cognitive engagement. Interact. Learn. Environ. 2020, 28, 464–481. [CrossRef]
61. Pelikan, E.R.; Korlat, S.; Reiter, J.; Holzer, J.; Mayerhofer, M.; Schober, B.; Spiel, C.; Hamzallari, O.; Uka, A.; Chen, J.; et al. Distance
learning in higher education during COVID-19: The role of basic psychological needs and intrinsic motivation for persistence
and procrastination—A multi-country study. PLoS ONE 2021, 16, e0257346. [CrossRef]
62. Chiu, T.K. Applying the self-determination theory (SDT) to explain student engagement in online learning during the COVID-19
pandemic. J. Res. Technol. Educ. 2021, 54 (Suppl. 1), S14–S30. [CrossRef]
63. Bayoumy, H.M.M.; Alsayed, S. Investigating relationship of perceived learning engagement, motivation, and academic perfor-
mance among nursing students: A multisite study. Adv. Med. Educ. Pract. 2021, 12, 351. [CrossRef] [PubMed]
64. Panigrahi, R.; Srivastava, P.R.; Panigrahi, P.K. Effectiveness of e-learning: The mediating role of student engagement on perceived
learning effectiveness. Inf. Technol. People 2020, 34, 1840–1862. [CrossRef]
65. Lozano-Jiménez, J.E.; Huéscar, E.; Moreno-Murcia, J.A. Effects of an autonomy support intervention on the involvement of higher
education students. Sustainability 2021, 13, 5006. [CrossRef]
66. Orazbayeva, B.; van der Sijde, P.; Baaken, T. Autonomy, competence and relatedness–the facilitators of academic engagement in
education-driven university-business cooperation. Stud. High. Educ. 2021, 46, 1406–1420. [CrossRef]
67. Deterding, S.; Sicart, M.; Nacke, L.; O’Hara, K.; Dixon, D. Gamification. Using game-design elements in non-gaming contexts. In
CHI’11 Extended Abstracts on Human Factors in Computing Systems; Association for Computing Machinery: New York, NY, USA,
2011; pp. 2425–2428.
68. Park, S.; Kim, S. Is sustainable online learning possible with gamification?—The effect of gamified online learning on student
learning. Sustainability 2021, 13, 4267. [CrossRef]
69. Ekici, M. A systematic review of the use of gamification in flipped learning. Educ. Inf. Technol. 2021, 26, 3327–3346. [CrossRef]
70. Hammill, J.; Nguyen, T.; Henderson, F. Encouraging the flip with a gamified process. Int. J. Educ. Res. 2021, 2, 100085. [CrossRef]
71. Langendahl, P.A.; Cook, M.; Mark-Herbert, C. Gamification in Higher Education: Toward a Pedagogy to Engage and Motivate;
Department of Economics, Swedish University of Agricultural Sciences: Uppsala, Sweden, 2016.
72. Deci, E.L.; Ryan, R.M. Optimizing students’ motivation in the era of testing and pressure: A self-determination theory perspective.
In Building Autonomous Learners; Springer: Singapore, 2016; pp. 9–29.
73. Rosenbaum, M.S.; Russell-Bennett, R.; Contreras-Ramírez, G. Business education in profound disruption. J. Serv. Mark. 2021, 35,
553–558. [CrossRef]
74. Chang, D.; Hwang, G.J.; Chang, S.C.; Wang, S.Y. Promoting students’ cross-disciplinary performance and higher order thinking:
A peer assessment-facilitated STEM approach in a mathematics course. Educ. Technol. Res. Dev. 2021, 69, 3281–3306. [CrossRef]
75. Kressler, B.; Kressler, J. Diverse student perceptions of active learning in a large enrollment STEM course. J. Scholarsh. Teach. Learn.
2020, 20, 40–64. [CrossRef]
76. Jusas, V.; Barisas, D.; Jančiukas, M. Game Elements towards More Sustainable Learning in Object-Oriented Programming Course.
Sustainability 2022, 14, 2325. [CrossRef]
77. Sailer, M.; Hense, J.U.; Mayr, S.K.; Mandl, H. How gamification motivates: An experimental study of the effects of specific game
design elements on psychological need satisfaction. Comput. Hum. Behav. 2017, 69, 371–380. [CrossRef]
78. Rincon-Flores, E.G.; Santos-Guevara, B.N. Gamification during COVID-19: Promoting active learning and motivation in higher
education. Australas. J. Educ. Technol. 2021, 37, 43–60. [CrossRef]
79. Santos-Villalba, M.J.; Olivencia, J.J.L.; Navas-Parejo, M.R.; Benítez-Márquez, M.D. Higher Education Students’ Assessments
towards Gamification and Sustainability: A Case Study. Sustainability 2020, 12, 8513. [CrossRef]
80. Ahmed, H.D.; Asiksoy, G. The Effects of Gamified Flipped Learning Method on Student’s Innovation Skills, Self-Efficacy towards
Virtual Physics Lab Course and Perceptions. Sustainability 2021, 13, 10163. [CrossRef]
81. Bredow, C.A.; Roehling, P.V.; Knorp, A.J.; Sweet, A.M. To flip or not to flip? A meta-analysis of the efficacy of flipped learning in
higher education. Rev. Educ. Res. 2021, 91, 878–918. [CrossRef]
82. Creswell, J.W.; Klassen, A.C.; Clark, V.L.P.; Smith, K.C. Best practices for mixed methods research in the health sciences. Bethesda
Natl. Inst. Health 2013, 12, 541–545.
83. Ho, C.M.; Yeh, C.C.; Wang, J.Y.; Hu, R.H.; Lee, P.H. Pre-class online video learning and class style expectation: Patterns, association,
and precision medical education. Ann. Med. 2021, 53, 1390–1401. [CrossRef]
84. Furdu, I.; Tomozei, C.; Kose, U. Pros and Cons Gamification and Gaming in Classroom.: Discovery Service for University of
South Carolina. BRAIN Broad Res. Artif. Intell. Neurosci. 2017, 8, 56–62.
85. Mazarakis, A. Using Gamification for Technology Enhanced Learning: The Case of Feedback Mechanisms. Bull. Tech. Comm.
Learn. Technol. 2015, 17, 6–9.
86. Hamari, J. Do badges increase user activity? A field experiment on the effects of gamification. Comput. Hum. Behav. 2017, 71,
469–478. [CrossRef]
87. Burguillo, J.C. Using game theory and competition-based learning to stimulate student motivation and performance. Comput.
Educ. 2010, 55, 566–575. [CrossRef]
88. Okura, M.; Carfì, D. Coopetition and game theory. J. Appl. Econ. Sci. 2014, 9, 29.
Sustainability 2022, 14, 5428 22 of 23

89. Seaborn, K.; Fels, D.I. Gamification in theory and action: A survey. Int. J. Hum. Comput. Stud. 2015, 74, 14–31. [CrossRef]
90. Skinner, E.; Furrer, C.; Marchand, G.; Kindermann, T. Engagement and disaffection in the classroom: Part of a larger motivational
dynamic? J. Educ. Psychol. 2008, 100, 765. [CrossRef]
91. Rotgans, J.I.; Schmidt, H.G. Cognitive engagement in the problem-based learning classroom. Adv. Health Sci. Educ. Theory Pract.
2011, 16, 465–479. [CrossRef]
92. Al-Zahrani, A.M. From passive to active: The impact of the flipped classroom through social learning platforms on higher
education students’ creative thinking. Br. J. Educ. Technol. 2015, 46, 1133–1148. [CrossRef]
93. Chi, M.T.; Wylie, R. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educ. Psychol. 2014, 49,
219–243. [CrossRef]
94. Hew, K.F.; Huang, B.; Chu, K.W.S.; Chiu, D.K. Engaging Asian students through game mechanics: Findings from two experiment
studies. Comput. Educ. 2016, 92, 221–236. [CrossRef]
95. Merriam, S.B.; Grenier, R.S. (Eds.) Qualitative Research in Practice: Examples for Discussion and Analysis; John Wiley & Sons: San
Francisco, CA, USA, 2019.
96. Fredricks, J.A.; Blumenfeld, P.C.; Paris, A.H. School engagement: Potential of the concept, state of the evidence. Rev. Educ. Res.
2004, 74, 59–109. [CrossRef]
97. McCallum, S.; Schultz, J.; Sellke, K.; Spartz, J. An examination of the flipped classroom approach on college student academic
involvement. Int. J. Learn. High. 2015, 27, 42–55.
98. Field, A. Discovering Statistics Using SPSS, 3rd ed.; Sage Publications Ltd.: London, UK, 2009.
99. Weisstein, E.W. Bonferroni Correction. 2004. Available online: https://mathworld.wolfram.com/BonferroniCorrection.html
(accessed on 28 March 2022).
100. Tomczak, M.; Tomczak, E. The need to report effect size estimates revisited. An overview of some recommended measures of
effect size. Trends Sport Sci. 2014, 1, 19–25.
101. Johnson, R.B. Examining the validity structure of qualitative research. Education 1997, 118, 282–292.
102. Glesne, C. Becoming Qualitative Researchers: An Introduction; Pearson: Upper Saddle River, NJ, USA, 2016.
103. Saldana, J. Fundamentals of Qualitative Research; Oxford University Press: Oxford, UK, 2011.
104. Maxwell, J.A. Qualitative Research Design: An Interactive Approach; Sage Publications: Newbury Park, CA, USA, 2012.
105. Flick, U. Designing Qualitative Research; Sage Publications: Newbury Park, CA, USA, 2018.
106. Li, B.Z.; Cao, N.W.; Ren, C.X.; Chu, X.J.; Zhou, H.Y.; Guo, B. Flipped classroom improves nursing students’ theoretical learning in
China: A meta-analysis. PLoS ONE 2020, 15, e0237926. [CrossRef] [PubMed]
107. Kormos, J.; Csizer, K. The interaction of motivation, self-regulatory strategies, and autonomous learning behavior in different
learner groups. Tesol Q. 2014, 48, 275–299. [CrossRef]
108. Hajarian, M.; Diaz, P. Effective Gamification: A Guideline for Gamification Workshop of WEEF-GEDC 2021 Madrid Conference.
In Proceedings of the 2021 World Engineering Education Forum/Global Engineering Deans Council (WEEF/GEDC), Madrid,
Spain, 15–18 November 2021; pp. 506–510.
109. Adams, A.E.; Garcia, J.; Traustadóttir, T. A quasi experiment to determine the effectiveness of a “partially flipped” versus “fully
flipped” undergraduate class in genetics and evolution. CBE Life Sci. Educ. 2016, 15, 11. [CrossRef]
110. McLean, S.; Attardi, S.M.; Faden, L.; Goldszmidt, M. Flipped classrooms and student learning: Not just surface gains. Adv. Physiol.
Educ. 2016, 40, 47–55. [CrossRef]
111. Smallhorn, M. The flipped classroom: A learning model to increase student engagement not academic achievement. Stud. Success
2017, 8, 43–53. [CrossRef]
112. Loh, C.Y.R.; Teo, T.C. Understanding Asian students learning styles, cultural influence and learning strategies. J. Educ. Soc. Pol.
2017, 7, 194–210.
113. Van Sickle, J.R. Discrepancies between student perception and achievement of learning outcomes in a flipped classroom.
J. Scholarsh. Teach. Learn. 2016, 16, 29–38. [CrossRef]
114. Zheng, L.; Bhagat, K.K.; Zhen, Y.; Zhang, X. The effectiveness of the flipped classroom on students’ learning achievement and
learning motivation. J. Educ. Technol. Soc. 2020, 23, 1–15.
115. Zhao, H. Investigating learners’ use and understanding of peer and teacher feedback on writing: A comparative study in a
Chinese English writing classroom. Assess. Writ. 2010, 15, 3–17. [CrossRef]
116. Hew, K.F. Student perceptions of peer versus instructor facilitation of asynchronous online discussions: Further findings from
three cases. Instr. Sci. 2015, 43, 1938. [CrossRef]
117. Roorda, D.L.; Zee, M.; Koomen, H.M. Don’t forget student-teacher dependency! A Meta-analysis on associations with students’
school adjustment and the moderating role of student and teacher characteristics. Attach Hum. Dev. 2021, 23, 490–503. [CrossRef]
118. Vaziri, S.; Vaziri, B.; Novoa, L.J.; Torabi, E. Academic Motivation in Introductory Business Analytics Courses: A Bayesian
Approach. INFORMS Trans. Educ. 2022, 22, 121–129. [CrossRef]
119. Long, T.; Logan, J.; Waugh, M. Students’ perceptions of the value of using videos as a pre-class learning experience in the flipped
classroom. TechTrends 2016, 60, 245–252. [CrossRef]
120. Lo, C.K.; Hew, K.F. The impact of flipped classrooms on student achievement in engineering education: A meta-analysis of
10 years of research. J. Eng. Educ. 2019, 108, 523–546. [CrossRef]
Sustainability 2022, 14, 5428 23 of 23

121. Lestari, I.W. Flipped classroom in Indonesian higher education: A mixed-method study on students’ attitudes and experiences.
Stud. Engl. Lang. Educ. 2021, 8, 243–257. [CrossRef]
122. Maciejewski, W. Flipping the calculus classroom: An evaluative study. Teach. Math. Appl. 2016, 35, 187–201. [CrossRef]
123. Karra, S.; Karampa, V.; Paraskeva, F. Gamification design framework based on self determination theory for adult motivation. In
Proceedings of International Workshop on Learning Technology for Education in Cloud; Springer: Cham, Switzerland, 2019; pp. 67–78.
124. Jensen, J.L.; Holt, E.A.; Sowards, J.B.; Ogden, T.H.; West, R.E. Investigating strategies for pre-class content learning in a flipped
classroom. J. Sci. Educ. Technol. 2018, 27, 523–535. [CrossRef]
125. Yorganci, S. Implementing flipped learning approach based on ‘first principles of instruction’ in mathematics courses. J. Comput.
Assist. Learn. 2020, 36, 763–779. [CrossRef]
126. Magana, A.J.; Vieira, C.; Boutin, M. Characterizing engineering learners’ preferences for active and passive learning methods.
IEEE Trans. Educ. 2017, 61, 46–54. [CrossRef]
127. Arribathi, A.H.; Suwarto; Rosyad, A.M.; Budiarto, M.; Supriyanti, D.; Mulyati. An analysis of student learning anxiety during the
COVID-19 Pandemic: A Study in Higher Education. J. Contin. High. Educ. 2021, 69, 192–205. [CrossRef]
128. Liu, J.; Liu, M.; Liang, W. The Dynamic COVID-Zero Strategy in China. China CDC Wkly. 2022, 4, 74–75. [CrossRef]