Simulation Games PDF

Computers & Education 149 (2020) 103813
Contents lists available at ScienceDirect
Computers & Education

journal homepage: http://www.elsevier.com/locate/compedu
The Co-Creative approach to digital simulation games in social

science education
Anna Sanina a, *, Evgeniia Kutergina b, Aleksey Balashov c
a
National Research University Higher School of Economics, St. Petersburg Campus, Department of Public Administration, 16 Ulitsa Soyuza
Pechatnikov, St.Petersburg, 190008, Russia
b
ITMO University, Department of Strategic Planning and Development, Mailing Address: 9 Lomonosova Street, St. Petersburg, 191002, Russia
c
North-Western Institute of Management of the Russian Federation Presidential Academy of National Economy and Public Administration
(RANEPA), Department of Public Administration, 57/43 Sredniy prospect, Vasilyevsky Island, St.Petersburg, 199178, Russia
A R T I C L E I N F O A B S T R A C T
Keywords: This paper focuses on the educational possibilities and potential of digital simulation games in
Co-creation in education higher education. It provides the detailed examination of the true experimental design of a co-
Collaborative learning creative gamified classroom that could be used in different academic subjects in social science
Digital simulation games
education. In this pedagogical experiment, we tested the effects of a co-creative gamification
Edutainment
classroom within a Public Sector Economics course attended by 253 first-year master’s students.
Educational outcomes
We used pre-test and post-test examinations, surveys, and interviews to evaluate and compare
effects on learning outcomes and course evaluations of different classroom modes (with and
without a co-creative approach and digital simulation games). This paper presents a quantitative
and qualitative analysis of the proposed experimental design, using treatment and control groups.
Our conclusions make a contribution to the discussion of the co-creative approach in education,
proving that its digital implementation can develop students’ generic and professional skills. We
also reveal a more conscious and motivated attitude toward the future profession of those stu
dents who participated in the process of creating the game.
Dear participants:
1. Introduction
Computer-based learning is one of the most established and powerful tools used widely in contemporary higher education. This
type of learning increases students’ motivation to acquire new knowledge and simplifies the evaluation process (Huang & Oh, 2018;
Woo, 2014). The development of IT infrastructure in universities (Kirkup & Kirkwood, 2005), the active learning approach (Prince,
2004), and collaborative and lifelong learning (Adams Becker, Cummins, Davis, Freeman, & Hall Giesinger, C. Ananthanarayanan,
2017) are significant trends that mark the new challenges of education development. Still, it is clear that any IT system becomes
educationally efficient only if it is enhanced by human participation (Gros, 2007; Molin, 2017). Students are not robots, and creative
and collaborative involvement in the process of computer-assisted learning can be used to multiply the efficiency of the educational
process.
Unlike 20th-century education, which concentrated on skills and knowledge, the education of the 21st century is aimed at
* Corresponding author.
E-mail addresses: asanina@hse.ru, anna.g.sanina@gmail.com (A. Sanina), eakutergina@corp.ifmo.ru (E. Kutergina), balashov-ai@ranepa.ru
(A. Balashov).
https://doi.org/10.1016/j.compedu.2020.103813
Received 18 July 2019; Received in revised form 13 January 2020; Accepted 20 January 2020
Available online 25 January 2020
0360-1315/© 2020 Elsevier Ltd. All rights reserved.
A. Sanina et al. Computers & Education 149 (2020) 103813
developing creativity, collaboration, and other “soft skills” as key qualities for student improvement (Burnard, 2006; Hamidi,
Wennberg, & Berglund, 2008). This challenges pedagogical instruments and approaches (Armbruster, Patel, Johnson, & Weiss, 2009;
Brush & Saye, 2000; Cornelius-White, 2007; Jones, 2007) and facilitates changes in enduring pedagogical tools. A diverse group of
particular applications of computer-based education—e.g., educational software, educational networks, online courses, computer
educational games, etc.—is being developed toward integration with educational instruments that promote collaboration and crea
tivity (Sawyer, 2004) and create value in the classroom (Díaz-M� endez & Gummesson, 2012; Ghinea, 2013). Moreover, soft pedagogical
instruments like creativity and collaboration become one of the cornerstones of educational efficiency (Burde, Kapit, Wahl, Guven, &
Skarpeteig, 2017; Greenhow, Robelia, & Hughes, 2009; Razzouk & Shute, 2012).
These trends define the importance of educational problems such as integration of computer-based education, collaboration, and
creative learning. How can the “human face” and “creative spirit” of IT-based technologies enhance the educational process? Are there
any new meanings of educational efficiency in the case of integration of computer-based education and soft pedagogical instruments?
What educational outcomes bring forth the integration of such methodologically different educational techniques in the pedagogical
process?
Previous studies show that digital simulations and games as well as co-creation are very important sources of professional expe
rience. However, none of the studies investigate the combination of digital and co-creation approaches in terms of learning outcomes
and students’ professional development and intention. Our study aims to fill this gap. It is based on a particular case of such integration
in social science education and discusses the issues of digital simulation-based learning accompanied by the co-creative approach. The
main purpose of this research is to investigate the educational effects of co-created digital simulation implementation and to compare
them with the implementation of digital games in the educational process without a co-creational approach.
The rest of the paper is organized as follows. Section 2 reviews contemporary approaches to digital simulation games and co-
creation as sources of professional experience in social science education. Section 3 describes the methods and procedures used in
our recent pedagogical experience of co-creation in the development of a particular form of computer-based education (i.e., digital
simulations) within a particular university course. Section 4 reports major results, both quantitative and qualitative. Some concluding
remarks and discussion in section 5 close the paper.
2. Literature review
Active usage of digital educational games and simulations in contemporary universities challenge longstanding educational par
adigms by establishing new educational environments. Unlike entertainment games developed mostly for fun and recreation,
educational games and simulators serve the goals of learning and behavior change (Connolly, Boyle E., MacArthur, Hainey, & Boyle J.,
2012). Thus, they conduct a framework of game-based learning (Prensky, 2007) to combine entertainment elements and ambitious
purposes and contribute to students’ exploration of practical aspects of social and business spheres, health, public policy, strategic
communication, etc.
Originally, games and simulations had different purposes and features. As discussed in the academic literature (Gredler, 2004;
Honey & Hilton, 2011; Van Horn, 2007), games suppose a spontaneous and informal process of playing and imply explicit goals and
rules, while simulations demand computational modeling of the reality built as a scenario for interactive exploration and investigation
of its certain parameters. However, the diffusion of digital technologies and affordability of computer-based education led to the
progressive convergence of games and simulations, as well as to the blending of the related concepts.
Previous studies highlight the lack of clear definitions and terminology (Klabbers, 2009; Vlachopoulos & Makri, 2017). Many
academic papers follow the convergence approach and analyze the effectiveness of simulations and games in the educational process
(Honey & Hilton, 2011), considering these educational tools as complementary or closely related (Loh, 2007; Narayanasamy, Wong,
Fung, & Rai, 2006; Sanina and Kutergina, 2016). This convergence approach led to the creation of a new term: “digital simulation
games” (Devlin-Scherer & Sardone, 2010; Klopfer; Roque, Huang, Wendel, & Scheintaub, 2009).
The digital simulation game is a goal-oriented re-enactment of a real-world process, system, or phenomenon intended to support
learning of academic content and created in the form of software or with the use of a digital platform (Gros, 2007; de Smale et al., 2016;
Vlachopoulos; Makri, 2017). Digital simulation games encourage students to apply their knowledge to real-world problems using a
scenario-based approach. This helps to train many critical skills like analytical skills, strategic skills, problem solving skills,
self-monitoring, and social skills, including interpersonal communication, teamwork, collaboration, negotiation, knowledge sharing,
group decision-making, etc. (Robertson, Schumacher, Gosman; Kanfer, Kelley, & DeVita, 2009; Susi, Johannesson, & Backlund, 2007).
Although merging games and simulations into one concept is debatable (Gredler, 2004), the integration of effective learning in
struments is important when discussing the real educational process. Both teachers and students are interested in high results, and the
combination of digital games and simulations helps to achieve better engagement in the learning process (Vlachopoulos & Makri,
2017), to assimilate classroom information and knowledge in a student-friendly way (Johnson, Adams Becker, Estrada, & Freeman,
2014), and to have chances to practice skills that are impossible to practice in the real world (Sitzmann, 2011). Combining the two
different logics of games and simulations fosters four core factors: motivational, interactive, fun, and multimedia (Jabbar & Felicia,
2015). Through these factors, digital simulation games support the development of emotions and cognition that are crucial for the
effective learning process.
Emotions and cognition are also developed by another educational tool: the co-creation approach. Co-creation involves students in
developing an educational product they will use for learning and practice. Recent studies provide evidence that the involvement of the
co-creation approach in the learning process leads to deeper understanding of knowledge and subject awareness, as well as enhancing
motivation and enthusiasm (Bovill, 2014; Bovill, Cook-Sather, & Felten, 2011), improving collaborative work and learning skills (Blau
2
& Shamir-Inbal, 2017; Bovill et al., 2011; De Jans, Van Geit, Cauberghe, Hudders, & De Veirman, 2017; Ribes-Giner, Perello-Marín, &
Díaz, 2016). Co-creation in educational games appears as a participatory design of the gaming environment developed by students and
instructors. Co-creation makes students equal subjects in the educational process, improving their self-awareness (Elsharnouby, 2015;
Lubicz-Nawrocka, 2018). This leads to increased satisfaction, trust, and loyalty from students (Blau & Shamir-Inbal, 2017; Bowden &
D’Alessandro, 2011; Ribes-Giner et al., 2016). It is proven that students can build good games for learning (Prensky, 2008), as well as
that the co-creative process is an effective tool for developing students’ creativity, communication, and positive group dynamics
(Brandt, Messeter, & Binder, 2008).
Both co-creation and digital simulation games stress relevant points of contemporary education that are connected to learning,
perceiving, and processing information. However, the participatory co-creative design of educational games is mostly perceived by
teachers and researchers as a separate educational task. It is rarely integrated into the regular educational process because it is time-
consuming and labor- and resource-intensive. Moreover, it commonly supposes that the students have technical or programming skills
that are uncommon in social science education, economics, public administration, business, and law.
Still, there are several reasons why co-creation may bring valuable improvements to digital simulation games in social science
education. First, it is now important for educational subjects to be oriented in the complicated environments of social relationships,
administrative decisions, and their consequences and impacts. Digital simulation games put students in a risk-free environment and
provide instant feedback on players’ actions. They may concern different kinds of assignments and situations like crisis management,
health care policy issues, city planning, traffic control, budget balancing, ethics training, and defensive driving (Susi et al., 2007), and
help students to practice situations that are impossible or too expensive to practice in their real-life educational process.
Second, in a time of constant economic change, the digital simulation game as a “finished product” does not reach the ambitious
goals of practically oriented education (Dede, 2002). The possible solution for the improvement of digital simulation games for
educational purposes is the collaborative creation (co-creation) approach. Co-creation fosters students’ engagement and involvement
in learning by making them a valuable part of the instructional process (Blau & Shamir-Inbal, 2017). This is mostly achieved through
enhancing motivation and enthusiasm, because students are usually fond of digital solutions in education. More importantly,
co-creation establishes sustainable dialogue between students and instructors (Bovill, 2014). Consequently, students feel more
responsible for their learning, for they understand that they are taken seriously. As a result, students and teachers tend to value the
processes of collaborative pedagogical planning more than the products of the process (Bovill et al., 2011; Pandey et al., 2017).
Third, co-creation improves essential collaborative work skills (Blau & Shamir-Inbal, 2017; Bovill et al., 2011; De Jans et al., 2017;
Ribes-Giner et al., 2016). Studies show that students who have participated in co-creation tend to have better opportunities on the
labor market, understanding what they are working on and the importance of their efforts (Blau & Shamir-Inbal, 2017; Pandey et al.,
2017; Salas Velasco, 2014).
Based on these arguments, it is possible to suggest that using digital simulation games in combination with the co-creation approach
as a part of the educational process in social science education can significantly improve students’ learning outcomes. In other words,
we can assume that co-created digital simulation games could provide higher educational and professional development outcomes
than basic implementation of digital games in the educational process. It could also provide better learning outcomes than traditional
educational workshop activities like reading, class discussions, and presentations.
These general suggestions lead us to some specific questions to be discussed in this paper:
RQ1. Does a co-creative gamified classroom support better student performance?

RQ2. Does a co-creative gamified classroom provide more positive course evaluation?
RQ3. Does a co-creative gamified classroom encourage a more conscious and motivated approach to students’ future profession?
Considering the advantages of digital simulations and co-creation, and taken into account our previous successful experience of
using digital simulation games in classroom (Sanina & Kutergina 2016), we developed a pedagogical experiment to compare the
differences in student performance, course outcomes, and course evaluation (dependent variables) in a Public Sector Economics class
between the gamified co-creative classroom, gamified classroom, and non-gamified classroom (independent variables).
3. Materials and methods
3.1. Participants and educational strategies
The participants of this study comprised 253 first-year master’s students studying Public Sector Economics at the North-West
Institute of Management, Branch of the Presidential Academy of the National Economy and Public Administration (St. Petersburg,
Russia) in 2017–2018. Forty-three students were enrolled full-time and 210 students were enrolled part-time. The sample has
approximately equal gender distribution (52% female and 48% male), ages 21–49, with the majority of students (42%) aged 22–24.
Before the course, students were randomly divided into three types of groups:
(1) Group A N ¼ 86 (15 full-time students and 71 part-time students) developed a digital simulation game and played it at the end of
the course;
(2) Group B N ¼ 84 (14 full-time students and 70 part-time students) did not develop a digital simulation game but systematically (i.
e., five times during the course) played a prearranged digital simulation game about the Municipal Budget;
(3) Group C N ¼ 83 (14 full-time students and 69 part-time students) did not develop or play a digital simulation game.
3
Lectures were the same for the three groups of students, but the assignments for the workshops were different. For each form of
study (i.e., full-time and part-time), groups B and C performed traditional workshop assignments: reading, class discussions, and
presentations on Public Sector Economics. Group B regularly played the digital simulation game about the Municipal Budget pre
arranged by our research team, using the same template as we offered for the first group. Group C did not play any digital simulation
game.
Group A had an assignment to develop a digital simulation game about the Municipal Budget. At the workshops, there was no
discussion about the lecture material or any presentation assignments. The workshop meetings were almost completely based on
teamwork. The goal of the workshops was to conduct the co-creation environment to develop the simulation game.
3.2. The game
According to the framework of the game, the assignment for Group A was to develop three simulation game scenarios concerning
public budget resource allocation and then to import those scenarios into the special ICT platform. The assignment was designed to
give students the opportunity to understand the administrative diversity of municipalities in the Russian Federation, to better
investigate the peculiarities of the allocation of budgets in various types of Russian municipal structures, and to test options for
administrative decisions in each case. Another goal of the assignment was to develop students’ technical skills. Thus, this assignment
aimed to enrich two key types of skills that should be developed at the university (Bridgstock, 2009), i.e., professional (dis
cipline-specific) and generic skills that cannot be developed outside a disciplinary context (Clanchy & Ballard, 1995) but have a crucial
importance in the contemporary labor market (Muja, Blommaert, Gesthuizen, & Wolbers, 2019). The proposed digital simulation game
integrates certain features of a simulator, a game, and the co-creation process (Blau & Shamir-Inbal, 2017; Clark, Tanner-Smith, &
Killingsworth, 2016; Gredler, 2004).
First, the simulation features are formed by the following conditions: (1) the content of the simulation game should reflect the case
study of a real city; (2) the simulation game should address the issues, threats, or problems arising in the real situation in that real city;
(3) the key activity of the simulation game should be decision making.
Second, the gamed features are formed by the following conditions: (1) all changes in the elements from period to period should
address important content; (2) dynamics of the simulation game, including rules, constraints, and scoring, should be easy to under
stand; (3) winning should be based on knowledge or skills, not random factors; (4) the simulation game should be based on competition
Table 1
Co-creative features of the assignment.
Assignment stages Predetermined course Learning tools used by Level of Level of Students’ Pedagogical rationale (Blau
supplies students for self- cooperation cooperation roles ( & Shamir-Inbal, 2017;
organization with the with the IT Hubbard Arnab; Morini, & Clarke,
instructors Service Desk et al., 2017) 2018; Geddes et al., 2017)
Investigation of the Tutorial; template; Discussion groups in High Low Researchers Active construction of
policy-related hands-on session social networks; knowledge through the
documents, collaborative variety of sources;
including the documents and generalization; improving
budget data database (Google financial and analytical
Drive documents and skills.
spreadsheet)
Writing the gaming Computer game Collaborative High Medium Producers Active learning through the
scripts simulation platform documents and assessment of the real-
guideline; database (Google world situation features in
mathematical model Drive documents) an iterative way to select
within the simulation the best solution of what to
game include in the simulation
game; teamwork and
collaboration; game design
skills.
Searching for the – Discussion groups in Low Low Producers; Active learning and
additional social networks; Researchers collaboration; game design
edutainment Shared folders (Google skills; creativity; openness
supplies Drive, Dropbox, of the course content.
(images, videos) Yandex Disk)
to enrich the
gaming scripts
Importing gaming Computer game Discussion groups in Medium High Producers Learning by doing – basic
scenarios into simulation platform social networks programming skills,
the ICT improving digital
platform competences.
Testing the digital – Discussion groups in High High Partners Testing implication of skills
simulation social networks and knowledge in new
game situation; quality control;
group and self-evaluation;
self-reflection.
4
between teams and a goal to achieve; (5) the simulation game scores should provide feedback to the players to allow for monitoring
their progress.
Third, the co-creative features of the assignment are formed by the logic of interaction between the teacher resource system
(assignment and predetermined course supplies), the student resource system (open sources of information and learning tools used for
self-organization), and the educational co-created system (Geddes, Cannon, & Cannon, 2017) built by the dynamics of students’
research experience, observations and reflections, generalization, and testing the implication of skills and knowledge in a new situ
ation. Table 1 presents the key components and activities as well as the pedagogical rationale (Blau & Shamir-Inbal, 2017) of the
co-creational process. On different stages of the assignment, students perform the roles of researchers, producers, and/or partners
(Hubbard, Brown, Deans, García; Pruna, & Mason, 2017) of the digital simulation game (see Table 2).
To develop three gaming scenarios concerning public budget resource allocation, each student group (i.e., full-time and part-time)
was divided into three teams. The first gaming scenario was developed for a rural settlement. Rural settlements in Russian public
administration unite common territories such as townships, villages, hamlets, farmyards, etc. The second gaming scenario was
developed for an urban settlement (a town in most cases). The third scenario was developed for a city of federal importance; in Russia,
this type of territory also has a municipal budget.
The basic structural unit of the simulation game template is the period. One simulation game comprised 12 periods, and each period
imitated one year. Each period contained a public management case situation in a genuine municipal unit. The assignment of each
period is to make a decision about the best way to allocate funds in accordance with case information and the sum of each municipal
unit’s budget resources. Thus, to develop the content of the educational game, students and course instructors had to generate 12
different public management case situations concerning genuine Russian municipal units, reasoning the best ways to allocate the
budget for each case. The main hero of the game is the head of the given municipal unit, “the Mayor.”
The practically articulated idea of the assignment was that every scenario should also carry the message of social and financial
responsibility for the municipal unit’s development. Public management case situations in the period included background infor
mation about social and economic situations in the municipal unit. Background information within the template consisted of the
following blocks:
Policies of the federal and regional center. This block contained information about the most important vectors of strategic
development of the country and the region within a simulated year.
Reports of the local government. This block provided information about problems of local social and economic development.
Expert assessment of different areas of local economic development. This block provided students with information about civil
society’s priorities for the given year.
Results of sociological monitoring of the population. This block contained information from surveys and interviews with locals
about their current problems and needs.
Table 2
Logic of the research process.
Research stages Stage features
1) Pre-test All students

2) Lectures All students
3) Type of the workshop assignment Group Aa Group Ba Group Ca
N ¼ 14 (fte) þ 71 (pte) N ¼ 15 (fte) þ N ¼ 14 (fte) þ 69 (pte)
70 (pte)
To develop a digital simulation Traditional Traditional workshop activities (reading, class
game workshop discussions, and presentations)
activities
(reading, class
discussions,
and
presentations)
þ trainings
with
prearranged
digital
simulation
game
5) Mode of digital training before the Was trained with a digital Was trained Was not trained with a digital simulation game
examination simulation game with a digital
simulation
game
6) Exam All students
7) Student survey All students
8) Interviews N ¼ 7 (fte) þ 5 (pte)a - –
a
fte ¼ full-time education; pte ¼ part-time education.
5
For the reasonable allocation of the municipal budget, a player was encouraged to take this background information about eco
nomic and social conditions in the municipal unit into consideration in each game period. The students had to organize the background
information so that it represented the interests of different stakeholders such as public servants, experts, and citizens. This was
necessary to bring the game closer to reality and was aimed to develop students’ strong professional competencies. If players over
looked background information containing prompts on budget allocation, they might choose the wrong strategy regarding how to
allocate the municipal budget.
Within the template of the game, the municipal budget should be distributed (in each period) to five key spheres: (1) utilities sector
and land improvements; (2) road facilities and urban transport; (3) health care and social security; (4) education and culture; (5)
development of the economy.
While developing the simulation game, students looked for background information in resources that belong to genuine Russian
municipal units (rural settlement, urban settlement, and city of federal importance). To fill in the template, students turned to policy
and practitioners’ resources such as different local government decrees, websites of local government units, municipal units’ daily
newspapers, press releases, reports from analytical commercial and governmental agencies, and events in various municipal units
published on social media.
The simulation game template for each period was built by a number of stable blocks and content blocks (Fig. 1). The stable blocks
on the main screen (the main tab) included the time counter, the line of periods of the simulation game, the fields for typing the
solutions in five areas of spending of the municipal budget, and the budget of the municipal entity for the current period. In the first
period, the budget size is the same for all players. In the other periods, the dynamics of the participants’ budget revenue change
according to the effectiveness of the players’ previous decisions. The university’s IT services created a mathematical model within the
simulation game: The closer the decision of the player is to the best solution for the period, the bigger the budget will be in the next
period. On the additional screen (the “Finance” tab), players can read the detailed report on the execution of the budget provided
automatically by the game’s mathematical model each period. This additional information helps students plan future decisions.
Apart from the stable blocks on the main screen, there are content blocks that change each period. These blocks include information
created by students: policies of the federal and regional center, reports from members of the government, expert assessment of the
Fig. 1. The Interface of the Game (1st period out of 12). Comments: 1: the time counter; 2: the line of periods of the game; 3: the budget of the
municipal entity for the current period; 4: the fields for typing in the solutions in five areas of spending of the municipal budget; 5: the main tab; 6:
the tab “Finance” (the report on the execution of the budget); 7: the news of the first game period: 8: the mayor’s mail; 9: expert assessment of the
importance of four spheres of governance in municipal entity (a chart); 10: policy of federal and regional center; 11: results of sociological
monitoring of the population; 12: reports from members of the government.
6
importance of spheres of governance in the municipal entity (in the format of a chart), and results of sociological monitoring of the
population.
Thus, in the gaming process, participants get acquainted with the relevant information (the case) and work out a variant of budget
revenue distribution. Distribution requires simple mathematical calculations. The participants enter the sums allocated for each sector
in the player space in the period screen. In the next game period, participants get new information (a new case). Additionally, players
see a results table displaying the rank of all players. The rank of the participant is calculated by a mathematical model on the basis of a
player’s answer compared with an ideal (right) answer. The player’s answer that is closest to the right answer receives one point.
Schematically, the process of passing through a period is shown in Fig. 2.
The game can be played either by one student or by several students. In the case of one student, s/he takes the role of mayor within a
game scenario and makes decisions about budget resource allocation. When several students play the game, one player takes the role of
mayor and works with the other players to make collective decisions about budget allocation. Individual or collective game mode
needs one electronic device with university network access. The choice of individual or collective game mode depends on the lecturer’s
preferences and the number of students in the group.
3.3. Research strategy
The design of the study is based on the outcome-based conception of educational efficiency. This conception links particular aspects
of students’ learning experiences and practical outcomes such as course recommendations for future students, intentions to take the
advanced course on the same subject, or intentions to take another course from the same instructor (Alves & Raposo, 2009; Ang,
Breyer, & Pitt, 2018; Curran & Rosen, 2006). Another important aspect of course satisfaction is positive emotions—e.g., the “joy of
learning” (Ang et al., 2018)—including students’ interest, enjoyment, and positive engagement during the lectures and workshops
(Athiyaman, 1997), as well as their positive acceptance of the given computer technology (Shroff, Deneen, & Ng, 2011). We were also
interested to measure the factor of students’ self-efficacy development, as capstone courses (like Public Sector Economics for Public
Administration students) have an important impact on pre-professional development (Dunlap, 2005). Finally, an important factor of
course evaluation is students’ self-assessment of their personal involvement in the course (i.e., class attendance, engagement in the
group assignments, etc.).
The outcome-based conception of educational efficiency supposes that classroom innovation like digital games and co-creation can
have a broader effect on the students. In particular, they can develop students’ self-confidence in mastering a subject or profession, as
well as their self-concepts and identities of further relevance for career choices (Ja €kel & Borshchevskiy, 2018). Work values like
Fig. 2. The period flowchart. Comments: At the very beginning player gets the web-link to the game and a code to enter the game. When the game
starts, player see on the screen content of the first period. The first step is to read and analyze case information that is displayed on the screen. The
second step is to think about the proportions of amount of money that should be allocated between 5 sectors. Then player make simple calculations
and enter the amounts of expense for 5 sectors. Due to the fact that the time for the period is limited by the course instructor, player has to manage to
enter the answer in time. Otherwise, the simulator automatically counts that the player did not allocate budget resources at all. When the time for
the first period is over, the second period automatically begins. Player sees new case information, player’s result (new sum of money), others players
results, and the winner of the first period (the player who gives solution that is the closest to the right answer). Additionally course instructor could
discuss players’ results. This game cycle repeats till the end of 12 period.
7
“helping others” and “contributing to society” are often used as a proxy of measures of the motivation of civil servants (Lee & Choi,
2016). Thus, we suggest that the design of the digital simulation game about public budgeting could cultivate the public service
motivation of public administration students.
In this game, the contextual assignment was to help students understand the work of a mayor, to make decisions as a mayor, and to
develop a personal understanding and feeling of the nature of public administration work. This means that the digital simulation game
can act as a socialization mechanism for those students who are inspired by public values and ethics of public institutions, and who, as
a final result, were more fascinated about their future profession. This is very important because, as was shown in previous work, “the
greater an individual’s public service motivation, the more likely the individual will seek membership in a public organization” (Perry
& Wise, 1990, p. 370). Prahalad and Ramaswamy (2004) propose that the co-creation experience depends highly on individuals, and
each person’s uniqueness affects the co-creation process. Thus, we extended our model of course evaluation to understand whether
students’ motives and orientations into public institutions and public service employment were higher due to the co-creative design of
the digital game.
Also, we were interested in discovering which features of the gamified co-creative classroom can lead to increasing positive interest
in a student’s future profession. For this, we collected five categories of variables: (1) the edutainment nature of the assignment in
general; (2) the use of computer technologies to develop the game; (3) the use of the co-creation approach to make the game; (4) the
collaborative nature of the assignment; (5) the simulation nature of the assignment.
To evaluate the outcomes of the digital co-created classroom, we developed a three-step model containing:
Student performance, measured through the examination results and game scores;
Table 3
The design of the questionnaire.
Element of course Elucidation of the factor Survey question to measure the factor Scale
evaluation
Satisfaction Students’ evaluation of the lectures’ and Considering the global experience with this 5-point rating scale,
workshops’ content (and course assignments in course, in general what is your level of anchored numerically
particular) as beneficial for their life and satisfaction? and verbally
professional expectations
Joy of learning Students’ evaluation of their interest, enjoyment What is your level of positive emotions for 5-point rating scale,
and positive engagement during the course having participation in this course? anchored numerically
(negative emotions were conditionally nullified, i. and verbally
e. and students were asked to choose “1” if their
emotions were negative)
Self-assessment Students’ self-evaluation of their in-person Until what point do you consider having an 5-point rating scale,
involvement in the course, including lecture and active participation in the course? anchored numerically
tutorial attendance, and participation in and verbally
assignments and class discussions
Self-efficacy Students’ evaluation of their achievements and After completion of the course, until what point 5-point rating scale,
self-development during the course do you consider feeling more confident about anchored numerically
your professional abilities, knowledge and skills? and verbally
Recommendation Whether the students would recommend this How likely you would recommend the course for 5-point rating scale,
intentions course for the next-year students if the content of the next-year students? anchored numerically
the course, as well as the persons of lecturer and and verbally
tutor will not change
Changes in the attitude to Whether the course have helped student to have a Until what point (if any) have this course 5-point rating scale,
future profession better attitude towards his/her future profession positively changed your attitude towards your anchored numerically
of civil servant (negative changes were future profession of civil servant? and verbally
conditionally nullified, and students were asked to
choose “1” if their attitude have not changed or
changed negatively)
Intentions to work in the Students’ intentions to apply for the job in public How likely you would apply for the job in public 5-point rating scale,
public service service service after the completion of the master anchored numerically
program? and verbally
Valuable features of the co- This question was shown only to those students of Upon the completion of the gamification 5-point rating scale,
created digital the A group, who chose “4” or “5” as the answer to assignment, until what point you would rate the anchored numerically
simulation game the question “Until what point (if any) have this following features of the game as valuable for and verbally
course positively changed your attitude towards you:
your future profession of civil servant?” the gamification nature of the
assignment in general;
the use of computer technologies to
develop the game;
the use of the co-creation approach to
make the game;
the collaborative (interactive) nature of
the assignment;
the simulation nature of the assignment
8
Course evaluation, measured through course satisfaction, emotions from the course, self-assessment, and course recommendation
intentions;
Course outcomes, measured through self-efficacy, attitude toward future profession, and intentions to work in public service.
This model and stated research questions led to the following hypotheses:
H1. A learning process based exclusively on traditional pedagogical techniques is the least efficient;
H2. The implementation of simulation games in the learning process increases learning outcomes significantly;
H3. The combination of the co-creation learning approach and game-based education could provide the highest learning outcomes;
H4. The more a student is actively engaged in the learning process, the more positively s/he reviews an attitude toward the future
profession;
H5. The better scores a student has in the digital game, the more positively s/he reviews an attitude toward the future profession.
3.4. Methods
This study used a mixed-method approach in which quantitative data collection and analysis is followed by the qualitative data
collection phase. The qualitative phase helps to explain why certain features of the game, tested quantitatively, were significant or not
significant for the students. The data of this study were collected through quantitative assessment (digital games results and exami
nation tests), questionnaire surveys, and personal interviews to provide methodological triangulation (Ivankova, Creswell, & Stick,
2006).
Before the course, all 253 students passed the pre-test evaluation consisting of 30 multiple-choice questions. The purpose of the pre-
test was to assess students’ knowledge about public finance issues as a central part of the Public Sector Economics class. Before the final
examination, students in the A and B groups were trained with the new version of the digital simulation game that partly used students’
developments but was changed by the course instructors. At the end of the simulation game, students got calculated percentages of
correctly solved cases. At the end of the course, all 253 students passed the examination test consisting of 30 multiple-choice questions
similar to the pre-test questions. Results of the exam were the post-test step of the experiment.
After that, students in all groups were asked to answer the feedback questionnaire concerning the evaluation of the course. All 253
students completed the survey questionnaires, and 12 students from the gamified co-creative classroom volunteered to be interviewed.
The design of the survey questionnaire is shown in Table 3. Each element of the students’ evaluation was measured using a single-
item scale. This type of measurement is proven to provide valid results when a detailed and comprehensive questionnaire is not
desirable or possible (Ang et al., 2018; Robins & Trzesniewski, 2001). It particularly fitted our assignment to have course evaluation in
the situation of students’ fatigue of several final examinations accompanied by teaching evaluation at the end of the term. To make
questions more concrete and unambiguous, we elucidated them in a way described in Table 3. The elucidation was stated under every
question for the student’s information. For all measures, we used 5-point rating scales, anchored numerically and verbally: 1 –
exceptionally low level of a feature, 2 – low level of a feature, 3 – an average level of a feature, 4 – high level of a feature, 5 –
exceptionally high level of a feature. Because we asked students about their personal experience and were interested in collecting more
reliable responses, we used a no-midpoint scale and offered a “no opinion” option that was treated as missing data (Tsang, 2012;
Nadler, Weston, & Voyles, 2015). The questionnaires were distributed electronically after the examination and were completed at the
classroom computers to avoid non-response bias and to ensure anonymity of the survey.
After processing the results of the survey, seven full-time and five part-time students from the gamified co-creative classroom were
interviewed face-to-face by a researcher who developed the project of the game but was not their course instructor. The interviews,
varying in length from 15 to 35 min, were audio-taped and then analyzed by the researchers. We used the interview data to better
understand the quantitative results and to answer the research questions.
4. Results
4.1. Grouping data
Full-time students and part-time students studied and completed assignments separately and in slightly different time. Of course,
the full-time education group had more opportunities for live interaction with the course instructor and teammates; however, for the
part-time students, the lack of full-time participation was compensated by the larger size of their teams (23 or 24 students vs. 5 students
in the team of the full-time group). Most importantly, there is no statistically significant association between the type of enrollment and
total game score. Therefore, for analytical purposes and discussing results, full-time students and part-time students are united in single
groups (A, B, and C).
Pre-test results in the experimental groups are approximately the same: MGroupA ¼ 12.45, SD ¼ 2.69; MGroupB ¼ 12.77, SD ¼ 2.87;
MGroupC ¼ 12.80, SD ¼ 2.83 (Fig. 3). ANOVA indicates that there are no significant differences among the performances of the groups (F
(2,250) ¼ 0.415, p > 0.05). Consequently, we can assume that students had approximately the same level of knowledge before the
course.
9
4.2. Students’ performance
Students in Group A had the best results of the examination test: MGroupA ¼ 26.01, SD ¼ 2.07; MGroupB ¼ 23.08, SD ¼ 3.13; MGroupC
¼ 20.95, SD ¼ 4.31 (Fig. 4). ANOVA indicated that there were significant differences among the performances of the groups (F (2,250)
¼ 50.5, p < 0.001). Post hoc pairwise comparison of means using Tukey’s HSD test showed that the differences between each pair of
group means were significant at the 0.01 level of significance. Consequently, the conclusion is that the implementation of digital
simulation games along with the co-creation approach was a significant factor in the performance of the students. Moreover, the
learning process based exclusively on traditional pedagogical techniques was the least effective. These results support the first and the
second hypotheses (H1, H2).
Students in groups A and B played simulation games before the examination test. Students in Group A had the best results: MGroupA
¼ 11.22, SD ¼ 0.67; MGroupB ¼ 9.90, SD ¼ 1.04 (Fig. 5). ANOVA indicated that there were significant differences among the per
formances of the group means (F (1,168) ¼ 91.46, p < 0.001). Hence, results support the third hypothesis (H3): The combination of the
co-creation learning approach and game-based education could provide the highest learning outcomes.
4.3. Course evaluation and learning outcomes
Students were asked to give feedback after the course. Table 4 includes a summary of student survey responses. In general, student
survey responses support the post-test and gaming results. The combination of the co-creation learning approach and digital simulation
games could provide the highest level of learning outcomes. Group A has the highest mean scores of each question from 1 to 7,
suggesting that game-based education along with a co-creation learning approach is the most effective way to increase the level of
students’ satisfaction, joy of learning, self-assessment, and self-efficacy. Students in the gamified co-creative classroom are more likely
to recommend such a course for incoming students.
This statement is supported by the interviews with the students:
It was one of the best courses, and one of the best assignments, because it let us feel as if we are important and can do something
important for the place we live in. < … > finally we’ve got an explanation on how strategic decisions are really made. I have
never done anything so practical, and sure, I will recommend this course exactly because it is so practical. (Woman, 23, full-
time)
I know a couple of people in our group who didn’t like the task, but that is because they work already and it is more comfortable
for them to perform those boring workshops, reading, and do some meaningless presentations. They were not fully engaged in
the course, actually. Those who were, we all liked it. (Woman, 26, part-time)
Students in Group A got additional questions to evaluate different features of the gamified co-creative classroom. We were
interested in learning if any particular aspects of the assignment were especially interesting for the students. Table 5 represents mean
scores of student survey responses.
The distribution of answers shows that the greatest points of interest are the gamification nature of the assignment (4.99) and using
computer technologies to develop the game (4.98). The co-creation approach and simulation nature of assignment received a slightly
lower point of students’ appreciation (4.87). The lowest mean score belongs to the collaborative or interactive feature of the task.
Although the new form of learning activity was positively assessed by students, the collaborative nature of the assignment is the
only feature of the game that got assessments of 3–5 points. While most of the students highly ranked this feature, a meaningful number
of students ranked it as 4, and a small number of students ranked it as 3 (Fig. 6). We might assume that in-group relationships between
group members influenced the level of satisfaction with the course. However, the association between the value of the collaborative
nature of the assignment and level of satisfaction with the course is weak (ρ ¼ 0.22, p < 0.05).
In the interviews, students mentioned the difficulties in organizing their teamwork outside of class. The self-study process as a team
effort was hard both for the full-time and part-time students. Negative group dynamics that could affect the value of collaboration
Fig. 3. Pre-test results in experimental groups A, B, and C.
10
Fig. 4. Post-test results in experimental groups A, B, and C.
Fig. 5. Simulation games results in experimental groups A and B.
Table 4
Course evaluation: selected student survey questions and means.
Student survey question Mean score
Group Group Group

A B C
(1) Considering the global experience with this course, in general what is your level of satisfaction? 4.77 4.60 4.25
(2) What is your level of positive emotions for having participation in this course? 4.73 4.54 4.17
(3) Until what point do you consider having an active participation in the course? 4.87 4.64 4.35
(4) After completion of the course, until what point do you consider feeling more confident about your 4.74 4.17 3.63
professional abilities, knowledge and skills?
(5) How likely you would recommend the course for the next-year students? 4.79 4.67 3.88
Table 5
Course features rating by the students.
Student survey question: Mean score
Upon the completion of the gamification assignment, until what point you would rate the following features of the game as valuable for you …
(8.1) … the gamification nature of the assignment in general 4.99

(8.2) … the use of computer technologies to develop the game 4.98
(8.3) … the use of the co-creation approach to make the game 4.87
(8.4) … the collaborative (interactive) nature of the assignment 4.6
(8.5) … the simulation nature of the assignment 4.87
became more acute in the second stage of development of the game when it was necessary to learn new skills. The course instructors
emphasized a special focus of the game development assignment for the acquisition of basic computer skills, but some students
preferred the strategy of separation by assignments. This caused dissension inside the group:
[At the second stage] we argued a lot, because some people knew how to work with the computer programming, but most of us
didn’t. < … > I felt stupid. I really did not like this stage of game elaboration. I tried to use manuals and so on, all the in
structions from the class, but that was not enough for sure. At the end, we decided to organize separate meetings for everybody
in the group, but only three of us came, and one of them … well, he explained something, but not too much. (Woman, 22, full-
time)
11
Fig. 6. Rating of collaborative (interactive) nature of the assignment by the students.
From the beginning, when we knew our assignment would have a “computer” part, we divided our team, thank God it was big
enough, into two smaller teams, and several people who were confident in computers, they did almost nothing for the first stage
because they did the computer part. It was, I would say, an effective way to perform a team assignment, but not an effective way
of learning, because, for example, I cannot repeat this all again by myself. On the other hand, I don’t think it’s bad. I know, like,
the methodology, and if I need I can always find an executor for the computer part. (Man, 30, part-time)
Among the six scenario teams, there were two groups of full-time students in which none of the participants had any programming-
related experience and who, according to the respondents, “were equally unfamiliar with the computer.” In these teams, the orga
nization of work was different. At the second stage of game elaboration, they practiced close interaction with the University IT
Department, reading computer literature and watching training videos, as well as participation in computer forums for beginners.
Despite different strategies of teamwork, most of the students gave the highest points to the computer-based technologies when
rating different features of the game (Mean score ¼ 4.98). As the above respondent noted, regardless of involvement in the computer
part of the game development, all members of the teams were able to understand the logic of the gameplay and obtain new, valuable
experience that will help them in their future profession. However, both quantitative and qualitative results led us to the thought that
the students’ self-study monitoring by the course instructor is important for the entire period of the course. If we want to teach new
skills, their acquisition should be controlled by intermediate tests or other types of control. Unfortunately, we did not introduce in
termediate forms of control in our experiment.
4.4. Students’ attitudes regarding their future profession
Both quantitative and qualitative results show that the educational approach combining digital simulation and co-creation posi
tively changes students’ attitudes toward their future profession (Table 6). Practical learning reinforces students’ intentions to work in
public service. There is an association between the use of the co-creation approach to make the game (Q8.3) and students’ interest in
their future profession (Q6) (ρGroupA ¼ 0.31, p < 0.01). Also, there is an association between the collaborative nature of the assignment
(Q8.4) and interest in their future profession (Q6) (ρGroupA ¼ 0.24, p < 0.05). There are no strong associations between the value of the
other two features of the gamified co-creation classroom (i.e., the gamification nature of the assignment in general and the use of
computer technologies) and interest in the future profession. This highlights the role of the co-creational and collaborative approach to
digital simulation as a factor of students’ satisfaction and professional development.
For students in Group A, there is an association between the level of participation in the course (Q3) and attitude toward the future
profession (Q6) (ρGroupA ¼ 0.27, p < 0.01). The same association for students in Group B and Group C is not statistically significant.
This indicates partial support for the fourth hypothesis (H4), and it can be assumed that the more a student is actively engaged in the
co-creative environment of the course, the more positively s/he reviews an attitude toward the future profession.
Moreover, on the basis of the game results of students in Group A and Group B, it can be concluded that the better score a student
has in the digital game, the more positively s/he reviews an attitude toward the future profession (ρ ¼ 0.29, p < 0.01). Consequently,
results support the fifth hypothesis (H5). In general, the idea of a co-creative gamification classroom works better than single gami
fication or traditional workshop assignments.
In their interviews, students stressed that the co-creative gamification assignment supported and improved their job motivation and
self-esteem as future civil servants, and also clarified the public administration assignment perception.
Table 6
Course evaluation in relation to public servants’ profession.
Student survey question Mean score
Group A Group B Group C
(1) Until what point (if any) have this course positively changed your attitude towards your future profession of 4.56 4.19 3.83
civil servant?
(2) How likely you would apply for the job in public service after the completion of the master program? 4.77 4.23 3.95
12
… if the civil service profession looks even a little like this—and now I believe it does, because we have not only studied
documents, but also talked to some public administration executors—then I am excited to be a civil servant. (Man, 23, part-
time)
My team was very inspired with the opportunity to make somebody feel like a mayor. The decision environment is not so easy,
and we were happy to have some hints for better decisions. (Woman, 22, full-time)
… Before, all the courses were too theoretical, too boring. I began to think that it’s so boring to be a civil servant. But this course
changed my thoughts … [Civil service] could be honest and honored work, I think. (Man, 28, part-time)
Most of the game scenario emphasized the social responsibility of the administrative job and the necessity of budget allocation
according to the people’s needs and priorities. For example, in one of the scenarios, students wrote the following preamble (Box 1):
Socially-oriented work values were considered by the students as among the most important points of motivation. The co-creative
gamified classroom experience supported our suggestion that the development of a computer game about public budgeting could
cultivate the public service motivation of the public administration students, and we consider this as the best outcome of this peda
gogical experience.
5. Discussion and conclusion
The main goal of our research was to analyze the educational effects of co-created digital simulation game implementation and to
compare them with more traditional types of educational environments, i.e., gamified and non-gamified classrooms. Previous studies
show that digital simulation games and co-creation are important sources of professional experience because they increase students’
satisfaction with the course; enhance motivation, enthusiasm, collaborative participation, and interactions with the instructor; and
help to develop students’ professional abilities by applying theoretical knowledge in the simulation of real-life situations. Our research
shows that combination of the co-creation technique and simulation game development as a part of the educational process in social
sciences can improve student performance, course outcomes, and course evaluation.
With the implementation of co-creation in the process of computer-based and game-based education, we take a step toward the
improvement of the skills that students may need in their potential future jobs. The pedagogical rationale of the co-creative approach is
aimed at the development of students’ soft and generic skills (i.e., communication, teamwork, creativity, and digital competence). Our
study shows that it also trains their professional skills, because to create the gamification assignment, the students must think like their
professional referents—mayors in the case of our simulation game. Moreover, with the co-creative gamification approach, students
learned from trial and error, thus better understanding their subject. The assignment to make decisions as a mayor forced the creators
of the game to develop a personal understanding and feeling of the nature of public administration work. In the context of all previous
courses the students had taken, it was an impossible task due to theoretical overloading. However, the co-creative gamification
assignment encouraged a more conscious and motivated approach to their future profession.
This study has several limitations. First of all, it is possible that some personal characteristics of students could influence the results
of the gamified co-creative assignment and evaluation of learning outcomes. On the one hand, recent research suggests an association
between a person’s demographic characteristics and outcomes of the co-creation process: According to a study by Cambra-Fierro,
P�erez, and Grott (2017) dedicated to understanding co-creation from a customer viewpoint, gender and age have an influence on the
co-creation framework. Additionally, students of the master’s programs are already highly interested in their future profession, having
consciously chosen this specialization. Moreover, most of the part-time students are working in civil service-related organizations. On
the other hand, although we did not release the difference in assignments for the students of the various groups, we could not keep
them isolated; thus, the low motivation and satisfaction of Group C may be partly caused by comparing themselves to the groups with
more interesting and engaging assignments.
Second, we did not divide students according to their computer skills and programming experience. On average, the students
enrolled in a social sciences education program do not have computer programming skills; thus, the game template was to be very
simple so that it could be confidently executed and tested over one semester. Still, some of the students did have advanced pro
gramming and digital skills. Differences in the students’ qualifications may have caused some analytical bias, because those groups
with enough experienced computer users could spend less time on the second stage of the game and thus had more time to prepare for
the examination test.
Third, in this study, students’ self-organization was traced back only through the interviews. According to University rules, we
could not change the timetable grid or the curriculum for only one group of students; thus, we had to include the development of the
game in the framework of the traditional grid of lectures and workshops. We did not monitor or control students’ self-organization
activities, but as the interviews showed, it was probably a good way to regulate the dynamics of learning the new skills and to pre
vent disputes concerning different programming experience. Different studies show that for beneficial co-creation to take place, both
sides of the co-creation process must be involved, engaged, and participating constructively and actively (Oertzen,
Odekerken-Schro €der, Brax, & Mager, 2018). Consequently, to get the most from the learning process with the co-creation technique, a
course should be organized so that all students are equally engaged in all stages of the simulation game elaboration.
Finally, we did face negative effects of digital simulation game co-creation. Some studies show that poorly designed games and
simulations could force frustration, loss of interest, dissatisfaction, bad feelings of players, guessing and the selection of wrong answers
instead of thoughtful choices, focusing on wrong stimulus that is counterproductive to the expected learning, etc. (Gredler, 1996; Can
& Cagiltay, 2006). When playing the co-created digital game, students who chose wrong strategies at the beginning of the game, lost
their interest in the middle of the game because it was almost impossible to win or improve the scores significantly. As a result, they
13
Box 1
The reflection of the idea of social responsibility in the game preamble developed by the students
You will take part in the game, “The management of the municipal budget of Yekaterinburg City.” In this game, you are playing
the role of the head of the Yekaterinburg Administration for the period 2006–2018. During the game, you will get acquainted
with the most urgent problems of the city. The purpose of the game is to learn how to make strategically important decisions on
city budget allocation, to practice analytical skills and theoretical knowledge in the field of state and municipal finance, and to
develop your attention and logical thinking. The main idea of the game is to show that the budget is not only a “wallet” that
receives money, but is a basis for city development according to priority areas on which money is distributed.
were dissatisfied with game and thus with the course. Despite the fact that such negative effect was only in isolated cases, we should
find ways to avoid it.
In general, the proposed experiential gaming framework of using a gamified co-creative classroom as part of the ongoing educa
tional process in social science can be used to design engaging educational games. Although this framework is a work in progress, the
quantitative and qualitative results show that it may create new learning environments and educational cultures that better correspond
with students’ habits and interests.
Author contribution
Anna Sanina, Evgeniia Kutergina, and Aleksey Balashov contributed to the idea, design and implementation of the research, to the
analysis of the results and to the writing of the manuscript.
Acknowledgments
This research was supported by Vladmir Potanin Foundation Scholarship, Russia (Project “Computer Simulators in Education",
Grant N ГК160000478). We are thankful to all the experts who provided evaluation of our project that greatly assisted the research. We
are also grateful to anonymous reviewers for their comments and suggestions that really improved the paper although they may not
agree completely with all of the interpretations provided by the authors.
References
Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., & Ananthanarayanan, V. (2017). NMC horizon report - 2017 higher education edition. Austin,
Texas.
Alves, H., & Raposo, M. (2009). The measurement of the construct satisfaction in higher education. Service Industries Journal, 29(2), 203–218. https://doi.org/
10.1080/02642060802294995.
Ang, L., Breyer, Y. A., & Pitt, J. (2018). Course recommendation as a construct in student evaluations: Will students recommend your course? Studies in Higher
Education, 43(6), 944–959. https://doi.org/10.1080/03075079.2016.1199543.
Armbruster, P., Patel, M., Johnson, E., & Weiss, M. (2009). Active learning and student-centered pedagogy improve student attitudes and performance in Introductory
Biology. Life Sciences Education, 8, 203–213. https://doi.org/10.1187/cbe.09.
Arnab, S., Morini, L., & Clarke, S. (2018). Co-Creativity with playful and gameful inspirations. In M. Ciussi (Ed.), Proceedings of the 12th European conference on game-
based learning, ECGBL 2018 (pp. 1–7). Academic Conferences and publishing limited.
Athiyaman, A. (1997). Linking student satisfaction and service quality perceptions: The case of university education. European Journal of Marketing, 31(7), 528–540.
https://doi.org/10.1108/03090569710176655.
Blau, I., & Shamir-Inbal, T. (2017). Re-designed flipped learning model in an academic course: The role of co-creation and co-regulation. Computers & Education, 115,
69–81. https://doi.org/10.1016/j.compedu.2017.07.014.
Bovill, C. (2014). An investigation of co-created curricula within higher education in the UK, Ireland and the USA. Innovations in Education & Teaching International, 51
(1), 15–25. https://doi.org/10.1080/14703297.2013.770264.
Bovill, C., Cook-Sather, A., & Felten, P. (2011). Students as co-creators of teaching approaches, course design, and curricula: Implications for academic developers.
International Journal for Academic Development, 16(2), 133–145. https://doi.org/10.1080/1360144X.2011.568690.
Bowden, J. L. H., & D’Alessandro, S. (2011). Co-creating value in higher education: The role of interactive classroom response technologies. Asian Social Science, 7(11),
35–47. https://doi.org/10.5539/ass.v7n11p35.
Brandt, E., Messeter, J., & Binder, T. (2008). Formatting design dialogues – games and participation. CoDesign, 4(1), 51–64. https://doi.org/10.1080/
15710880801905724.
Bridgstock, R. (2009). The graduate attributes we’ve overlooked: Enhancing graduate employability through career management skills. Higher Education Research and
Development, 28(1), 31–44. https://doi.org/10.1080/07294360802444347.
Brush, T., & Saye, J. (2000). Implementation and evaluation of a student-centered learning unit: A case study. Educational Technology Research & Development, 48(3),
79–100. https://doi.org/10.1007/BF02319859.
Burde, D., Kapit, A., Wahl, R. L., Guven, O., & Skarpeteig, M. I. (2017). Education in emergencies: A review of theory and research. Review of Educational Research, 87
(3), 619–658. https://doi.org/10.3102/0034654316671594.
Burnard, P. (2006). Reflecting on the creativity agenda in education. Cambridge Journal of Education, 36(3), 313–318. https://doi.org/10.1080/03057640600865801.
Cambra-Fierro, J., P� erez, L., & Grott, E. (2017). Towards a co-creation framework in the retail banking services industry: Do demographics influence? Journal of
Retailing and Consumer Services, 34, 219–228. https://doi.org/10.1016/j.jretconser.2016.10.007.
Can, G., & Cagiltay, K. (2006). Turkish prospective teachers’ perceptions regarding the use of computer games with educational features. Journal of Educational
Technology & Society, 9(1), 308–321.
Clanchy, J., & Ballard, B. (1995). Generic skills in the context of higher education. Higher Education Research and Development, 14(2), 155–166.
14
Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational
Research, 86(1), 79–122. https://doi.org/10.3102/0034654315582065.
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious
games. Computers & Education, 59(2), 661–686.
Cornelius-White, J. (2007). Learner-centered teacher-student relationships are effective: A meta-analysis. Review of Educational Research, 77(1), 113–143. https://doi.
org/10.3102/003465430298563.
Curran, J. M., & Rosen, D. E. (2006). Student attitudes toward college courses: An examination of influences and intentions. Journal of Marketing Education, 28(2),
135–148. https://doi.org/10.1177/0273475306288401.
De Jans, S., Van Geit, K., Cauberghe, V., Hudders, L., & De Veirman, M. (2017). Using games to raise awareness: How to co-design serious mini-games? Computers &
Education, 110, 77–87. https://doi.org/10.1016/j.compedu.2017.03.009.
Dede, M. J. (2002). The praxis journal: Integrating theory and practice. Journal of Public Affairs Education, 8(4), 287–297. https://doi.org/10.1080/
15236803.2002.12023561.
Devlin-Scherer, R., & Sardone, N. B. (2010). Digital simulation games for social studies classrooms. The Clearing House: A Journal of Educational Strategies, Issues and
Ideas, 83(4), 138–144. https://doi.org/10.1080/00098651003774836.
Díaz-M�endez, M., & Gummesson, E. (2012). Value co-creation and university teaching quality. Journal of Service Management, 23(4), 571–592. https://doi.org/
10.1108/09564231211260422.
Dunlap, J. C. (2005). Problem-based learning and self-efficacy: How a capstone course prepares students for a profession. Educational Technology Research &
Development, 53(1), 65–83. https://doi.org/10.1007/BF02504858.
Elsharnouby, T. H. (2015). Student co-creation behavior in higher education: The role of satisfaction with the university experience. Journal of Marketing for Higher
Education, 25(2), 238–262.
Geddes, B. C., Cannon, H. M., & Cannon, J. N. (2017). Conceptualizing Co-creative strategies in experiential education: Individual versus group Approaches. In
Developments in business simulation and experiential learning: Proceedings of the annual ABSEL conference (Vol. 44) (1).
Ghinea, G. (2013). Co-creation of value in higher education: Using social network. Journal of Higher Education Policy and Management, 35(1), 45–53. https://doi.org/
10.1080/1360080X.2013.748524.
Gredler, M. E. (1996). Educational games and simulations: A technology in search of a (research) paradigm. Technology, 39, 521–540.
Gredler, M. E. (2004). Games and simulations and their relationships to learning. In D. Jonassen (Ed.), Handbook of research for educational communications and
technology (pp. 571–581). Mahwah, NJ: Lawrence Erlbaum Associates.
Greenhow, C., Robelia, B., & Hughes, J. E. (2009). Learning, teaching, and scholarship in a digital age. Educational Researcher, 38(4), 246–259. https://doi.org/
10.3102/0013189X09336671.
Gros, B. S. (2007). Digital games in education: The design of games-based learning environments. Journal of Research on Technology in Education, 40(1), 23–39. https://
doi.org/10.1080/15391523.2007.10782494.
Hamidi, Y. D., Wennberg, K., & Berglund, H. (2008). Creativity in entrepreneurship education. Journal of Small Business and Enterprise Development, 15(2), 304–320.
https://doi.org/10.1108/14626000810871691.
Honey, M. A., & Hilton, M. (2011). Learning science through computer games and simulations. The National Academies Press.
Huang, W. D., & Oh, E. G. (2018). Motivational support from digital game-based learning environments (DGBLEs) for scientific topics designed by novice end users.
Educational Media International, 55(2), 123–136. https://doi.org/10.1080/09523987.2018.1484043.
Hubbard, K. E., Brown, R., Deans, S., García, M. P., Pruna, M.-G., & Mason, M. J. (2017). Undergraduate students as co-producers in the creation of first-year practical
class resources. Higher Education Pedagogies, 2(1), 58–78. https://doi.org/10.1080/23752696.2017.1338529.
Ivankova, N. V., Creswell, J. W., & Stick, S. L. (2006). Using mixed-methods sequential explanatory design: From theory to practice. Field Methods, 18(1), 3–20.
https://doi.org/10.1177/1525822X05282260.
Jabbar, A. I., & Felicia, P. (2015). Gameplay engagement and learning in game-based learning: A systematic review. Review of Educational Research, 85(4), 740–779.
https://doi.org/10.3102/0034654315577210.
J€
akel, T., & Borshchevskiy, G. A. (2018). Who wants to work in bureaucracy? Career intentions of post-millennial students. Teaching Public Administration, 37(1),
67–91. https://doi.org/10.1177/0144739418806553.
Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2014). NMC horizon report - 2014 (K-12 Edition), ISBN 978-0-9897335-5-7. Austin, Texas.
Jones, L. (2007). The Student-centered classroom. New York: Cambridge University Press.
Kirkup, G., & Kirkwood, A. (2005). Information and communications technologies (ICT) in higher education teaching - a tale of gradualism rather than revolution.
Learning, Media and Technology, 30(2), 185–199. https://doi.org/10.1080/17439880500093810.
Klabbers, J. H. G. (2009). Terminological ambiguity game and simulation. Simulation & Gaming, 40(4), 446–463.
Klopfer, E., Roque, R., Huang, W., Wendel, D., & Scheintaub, H. (2009). The simulation cycle: Combining games, simulations, engineering and science using StarLogo
TNG. E-learning, 6(1), 71. https://doi.org/10.2304/elea.2009.6.1.71.
Lee, G., & Choi, D. L. (2016). Does public service motivation influence the college students’ intention to work in the public sector? Evidence from korea. Review of
Public Personnel Administration, 36(2), 145–163. https://doi.org/10.1177/0734371X13511974.
Loh, C. S. (2007). Designing online games assessment as: Information trails. In Games and simulations in online learning: Research and development frameworks (pp.
323–348). IGI Global.
Lubicz-Nawrocka, T. (2018). From partnership to self-authorship: The benefits of Co-creation of the curriculum. International Journal for Students as Partners, 2(1),
47–63.
Molin, G. (2017). The role of the teacher in game-based learning: A review and outlook. In Serious games and edutainment applications (pp. 649–674). Cham: Springer
International Publishing. https://doi.org/10.1007/978-3-319-51645-5_28.
Muja, A., Blommaert, L., Gesthuizen, M., & Wolbers, M. H. (2019). The role of different types of skills and signals in youth labor market integration. Empirical Research
in Vocational Education and Training, 11(1), 6.
Nadler, J. T., Weston, R., & Voyles, E. C. (2015). Stuck in the middle: The use and interpretation of mid-points in items on questionnaires. The Journal of General
Psychology, 142(2), 71–89. https://doi.org/10.1080/00221309.2014.994590.
Narayanasamy, V., Wong, K. W., Fung, C. C., & Rai, S. (2006). Distinguishing games and simulation games from simulators. Computer Entertainment, 4(2), 9.
Oertzen, A. S., Odekerken-Schr€ oder, G., Brax, S. A., & Mager, B. (2018). Co-creating services - conceptual clarification, forms and outcomes. Journal of Service
Management, 29(4), 641–679. https://doi.org/10.1108/JOSM-03-2017-0067.
Pandey, V., Amir, A., Debelius, J., Hyde, E. R., Kosciolek, T., Knight, R., et al. (2017). Gut instinct: Creating scientific theories with online learners. In Proceedings of the
2017 CHI conference on human factors in computing systems (pp. 6825–6836). Denver, CO: ACM. https://doi.org/10.1145/3025453.3025769.
Perry, J., & Wise, L. (1990). The motivational bases of public service. Public Administration Review, 50(3), 367–373. https://doi.org/10.2307/976618.
Prahalad, C. K., & Ramaswamy, V. (2004). Co-creating unique value with customers. Strategy & Leadership, 32(3), 4–9. https://doi.org/10.1108/
10878570410699249.
Prensky, M. (2007). Digital game-based learning. St. Paul, MN: Paragon House.
Prensky, M. (2008). Students as designers and creators of educational computer games: Who else? British Journal of Educational Technology, 39(6), 1004–1019. https://
doi.org/10.1111/j.1467-8535.2008.00823_2.x.
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231. https://doi.org/10.1002/j.2168-9830.2004.
tb00809.x.
Razzouk, R., & Shute, V. (2012). What is design thinking and why is it important? Review of Educational Research, 82(3), 330–348. https://doi.org/10.3102/
0034654312457429.
15
Ribes-Giner, G., Perello-Marín, M. R., & Díaz, O. P. (2016). Co-creation impacts on student behavior. Procedia - Social and Behavioral Sciences, 228, 72–77. https://doi.
org/10.1016/j.sbspro.2016.07.011.
Robertson, B., Schumacher, L., Gosman, G., Kanfer, R., Kelley, M., & DeVita, M. (2009). Simulation-based crisis team training for multidisciplinary obstetric providers.
Simululation in Healthcare, 4(2), 77–83. https://doi.org/10.1097/SIH.0b013e3181917cd.
Robins, R., & Trzesniewski, K. (2001). Measuring global self-esteem: Construct validation of a single-item measure and the Rosenberg Self-Esteem Scale. Personality
and Social Psychology Bulletin, 27(2), 151–161. https://doi.org/10.1177/0146167201272002.
Salas Velasco, M. (2014). Do higher education institutions make a difference in competence development? A model of competence production at university. Higher
Education, 68(4), 503–523. https://doi.org/10.1007/s10734-014-9725-1.
Sanina, A., & Kutergina, E. (2016). Using computer-based simulation games to improve the competencies of MPA graduates. In Proceedings of the 15th European
conference on e-learning ECEL 2016 (pp. 622–630). Prague: Charles University.
Sawyer, R. K. (2004). Creative teaching: Collaborative discussion as disciplined improvisation. Educational Researcher, 33(2), 12–20. https://doi.org/10.3102/
0013189X033002012.
Shroff, R. H., Deneen, C. D., & Ng, E. M. W. (2011). Analysis of the technology acceptance model in examining students’ behavioural intention to use an e-portfolio
system. Australasian Journal of Educational Technology, 27(4), 600–618. http://www.ascilite.org.au/ajet/ajet27/shroff.htm.
Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489–528.
https://doi.org/10.1111/j.1744-6570.2011.01190.x.
de Smale, S., Overmans, T., Jeuring, J., & van de Grint, L. (2016). The effect of simulations and games on learning objectives in tertiary education: A systematic
review. In A. De Gloria, & R. Veltkamp (Eds.), International conference on games and learning alliance (pp. 506–516). Cham: Springer.
Susi, T., Johannesson, M., & Backlund, P. (2007). Serious games – an overview. Sk€
ovde. Retrieved from http://www.autzones.com/din6000/textes/semaine12/SusiEtAl
(2005).pdf.
Tsang, K. K. (2012). The use of midpoint on Likert Scale: The implications for educational research. Hong Kong Teachers’ Centre Journal, 11(1), 121–130.
Van Horn, R. (2007). Educational games. Phi Delta Kappan, 89(1), 73–74.
Vlachopoulos, D., & Makri, A. (2017). The effect of games and simulations on higher education: A systematic literature review. International Journal of Educational
Technology in Higher Education, 14(1), 22.
Woo, J.-C. (2014). Digital game-based learning supports student motivation, cognitive success, and performance outcomes. Journal of Educational Technology &
Society, 17, 291–307. https://doi.org/10.2307/jeductechsoci.17.3.291.
16

Simulation Games PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Simulation Games PDF

Uploaded by

Copyright:

Available Formats

Computers & Education 149 (2020) 103813

Contents lists available at ScienceDirect

Computers & Education

The Co-Creative approach to digital simulation games in social

RQ1. Does a co-creative gamified classroom support better student performance?

3. Materials and methods

3.1. Participants and educational strategies

3.2. The game

1) Pre-test All students

3.3. Research strategy

4.1. Grouping data

4.2. Students’ performance

4.3. Course evaluation and learning outcomes

Fig. 3. Pre-test results in experimental groups A, B, and C.

Fig. 4. Post-test results in experimental groups A, B, and C.

Fig. 5. Simulation games results in experimental groups A and B.

Group Group Group

(8.1) … the gamification nature of the assignment in general 4.99

Fig. 6. Rating of collaborative (interactive) nature of the assignment by the students.

4.4. Students’ attitudes regarding their future profession

Group A Group B Group C

5. Discussion and conclusion

You might also like