See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/260255667

Technologies of Inclusive Well-Being: Serious Games, Alternative Realities,

and Play Therapy

Book  in  Studies in Computational Intelligence · January 2014

DOI: 10.1007/978-3-642-45432-5

CITATIONS READS

17 849

43 authors, including:

Anthony Lewis Brooks Sheryl Brahnam

Aalborg University Missouri State University
114 PUBLICATIONS   530 CITATIONS    184 PUBLICATIONS   4,530 CITATIONS   

SEE PROFILE SEE PROFILE

Lakhmi C. Jain Gabriel Moyà Alcover

911 PUBLICATIONS   10,090 CITATIONS   

University of the Balearic Islands
39 PUBLICATIONS   378 CITATIONS   
SEE PROFILE
SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Multicenter, double-blind, parallel-group, randomized, controlled study to investigate the efficacy and safety of orally administered Bl 425 809 in patients with cognitive
impairment due to Alzheimer's Disease View project

HCI Ideas View project

All content following this page was uploaded by Bill Kapralos on 13 November 2014.

The user has requested enhancement of the downloaded file.

Chapter 14
An Overview of Virtual Simulation
and Serious Gaming for Surgical
Education and Training

Bill Kapralos, Fuad Moussa and Adam Dubrowski

Abstract The rising popularity of video games has seen a recent push towards the
application of serious games to medical education and training. With their ability
to engage players/learners for a specific purpose, serious games provide an
opportunity to acquire cognitive and technical surgical skills outside the operating
room thereby optimizing operating room exposure with live patients. However,
before the application of serious games for surgical education and training
becomes more widespread, there are a number of open questions and issues that
must be addressed including the relationship between fidelity, multi-modal cue
interaction, immersion, and knowledge transfer and retention. In this chapter we
begin with a brief overview of alternative medical/surgical educational methods,
followed by a discussion of serious games and their application to surgical edu-
cation, fidelity, multi-modal cue interaction and their role within a virtual simu-
lations/serious games. The chapter ends with a description of the serious games
surgical cognitive education and training framework (SCETF) and concluding
remarks.

Keywords Serious games  Virtual simulation  Surgical education

B. Kapralos (&)
Faculty of Business and Information Technology, Health Education Technology Research
Unit (HETRU), University of Ontario Institute of Technology,
Oshawa, ON L1H 7K4, Canada
e-mail: bill.kapralos@uoit.ca
F. Moussa
Division of Cardiac and Vascular Surgery, Schulich Heart Centre, Sunnybrook
Health Sciences Centre, Toronto, ON, Canada
e-mail: Fuad.Moussa@sunnybrook.ca
A. Dubrowski
Disciplines of Emergency Medicine and Pediatrics, Memorial University of Newfoundland,
St. John’s, Newfoundland, Canada
e-mail: adam.dubrowski@med.mun.ca

A. L. Brooks et al. (eds.), Technologies of Inclusive Well-Being, 289

Studies in Computational Intelligence 536, DOI: 10.1007/978-3-642-45432-5_14,
 Springer-Verlag Berlin Heidelberg 2014
290 B. Kapralos et al.

14.1 Introduction

The acquisition of medical skills in general and surgical skills (both cognitive and
technical) in particular, has historically been based on Halsted’s apprenticeship
model whereby the resident (trainee) acquires the required skills and knowledge in
the operating room [1]. However, the present era brings with it stresses on the
apprenticeship model for surgical training, leading to increased resource con-
sumption (e.g., monetary, faculty time, and time in the operating room), and
increased costs [2]. For example, a study conducted by Lavernia et al. [3]
examined the cost of performing a total knee arthroplasty surgical procedure
(replacement of the painful arthritic knee joint surfaces with metal and polyeth-
ylene components that serve to function in the way that bone and cartilage pre-
viously had), in teaching hospitals vs. non-teaching hospitals. They found that
patients who underwent surgery at a teaching hospital had higher associated
charges ($30,311.00 ± $3,325.00) and longer surgeries (190 ± 19 min) as
opposed to those who underwent similar surgery in a non-teaching hospital
($23,116.00 ± $3,341.00, 145 ± 29 min). They attribute this increase in resource
consumption to the hands-on approach required to train residents.
Compounding the problem is the growing trend towards decreasing resident
work hours in North America and globally [4]. Thus the available training time in
the operating room and consequently operative exposure, teaching, and feedback
are continuously shrinking [5]. Therefore, the available operative time must be
maximized in order to maintain a high level of surgical training. Although the
amount of repetition necessary to obtain the surgical competence required of
residents is still unclear, medical literature suggests that technical expertise is
acquired through years of practice [6] and indicates a positive correlation between
volume and patient outcome [7]. As a result, traditional educational methods in
surgery have come under increasing scrutiny [8] and as Murphy et al. [9] describe,
the current situation calls for a reassessment of medical education practises. It is
evident that given the increasing time constraints, trainees are under great pressure
to acquire complex surgical cognitive and technical skills. Therefore, efforts must
be made to optimize operative room exposure by devising training opportunities
using artificial settings before exposure to patients.

14.1.1 Alternative Educational Models

In accordance with new educational models, such as competency based curricular

approaches [10], new teaching modalities and technologies are necessary to augment
the traditional teaching practices in light of the shrinking operative time. Other
available alternative methods for surgical training include the use of animals,
cadavers, or plastic models; each option with its share of problems [11]. More spe-
cifically, the use of animals for medical education may be prohibited in some
14 An Overview of Virtual Simulation and Serious Gaming 291

countries, animal anatomy can vary greatly from humans, and there are ethical con-
cerns with the use of animals in medical research and education [12]. Cadavers cannot
be used multiple times, and plastic models don’t necessarily provide realistic visual
and haptic feedback [11]. In addition to availability issues, and changed tissue
behavior, cadavers are expensive (up to $5,000 each), require specialized facilities,
and disposal arrangements [12, 13]. Furthermore, animals and cadavers can be cost
prohibitive when considering extensive training [14], and ethical issues are being
raised with respect to practising surgical procedures on anesthetized humans and
animals [15]. Simulation (and virtual simulation in particular) offers a viable alter-
native to practice in an actual operating room, offering residents the opportunity to
train until they reach a specific competency level. Unlike working with live patients
(and animals), virtual simulation allows trainees to intentionally make and correct
mistakes [16]. In addition, according to Reiner and Harders [12], additional advan-
tages of virtual reality-based technologies include: allowing students to practice
independent of busy operating room schedules and patients, allow for a large number
of diverse anatomies and pathologies in a small period of time, allow for the training of
rare yet dangerous complications that a trainee may not otherwise have the oppor-
tunity to experience, allow for objective assessment, and can lead to reduced costs.
The rising popularity of video games has seen a recent push towards the
application of video game-based technologies to teaching and learning. Serious
games (that is, video games that are used for training, advertising, simulation, or
education [17]), provide a high level of interactivity not easily captured in tradi-
tional teaching/learning environments. In contrast to traditional teaching envi-
ronments where the teacher controls the learning (e.g., teacher-centered), serious
games and virtual simulations present a learner-centered approach to education, so
that the player controls the learning through interactivity thus allowing the player
to learn via an active, critical learning approach [18]. Game-based technologies
have also been used for many years as training simulators for vehicle control (e.g.,
flight simulators) and are growing in popularity within medical education
including surgery. Through game constructs, realistic situations can be simulated
to provide valuable experience to support discovery and exploration in a fun,
engaging, and cost-effective manner.
Although (virtual) simulations and serious games are similar (according to
Becker and Parker [19], all serious games (or game simulations as they refer to
them) are games, and all games are simulations [19]; see Fig. 14.1), and can employ
identical technologies (hardware and software). Being a video game, serious games
should strive to be fun and include some of the primary aspects of games including
challenge, risk, reward, and loss or more formally, as defined by Thiagarajin and
Stolovich [20], serious games should include the following five characteristics:
1. Conflict: can be described as challenge.
2. Constraints on a player’s behaviors: rules.
3. Closure: the game must come to an end.
4. Contrivance: all games are contrived situations.
5. Correspondence: designed to respond to some selected aspects of reality.
292 B. Kapralos et al.

Fig. 14.1 The relationship

between simulation, games,
and serious games (or
simulation games as referred
to by Becker and Parker
[19]). After Becker and
Parker [19]

14.1.2 Open Problems with Virtual Simulation and Serious

Games

Despite the benefits of virtual simulation and serious games, there are a number of
open, fundamental issues that must be addressed before they become more
widespread. Tashiro and Dunlap [21] developed a typology of serious games for
healthcare education and explored the strengths and limitations of serious games
for improving clinical judgment. They identified seven areas that require research
and improvements for the effective development of serious games: (1) disposition
to engage in learning, (2) impact of realism/fidelity on learning, (3) threshold for
learning, (4) process of cognitive development during knowledge gain, (5) stability
of knowledge gain (retention), (6) capacity for knowledge transfer to related
problems, and (7) disposition toward sensible action within clinical settings. Our
own work is focused on the impact of realism/fidelity and multi-modal interactions
on learning and for the scope of this paper, we will therefore focus on this problem
specifically.
Fidelity and Multi-Modal Interactions In the context of a simulation (and a
serious game), fidelity denotes the extent to which the appearance and/or behavior
of the simulation matches the appearance and behavior of the real system [22, 23].
Fidelity can be divided into two components: (1) psychological fidelity, and (2)
physical fidelity [24]. Psychological fidelity denotes the degree that the skills
inherent in the real task being simulated are captured within the simulation [24]
14 An Overview of Virtual Simulation and Serious Gaming 293

and may also include the degree of reality perceived by the user of the simulation
(the trainee) [25]. Physical fidelity covers the degree of similarity between the
training situation and the operational situation which is simulated [23, 24].
Physical fidelity can be further divided into equipment fidelity that denotes the
degree that the simulation replicates reality and environmental fidelity that denotes
the degree that the simulation replicates the sensory cues [24, 25].
Knowledge transfer can be defined as the application of knowledge, skills, and
attitudes acquired during training to the environment in which they are normally
used [26]. Ker and Bradley [24] describe twelve factors that most effectively
promote learning transfer. With respect to clinical training, one of these factors
includes the re-creation of the real clinical environment, which both aids in the
suspension of disbelief and in the transfer of competence to performance. This
effect is also consistent with a situated learning approach whereby the learning
environment is modeled in the context that the knowledge is expected to be
applied [27, 28]. Therefore, a serious game/virtual simulation should attempt to
remain faithful in its representation of the real environment (where multiple senses
are engaged simultaneously at any time). However, it is unclear just how close this
relationship needs to be. In other words, (1) how much fidelity is actually needed
to maximize transfer and retention? and (2) what effect do multi-modal interac-
tions have on knowledge transfer and retention? These questions have a number of
implications when considering that any training device-be it a virtual or physical
simulator-will never be able to completely replicate the real world, and in virtual
worlds we have (typically) eliminated one sensory modality-smell, and reduced or
restricted the haptic senses (touch and movement). Therefore, complete (perfect)
multi-sensory fidelity appears to be impossible to achieve, at least with our current
technology. Furthermore, it remains unclear if such a high level of fidelity is
actually needed for either enjoyment or knowledge transfer and retention, and
striving to reach higher levels of fidelity can also lead to increased computational
requirements (processing time). Moreover, despite the great computing hardware
advances, particularly with respect to graphics rendering, real-time high fidelity
audio and visual rendering particularly of complex environments, is still not
feasible [29]. In addition, striving for such high fidelity environments increases the
probability of lag and subsequent discomfort and simulator sickness [30]. Finally,
here fidelity has been explicitly defined (see above) but as Cook et al. [31]
describe, fidelity itself is complex, and encompasses various aspects of a simu-
lation activity including the sensory modalities (visual, auditory, olfactory and
haptics), learning objectives and task demands and therefore, referring to a sim-
ulation as high fidelity can convey diverse meanings.
294 B. Kapralos et al.

14.1.3 Paper Overview

In the remainder of this paper we present an overview of serious games with an

emphasis on medical/surgical education. We focus specifically on fidelity and
multi-modal interactions, a problem that we believe can have significant impli-
cations in the widespread use of serious games. We also outline some of our
current work in the serious games domain applied to surgical education and
training. More specifically, in Sect. 14.2, an overview of serious games and virtual
simulations in a surgical learning context is provided. Section 14.3 focuses on
fidelity, and multi-modal interactions while Sect. 14.4 introduces the serious game
surgical cognitive education and training framework (SCETF) being developed
both as a learning tool for surgical cognitive education and training and as a
research tool to examine the role of fidelity and multi-modal interactions on
learning are also provided. Finally, concluding remarks are provided in Sect. 14.5.

14.2 Virtual Simulation and Gaming in Surgical Education

Simulations range from de-contextualized bench models and virtual reality (VR)—
based environments, to high fidelity recreations of actual operating rooms [32]
whose fidelity is high enough to allow training that is equal to and at times better
than traditional methods [15]. Virtual reality has been widely and successfully
used for training and education in a variety of industries including nuclear, avi-
ation, military, and surgery for many years [15, 33]. One of the prevailing argu-
ments for using simulation in the learning process of trainees is their ability to
engage the trainee in the active accumulation of knowledge by doing. Focusing on
laparoscopic surgery education and training, according to Smith [15], the literature
supports four hypotheses with respect to the impact and acceptance of virtual
reality and game-based technologies:
1. Training in laparoscopic surgery can be accomplished at a lower cost using
virtual reality and game-based technologies than existing methods of training.
2. Virtual reality and game-based training environments provide better access to
representative patient symptoms and allow more repetitive practise that existing
training methods and approaches.
3. Virtual reality and game-based training environments can reduce the training
time required to achieve proficiency in laparoscopic procedures.
4. Virtual reality and game-based training can reduce the number of medical
errors caused by residents and surgeons learning to perform laparoscopic
procedures.

Smith [15] focused on laparoscopic surgery due to the similarities between this
type of surgery and virtual reality systems, and the surgical interface used during
laparoscopic surgery lends itself to virtual reality and gaming. However, this is not
14 An Overview of Virtual Simulation and Serious Gaming 295

to say that virtual reality and game-based training is only applicable to laparo-
scopic surgery. Rather, virtual reality and game-based technologies are becoming
more widely accepted methods of training within medical education curriculums in
general and the technological problems plaguing such systems in earlier years are
quickly being overcome [15]. The use of virtual reality in medical education in
general is one part of a change over in medical education that also includes the use
of mannequins, task trainers, and online learning modules [34]. In fact, Smith [15]
proposed a model of medical education where virtual reality and game-based
learning technologies are the next major transformation of the medical education
curriculum.
To date, simulation in the surgical domain has been primarily developed and
studied as an educational tool for the development of foundational skills (i.e., basic
technical skills and appropriate use of instruments). In addition, simulators can be
rather complex and very costly. For example, the vascular intervention system
training (VIST) simulator allows surgeons to practice laparoscopic surgical skills
while providing the context of a surgical procedure. The VIST simulator displays
photorealistic organs that can be cut and sutured and includes the simulation of
cauterization. However, the VIST simulator costs approximately $300,000 US
making it prohibitively expensive for many institutions [35].

14.2.1 Transfer of Skills to the Operating Room

Competent surgical performance requires mastery of not only technical skills but
cognitive skills as well (i.e., the capability of responding and adapting to the wide
range of contextual variations that may require adjustments to the standard
approach) [36]. As with the development of technical skills, cognitive judgment
takes practice to develop [6]. It has been suggested that cognitive skills training
could accelerate the understanding and planning of a particular procedure, leads to
a reduction in the training time required to become proficient with the procedure,
may provide greater meaning to the actions being practiced, and creates more
effective learning while making more efficient use of resources [6, 36]. It has also
been suggested that certain non-technical aspects of performance can enhance or,
if lacking, contribute to deterioration of surgeons’ technical performance [37]. The
addition of cognitive skills training to residents early on provides the opportunity
for residents to detect errors and this ultimately helps prevent errors in the oper-
ating room; cognitive skills training helps surgeons judge the correctness of their
own actions [36]. Finally, is has been suggested that cognitive skills training may
help accelerate the understanding and planning of a particular surgical procedure,
providing the surgeon with greater meaning to the actions being practised, and
reduce the overall training time required to become competent both cognitively
and technically [6].
296 B. Kapralos et al.

According to Fitts and Posner [38], learning is a sequential process and we

move through three distinct phases when learning a new skill. The three stages are
as follows:
1. Cognitive phase: Identification and development of the component parts of the
skill—involves formation of a mental picture of the skill.
2. Associative phase: Linking the component parts into a smooth action—involves
practicing the skill and using feedback to perfect the skill.
3. Autonomous phase: Developing the learned skill so that it becomes auto-
matic—involves little or no conscious thought or attention whilst performing
the skill—not all performers reach this stage.
Serious games and virtual simulations particularly lend themselves to the
cognitive phase, allowing trainees to focus on the understanding and planning of a
particular surgical procedure. Focusing on the cognitive aspects of a procedure can
also be cost-effective given that potentially specialized (and costly) equipment
(e.g., haptic devices), typical in technical skills training are not generally required.
Furthermore, such virtual simulations/serious games can be developed to run on
common computing and mobile platforms ensuring that the applications are
available to the trainees/residents outside of the regular training/educational set-
ting, at all times.
Serious games provide an opportunity to acquire cognitive and technical sur-
gical skills outside the operating room thereby optimizing operating room expo-
sure with live patients, and this is particularly so when considering novice trainees
in the cognitive stage of motor skills acquisition where the majority of errors
occur. In addition to their use in training and as part of a curriculum, virtual
simulations and serious games can be used by surgeons to rehearse/practise an
operation (particularly complex cases) that they are about to perform using data
that simulates the patient that they will be operating on [34]. For example, a
research study (by researchers with Beth Israel and the National Institute on Media
and the Family at Iowa State University), that investigated whether good video
game skills translate into surgical skills demonstrated that laparoscopic surgeons
who played video games at least 3 h each week made approximately 37 % less
mistakes in laparoscopic surgery and performed the task 27 % faster than their
counterparts who did not play video games [39]. Further work remains, but in
addition to the use of serious games, video game playing may also be an integral
part of the training program of future surgeons. As Dr. Paul J. Lynch, a Beth Israel
anesthesiologist who has studied the effects of video games for years comments
[40]: ‘‘The study landmarks the arrival of Generation X into medicine. We grow
up with computers, with PDAs, with video games systems, with the Internet, with
handheld video games, with cable TV, with remote controls. We’ve grown up
saturated in this technology era that we are in and now we are bringing these skills
into the medical profession.’’ Finally, according to Professor Pamela Andreatta,
the Director of the Clinical Simulation Center at the University of Michigan
Medical School in Ann Arbor [34], although there seems to be obvious patient
safety benefits to training outside of the patient setting and preliminary data
14 An Overview of Virtual Simulation and Serious Gaming 297

suggests that simulation is effective in the early learning phase of clinical skills
acquisition, a general conclusion stating that simulation (and serious games) has a
significant impact on clinical applications cannot be made given the lack of data to
date further research is required.

14.3 Serious Games: Fidelity and Multi-Modal

Interactions

In the real world, our senses are constantly exposed to stimuli from multiple
sensory modalities (visual, auditory, vestibular, olfactory, and haptic), and
although the process is not exactly understood, we are able to integrate/process this
multisensory and acquire knowledge of multisensory objects [41]. As described
previously, it is currently beyond our capability to faithfully account for all of the
human senses within a virtual simulation/serious game, and although the emphasis
of (virtual) simulations in general (including serious games) is on the visuals/
graphics [42], visuals within such environments are rarely presented in silence but
rather, include sound of some type. In the real world, visuals and auditory stimuli
influence one another.
Various studies have examined the perceptual aspects of audio-visual cue
interaction, and it has been shown that sound can potentially attract part of the user’s
attention away from the visual stimuli and lead to a reduced cognitive processing of
the visual cues [43]. Bonneel et al. [44] examined the influence of the level of detail
of auditory and visual stimuli in the perception of audiovisual material rendering
quality. In each trial of their experiment, participants were presented with two
sequences, each sequence of an object falling on a table, bouncing twice and pro-
ducing audible bounce sounds. One of the sequences was a reference (highest
quality with respect to sound and graphics) while for the other sequence, auditory
and visual levels of detail varied. Auditory level of detail was defined with respect to
modal synthesis (a physical-based model of a vibrating object), while visual level of
detail was defined with respect to the bidirectional reflection distribution function
(BRDF) which describes the reflection, absorption, and transmission of light at the
surface of a material [45]. The participants’ task for each trial was to rate, on a scale
of 0–100, the similarity of the falling objects in the two sequences. The authors
observed significant interactions between visual and auditory level of details and the
perceived material quality. In other words, visual level of detail was perceived to be
higher as the auditory level of detail was increased.
Mastoropoulou et al. [43] examined the influence of sound effects on the per-
ception of motion smoothness within an animation and more specifically, on the
perception of frame-rate. Their study involved forty participants that viewed pairs of
computer generated walkthrough animations at five different frame-rates. The
visuals were consistent across the animation pairs although the pairs differed with
respect to sound; one contained sound effects while the other did not (it was silent).
The participants’ task was to choose which animation had a smoother motion.
298 B. Kapralos et al.

There was a significant effect of sound on perceived smoothness and more specif-
ically, sound attracted a viewer’s attention from the visuals leading to a greater
difficulty in distinguishing smoothness variations between animations containing
sound cues displayed at different rates, than between silent animations [43]. It was
inferred that sound stimuli attract part of the viewer’s attention away from any visual
defects inherent in low frame-rates [43]. Similarly, Hulusic et al. [46] examined the
interaction of sound with visuals for the purpose of reducing computational
requirements of visual rendering with the use of motion-related sound effects. They
found that such sound effects allowed slow animations to be perceived as smoother
than fast animations and that the addition of footstep sound effects to walking
(visual) animations increased the animation smoothness perception. Hulusic et al.
[46] conclude that for certain conditions, the rendering rate can be reduced by
incorporating the appropriate sound effects, leading to a reduction in the required
computation without the viewer being aware of this reduction.
Greater details regarding the influence of sound over visual rendering is pro-
vided by Hulusic et al. [47] while an overview of ‘‘crossmodal influences on visual
perception’’ is provided by Shams and Kim [48].

14.4 The Serious Games Surgical Cognitive Education

and Training Framework

Given the strain on resources associated with the current master-apprenticeship

surgical training model, and the importance of cognitive skills training, develop-
ment of a multi-modal, serious game surgical cognitive education and training
framework (SCETF) has recently begun. Domain-specific surgical modules can
then be built on top of the existing framework, utilizing common simulation
elements/assets and ultimately reducing development costs. The SCETF focus is
on the cognitive components of a surgical procedure and more specifically, the
proper identification of the sequence of steps comprising a procedure, the instru-
ments and anatomical/physiological knowledge required for performing each step,
and the ability to respond to unexpected events while carrying out the procedure.
By clearly understanding the steps of a procedure and the surgical knowledge that
goes along with each step, trainees are able to focus solely on the technical aspect
of the procedure. In other words, with respect to the three stages to skills acqui-
sition described by Fitts and Posner [38], trainees can transition from the cognitive
through the integrative, and perhaps into the automatic stage) in higher fidelity
models or in the operating room thus making more efficient use of the limited
available resources. The SCETF is also being developed as a research tool where
various simulation parameters (e.g., levels of audio/visual fidelity) can be easily
adjusted allowing for the controlled testing of such factors on knowledge transfer
and retention and this will ultimately lead to more effective serious games. The
SCETF consists of graphical and spatial sound rendering engines and various other
14 An Overview of Virtual Simulation and Serious Gaming 299

Fig. 14.2 Sample SCETF

screenshot

components that are common (generic) to all serious games including a scenario
editor (currently being developed) that allows users of the module (educators/
instructors) to create and/or modify/edit specific scenarios using a graphical-based
user interface. The scenario editor that allows a scenario to be easily developed by
clicking and dragging various interface components in a ‘‘what you see is what you
get’’ (WSYWIG), manner.
Within a given module, trainees take on the role of the surgeon, viewing the
environment through their avatar in a first-person perspective and therefore, only
their hand is visible (see Fig. 14.2). In each module, the task of the trainee is to
complete the surgical procedure following the appropriate steps and choosing the
correct tools for each step. Along the way, complications can arise that will require
some action from the trainee. These complications will appear in the form of visual
or auditory cues adapted according to predefined features of the simulated surgical
scenario. Several other non-player characters (NPCs) also appear in the scene
including the patient (lying on a bed), assistants, and nurses. The SCETF includes
networking capabilities to allow these NPCs to be controlled by other users and
provide an entire surgical team the opportunity to practice remotely and allow for
interprofessional education. The trainee (user) can move and rotate the ‘‘camera’’
using the mouse in a first-person manner thus allowing them to move within the
scene. A cursor appears on the screen and the trainee can use this cursor to point at
specific objects and locations in the scene.
A SCETF module for the off-pump coronary artery bypass (OPCAB) grafting
cardiac surgical procedure is currently being developed [49]. The OPCAB pro-
cedure itself is complex and technically challenging and it has been suggested that
appropriate training be provided before being performed on patients [50]. We
hypothesize that by learning the OPCAB procedure in a first-person-shooter
gaming environment, trainees will have a much better understanding of the pro-
cedure than by traditional learning modalities and therefore, we anticipate that a
serious game for OPCAB training will be a beneficial educational tool.
300 B. Kapralos et al.

14.4.1 Graphical Rendering

The 3D graphics rendering engine was developed completely in-house and is

based on the C++ programming language and the OpenGL 3D graphics API. Real-
time rendering is accomplished using the graphics processing unit (GPU) via the
OpenGL shading language (GLSL). The SCETF utilizes GPU-based effects such
as outer glow (used to indicate a selectable object), reflection mapping, bloom
filtering (to provide more realistic lighting effects particularly when considering
objects in front of a light source), and various effects to generate realistic metal
effects given the widespread use of metal (stainless steel) in an operating room. An
example of the outer glow effect is provided in Fig. 14.2. In this example, the outer
glow surrounds the outline of the operating room nurse after the end-user (taking
on the role of the surgeon in a first-person perspective), moves the cursor over the
nurse to indicate that they are able to interact with the nurse. Interaction is initiated
by clicking the left mouse button. Finally, the SCETF also supports stereoscopic
3D viewing using active stereo technologies were the user wears a pair of LCD
shutter glasses that are synchronized with the display refresh rate. Stereoscopic 3D
viewing has been linked to increased player engagement in the gameplay [51], and
increased engagement in an educational setting has been linked to higher academic
achievement. With respect to medical education and training, it has been suggested
that the use of stereoscopic 3D can (1) lead to improved understanding of ana-
tomical relationships and pathology, (2) improve the quality of the student’s
learning experience, and (3) create more life-like training simulations [52]. Fur-
thermore, stereoscopic 3D provides the ability to establish foreground and back-
ground information [53], which can be useful in a variety of training situations.
However, the technology can also be problematic, as it may lead to users hyper-
focusing on the foreground while ignoring potentially important information
within the periphery [53]. Therefore, the usefulness of stereoscopic 3D for training
simulations and serious games needs further study.

14.4.2 Sound Rendering

Sound plays an important role in the operating room. For example, with respect to
the OPCAB procedure, the surgeon listens to the auditory component of the
electrocardiogram; they can hear the onset of changes such as bradycardia
(slowing of the heart rate), tachycardia (speeding up of the heart rate), decreasing
blood oxygen saturation, and hemodynamic changes. As a result, appropriate
sound modeling should be included within a virtual operating room intended for
education and training. The addition of realistic spatial sound (that is, the simu-
lation of realistic ‘‘spatial’’ auditory cues within a virtual environment such that it
allows users to perceive the position of a sound source at an arbitrary position in
three-dimensional space; see [54]), within virtual environments in general is
14 An Overview of Virtual Simulation and Serious Gaming 301

beneficial for a number of reasons. More specifically, spatial sound can add a new
layer of realism [55], contributes to a greater sense of presence, or immersion [56],
can improve task performance [57], convey information that would otherwise be
difficult to convey using other modalities (e.g., vision) [57], and improve navi-
gation speed and accuracy [58]. In addition, when applied to virtual environments
for medical training, sound can play an important role by aiding in the acquisition
of enhanced skills and dexterity, and lead to increased effectiveness [12]. In
addition, although greater studies are required, it has been suggested that sound
can influence higher-level cognitive processes [59].
The SCETF supports high fidelity sound to allow for the inclusion of spatial
sounds such as those described above. The SCETF also supports spatial (3D)
sound rendering including reverberation and occlusion/diffraction modeling using
novel GPU-based methods that approximate such effects at interactive rates [60,
61]. The system also supports head-related transfer functions (HRTFs) which
describe the individualized location dependent filtering of a sound by the listener’s
head, shoulders, upper torso, and most notably, the pinna [42]. HRTF filtering is
accomplished using GPU-based convolution ensuring interactive frame-rates [62].
Spatial sound by default is not activated; it is an option that can be chosen during
start-up. By default, sounds are non-spatialized and output in a traditional stereo
format. Although each module has its corresponding sounds (e.g., background
sound, sound effects, and dialogue), the user is also provided the opportunity to
provide their own sounds during start-up (e.g., the user can provide their own
background sound or can choose to have no sound output at all).

14.4.3 Multi-cue Interaction and Cue Fidelity

As previously described, the SCETF is being developed as a research tool to

enable the investigation of the effect of multi-modal cue interaction on knowledge
transfer and retention. Currently, the SCETF supports the alteration of audio and
visual fidelity and the interaction of audio and visual cues particularly if they are
incongruent and mis-matched (i.e., high quality audio and poor quality visuals and
vice versa). With respect to audio fidelity currently, the following options are
supported: (1) spatial sound vs. non-spatial sound, (2) no sound at all (background
sound and/or all sound effects are turned off), (3) adjustable quantization levels, (4)
addition of white noise to background sounds and/or sound effects, and (5)
adjustment of loudness and dynamic range. Such effects (and their corresponding
settings) are chosen during start-up and cannot be adjusted dynamically. Visual
(graphical) fidelity ranges from high to low quality, defined with respect to
polygon count, and resolution (both texture resolution and overall resolution).
These particular fidelity measures cannot be adjusted dynamically but rather, their
settings must be specified during an initialization phase at start-up. The SCETF
also provides for the dynamic adjustment of visual fidelity through various
graphical filtering effects implemented using the graphics processing unit (GPU).
302 B. Kapralos et al.

The degree of filtering introduced by each of these effects can be dynamically

adjusted via a slider control and these effects can also be combined dynamically.
For example, the scene can be blurred using the blurring filter and noise can also
be added with the noise filer.
In addition to its use as a training tool the OPCAB module that is currently
being developed will also serve as a testbed to allow for the methodical investi-
gation of the effect that varying audio/visual simulation fidelity and the interaction
of audio/visual cues have on knowledge/skills/behaviours (KSB) transfer and
retention. Furthermore, the OPCAB module will be used to measure the feasibility/
usefulness of using serious games as a tool for assessing/screening cognitive skills
(e.g., surgical steps and knowledge) in surgical training.

14.5 Conclusions

Surgical training has predominantly taken place in operating rooms placing a drain
on the limited available operating room resources. Simulations, both physical and
virtual, have been effectively used to complement residents’ training and educa-
tion. Serious games, or the use of video game-based technologies for applications
whose primary purpose is other than entertainment, are becoming very popular in a
variety of applications including medical education in general. This popularity
stems in part from the current generation of tech-savvy learners who play games.
In addition to promoting learning via interaction, serious games allow users to
experience situations that are difficult (even impossible), to achieve in reality and
they support the development of various skills including analytical and spatial,
strategic, recollection, and psychomotor skills as well as visual selective attention.
Despite the growing popularity of serious games and their inherent benefits, before
their use becomes more widespread, a number of open problems must be
addressed. Here, in this paper, we focused on the problem multi-modal interaction
and fidelity on learning. Although great progress has been with respect to both of
these problems, many open issues remain and plenty of work remains. In this paper
we have also described the serious games surgical cognitive education and training
framework that is currently being developed specifically for cognitive surgical
skills training. Domain-specific surgical modules can then be built on top of the
existing framework utilizing common simulation elements and assets and ulti-
mately reducing development time and costs. The SCETF is also being developed
as a research tool where various simulation parameters such as levels of audio and
visual fidelity, can be easily adjusted allowing for the controlled testing of these
factors on knowledge transfer and retention. In addition to examining the effect
multi-modal interactions may have on knowledge transfer and retention, the
SCETF also allows us to methodically investigate perceptual-based rendering and
more specifically, the role of sound with respect to visual quality perception which
may ultimately lead to reduced rendering (computational) requirements and ulti-
mately allow for more effective virtual simulations and serious games.
14 An Overview of Virtual Simulation and Serious Gaming 303

As serious gaming becomes more widespread, care must be taken to ensure that
they are properly designed to meet their intended goals. With traditional enter-
tainment game development, designers/developers start fresh with a blank slate,
and are primarily concerned with creating an engaging gameplay experience that
will keep the players playing the game; to improve gameplay and engagement,
they are free to modify the design of the game throughout the entire design and
development process [19, 63]. However, serious games designers/developers are
not afforded this luxury but rather, must strictly adhere to the content/knowledge
base while ensuring that their end product is not only fun and engaging, but is also
an effective teaching tool [19, 63]. In addition to knowledge and expertise in game
design and development, serious games designers/developers must therefore also
be knowledgeable in the specific content area covered by the serious game and
possess some knowledge in teaching methods and instructional design in partic-
ular. In other words, the development of effective serious games is not a trivial task
and knowledge in game design solely is not sufficient to develop an effective
serious game. Serious games development is an interdisciplinary process, bringing
together experts from a variety of fields including game design and development
and although serious games designers are not expected to be experts in instruc-
tional design and the specific content area, possessing some knowledge in these
areas will, at the very least, promote effective communication between the inter-
disciplinary team members.

Acknowledgments The financial support of the Social Sciences and Humanities Research
Council of Canada (SSHRC), in support of the iMMERSE project, and the Canadian Network of
Centres of Excellence (NCE) in support of the Graphics, Animation, and New Media (GRAND)
initiative is gratefully acknowledged.

References

1. Halsted, W.S.: The training of the surgeon. Johns Hopkins Hosp. Bull. 15(162), 267–275
(1904)
2. Reznick, R.K.: Teaching and testing technical skills. Am. J. Surg. 165(3), 358–361 (1993)
3. Lavernia, C.J., Sierra, R.J., Hernandez, R.A.: The cost of teaching total knee arthroplasty
surgery to orthopaedic surgery residents. Clin. Orthop. Relat. Res. 380, 99–107 (2000)
4. Zuckerman, J., Kubiak, E., Immerman, I., DiCesare, P.: The early effects of code 405 work
rules on the attitudes of orthopaedic residents and attending surgeons. J. Bone Joint Surg.
87(A(4)), 903–908 (2005)
5. Weatherby, B.A., Rudd, J.N., Ervin, T.B., Staff, P.R.: The effect of resident work hour
regulations on orthopaedic surgical education. J. Surg. Orthop. Adv. 16(1), 19–22 (2007)
6. Ericsson, K.A., Krampe, R., Tesch-Romer, C.: The role of deliberate practice in the
acquisition of expert performance. Psychol. Rev. 100(3), 363–406 (1993)
7. Halm, E.A., Lee, C., Chassin, M.R.: Is volume related to outcome in healthcare? A
systematic review and methodological critique of the literature. Ann. Intern. Med. 137(6),
511–520 (2002)
8. Gorman, P.J., Meier, A.H., Rawn, C., Krummel, T.M.: The future of medical education is no
longer blood and guts, it is bits and bytes. Am. J. Surg. 180(5), 353–356 (2000)
304 B. Kapralos et al.

9. Murphy, J.G., Torsherand, L.C., Dunn, W.F.: Simulation medicine in intensive care and
coronary care education. J. Crit. Care 22(1), 51–55 (2007)
10. Sonnadara, R.R., Garbedian, S., Safir, O., Nousiainen, M., Alman, B., Ferguson, P., Kraemer,
W., Reznick, R.: Orthopaedic boot camp ii: examining the retention rates of an intensive
surgical skills course. Surgery 151(6), 803–807 (2012)
11. Heng, P.A., Cheng, C.Y., Wong, T.T., Xu, Y., Chui, Y.P., Tso, S.K.: A virtual reality training
system for knee arthroscopic surgery. IEEE Trans. Inf. Technol. Biomed. 8(2), 217–227
(2004)
12. Riener, R., Harders, M.: Psychology Applied to Work. Springer, London (2012)
13. Meyerson, S.L., LoCascio, F., Baldersonand, S.S., D’Amico, T.A.: An inexpensive,
reproducible tissue simulator for teaching thoracoscopic lobectomy. Ann. Thorac. Surg.
89(2), 594–597 (2010)
14. Bridges, M., Diamond, D.L.: The financial impact of teaching surgical residents in the
operating room. Am. J. Surg. 177(1), 28–32 (1999)
15. Smith, R.: Game Technology in Medical Education: An Inquiry into the Effectiveness of New
Tools. Modelbenders LLC, USA (2009)
16. Ziv, A., Ben-David, S., Ziv, M.: Simulation based medical education: an opportunity to learn
from errors. Med. Teach. 27(3), 193–199 (2005)
17. Susi, T., Johannesson, M., Backlund, P.: Serious games—an overview. Technical Report HS-
IKI-TR-07-001, School of Humanities and Informatics, University of Skovde, Sweden, Feb
2007
18. Stapleton, A.J.: Serious games: serious opportunities. In: Proceedings of the 2004 Australian
Game Developers’ Conference, pp. 1–6. Melbourne (2004)
19. Becker, K., Parker, J.: The Guide to Computer Simulations and Games. Wiley, Indianapolis
(2012)
20. Thiagarajan, S., Stolovitch, H.D.: Instructional Simulation Games. Educational Technology
Publications, Englewood Cliffs (1978)
21. Tashiro, J., Dunlap, D.: The impact of realism on learning engagement in educational games.
In: Proceedings of ACM Future Play 2007, pp. 113–120. Toronto (2007)
22. Farmer, E., von Rooij, J., Riemersma, J., Joma, P., Morall, J.: Handbook of Simulator Based
Training. Ashgate Publishing Limited, Surrey, UK (1999)
23. Hays, R.T., Singer, M.: Simulation Fidelity in Training System Design. Springer, New York
(1989)
24. Ker, J., Bradley, P.: Simulation in medical education. In: Swanwick, T. (ed.) Understanding
Medical Education: Evidence, Theory and Practice, pp. 164–180. Wiley, New York (2010)
25. Rehmannand, A.J., Mitman, R.D., Reynolds, M.C.: A handbook of flight simulation fidelity
requirements of human factors research. Technical report, Crew System Ergonomics
Information Analysis Center, Wright-Patterson Air Force Base, Dayton, 1995
26. Muchinsky, P.M.: Psychology Applied, to Work. Wadsworth (1999)
27. Brown, J.S., Collins, A., Duguid, S.: Situated cognition and the culture of learning. Educ.
Researcher 18(1), 32–42 (1989)
28. Dalgarno, B., Lee, M.J.W.: What are the learning affordances of 3-D virtual environments?
Br. J. Educ. Technol. 41(1), 10–32 (2010)
29. Hulusic, V., Aranh, M., Chalmers, A.: The influence of cross-modal interaction on perceived
rendering quality thresholds. In: Proceedings of the 16th International Conference in Central
Europe on Computer Graphics, Visualization and Computer Vision, pp. 41–48. Plzen–Bory,
Czech Republic, 4–7 Feb 2008
30. Blascovich, J., Bailenson, J.: Infinite Reality. Harper Collins, New York (2011)
31. Cook, D.A., Hamstra, S.J., Brydges, R., Zendejas, B., Szostek, J.H., Wang, A.T., Erwin, P.J.,
Hatala, R.: Comparative effectiveness of instructional design features in simulation-based
education: systematic review and meta-analysis. Med. Teach. 35(1), e867–e898 (2013)
32. Kneebone, R.L.: Practice, rehearsal, and performance: an approach for simulation-based
surgical and procedure training. J. Am. Med. Assoc. 302(12), 1336–1338 (2009)
14 An Overview of Virtual Simulation and Serious Gaming 305

33. Ziv, A., Wolpe, P.R., Small, S.D., Glick, S.: Simulation-based medical education: an ethical
imperative. Acad. Med. 78(8), 19–22 (2003)
34. Raths, D.: Virtual reality in the OR. Training Dev. Mag. 6(8), 36–40 (2006)
35. Gallagher, A., Ritter, M., Champion, H., Higgins, G., Fried, M., Moses, G., Smith, D., Satava,
R.: Virtual reality simulation for the operating room, proficiency-based training as a
paradigm shift in surgical skills training. Ann. Surg. 241(2), 364–372 (2005)
36. Kohls-Gatzoulis, J.A., Regehr, G., Hutchison, C.: Teaching cognitive skills improves
learning in surgical skills courses: a blinded, prospective, randomized study. Can. J. Surg.
47(4), 277–283 (2004)
37. Hull, L., Arora, S., Aggarwal, R., Darzi, A., Vincent, C., Sevdalis, N.: The impact of
nontechnical skills on technical performance in surgery: a systematic review. J. Am. Coll.
Surg. 214(4), 214–230 (2012)
38. Fitts, P.M., Posner, M.I.: Human Performance. Brooks and Cole, Oxford, England (1967)
39. Rosser Jr, J.C., Lynch, P.J., Haskamp, L.A., Gentile, D.A., Yalif, A.: The impact of video
games in surgical training. Arch. Surg. 142(2), 181–186 (2007)
40. CBS News. Video gamers make good surgeons. http://www.cbsnews.com/2100-204_162-
610601.htm/. 9 Feb 2009
41. Seitz, A.R., van Wassenhove, V., Shams, L.: Simultaneous and independent acquisition of
multisensory and unisensory associations. Perception 36(10), 1445–1453 (2007)
42. Carlile, S.: Virtual Auditory Space: Generation and Application. R.G. Landes, Austin TX,
USA (1996)
43. Mastoropoulou, G., Debattista, K., Chalmers, A., Troscianco, T.: The influence of sound
effects on the perceived smoothness of rendered animations. In: Proceedings of the
Symposium on Applied Perception in Graphics and Visualization 2005, pp. 9–15. La Coruna,
Spain, 26–28 Aug 2005
44. Bonneel, N., Suied, C., Viaud-Delmon, I., Drettakis, G.: Bimodal perception of audio-visual
material properties for virtual environments. ACM Trans. Appl. Percept. 7(1), 1–16 (2010)
45. Angel, E.: Interactive Computer Graphics: A Top-Down Approach Using OpenGL. TPearson
Education Inc., Boston (2008)
46. Hulusic, V., Debattista, K., Aggarwal, V., Chalmers, A.: Maintaining frame rate perception in
interactive environments by exploiting audio-visual cross-modal interaction. Vis. Comput.
27(1), 57–66 (2011)
47. Hulusic, V., Harvey, C., Debattista, K., Tsingos, N., Walker, S., Howard, D., Chalmers, A.:
Acoustic rendering and auditory-visual cross-modal perception and interaction. Comput.
Graph. Forum 31(1), 102–131 (2012)
48. Shams, L., Kim, R.: Crossmodal influences on visual perception. Phys. Life Rev. 7(3),
295–298 (2010)
49. Sabri, H., Cowan, B., Kapralos, B., Moussa, F., Cristancho, S., Dubrowski, A.: Off- pump
coronary artery bypass surgery procedure training meets serious games. In: Proceedings of
the International Symposium on Haptic Audio-Visual Environments and Games,
pp. 123–127. Phoenix, 16–17 Oct 2010
50. Fann, J.I., Caffarelli, A.D., Georgette, G., Howard, S.K., Gaba, D.M., Young-blood, P.,
Mitchell, R.S., Burdon, T.A.: Improvement in coronary anastomosis with cardiac surgery
simulation. J. Thorac. Cardiovasc. Surg. 136(6), 1486–1491 (2008)
51. Hogue, A., Kapralosand, B., Zerebeckiand, C., Tawadrous, M., Stanfield, B., Hogue, U.:
Stereoscopic 3d video games and their effects on engagement. In: Proceedings of the
Stereoscopic Displays and Applications XXIII, pp. 828816–828816-7. Burlingame, CA,
23–25 Jan 2012
52. UCSF.: Medical education: Potential contributions of stereoscopic 3D. School of Medicine,
Office of Educational Technology, University of California San Francisco (2013)
53. Hollander, A., Roseand, H., Kollin, J., Moss, W.: Attack of the s. Mutans!: a stereoscopic-3d
multiplayer direct-manipulation behavior-modification serious game for improving oral
health in pre-teens. In: Proceedings of the Stereoscopic Displays and Applications XXIII,
pp. 828816–828816-7. Burlingame, CA, 23–25 Jan 2012
 306 B. Kapralos et al.

54. Kapralos, B., Jenkin, M., Milio, E.: Virtual audio systems. Presence: Teleoperators Virtual
Env. 17(6), 527–549 (2008)
55. Bonneel, N., Suied, C., Viaud-Delmon, I., Drettakis, G.: Inter-active sound propagation using
compact acoustic transfer operators. ACM Trans. Graph. 31, 7 (2012)
56. Nordahl, R., Turchet, L., Serafin, S.: Sound synthesis and evaluation of interactive footsteps
and environmental sounds rendering for virtual reality applications. IEEE Trans.
Visualizations Comput. Graph. 17(1), 1234–1244 (2011)
57. Zhou, Z.Y., Cheok, A.D., Qiu, Y., Yang, X.: The role of 3-d sound in human reaction and
performance in augmented reality environments. IEEE Trans. Syst. Man Cybern.—Part A:
Syst. Hum. 37, 262–272 (2007)
58. Makino, H., Ishii, I., Nakashizuka, M.: Development of navigation system for the blind using
GPS and mobile phone communication. In: 18th Annual Meeting of the IEEE Engineering in
Medicine and Biology Society, pp. 506–507. Amsterdam, the Netherlands, 31 Oct–3 Nov
2005
59. Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J.:
Sonification Report: Status of the Field and Research Agenda. Technical report, Department
of Psychology, University of Nebraska—Lincoln, Lincoln (2010)
60. Cowan, B., Kapralos, B.: A gpu-based method to approximate acoustical reflectivity.
J. Graph. GPU Game Tools 15(4), 210–215 (2011)
61. Cowan, B., Kapralos, B.: Gpu-based real-time acoustical occlusion modeling. Virtual Reality
14(3), 183–196 (2010)
62. Cowan, B., Kapralos, B.: Gpu-based one-dimensional convolution for real-time spatial sound
generation. Loading Feature Issue: FuturePlay 2008 Ed. 3(9) (2009)
63. Iuppa, N., Borst, T.: End-to-End Game Development: Creating Independent Serious Games
and Simulations from Start to Finish. Focal Proess, Oxford (2010)

View publication stats