Focus Theme – Original Articles
Learning to Detect Cognitive
Impairment through Digital Games
and Machine Learning Techniques*
A Preliminary Study
Sonia Valladares-Rodriguez1; Roberto Pérez-Rodriguez1; J. Manuel Fernandez-Iglesias1;
Luis E. Anido-Rifón1; David Facal2; Carlos Rivas-Costa1
1Department of Telematic Engineering, University of Vigo, Vigo, Pontevedra, Spain;
2
Department of Developmental Psychology, Facultade de Psicoloxia, Universidade de Santiago de Compostela,
Santiago de Compostela, Spain
Keywords
Alzheimer’s disease, early detection, machine
learning, data analytics, serious games,
Classification and Regression Trees, logistic
regression, random forest, Support Vector
Machine
Summary
Objective: Alzheimer’s disease (AD) is one of
the most prevalent diseases among the adult
population. The early detection of Mild Cog-
nitive Impairment (MCI), which may trigger
AD, is essential to slow down the cognitive
decline process.
Methods: This paper presents a suit of seri-
ous games that aims at detecting AD and
MCI overcoming the limitations of traditional
tests, as they are time-consuming, affected
by confounding factors that distort the result
and usually administered when symptoms
are evident and it is too late for preventive
measures. The battery, named Panoramix, as-
sesses the main early cognitive markers (i.e.,
memory, executive functions, attention and
gnosias). Regarding its validation, it has been
tested with a cohort study of 16 seniors, in-
cluding AD, MCI and healthy individuals.
Correspondence to:
Sonia Valladares-Rodriguez
University of Vigo
Department of Telematic Engineering
Campus Lagoas-Marcosende Vigo, Galicia Vigo
Vigo Pontevedra 36310
Spain
E-mail: soniavr@det.uvigo.es
* Supplementary material published on our
website https://doi.org/10.3414/ME17-02-0011
© Georg Thieme Verlag KG 2018
Results: This first pilot study offered initial
evidence about psychometric validity, and
more specifically about construct, criterion
and external validity. After an analysis using
machine learning techniques, findings show
a promising 100% rate of success in classifi-
cation abilities using a subset of three games
in the battery. Thus, results are encouraging
as all healthy subjects were correctly dis-
criminated from those already suffering AD
or MCI.
Conclusions: The solid potential of digital
serious games and machine learning for the
early detection of dementia processes is
demonstrated. Such a promising perform-
ance encourages further research to event-
ually introduce this technique for the clinical
diagnosis of cognitive impairment.
Methods Inf Med 2018; 57: 197–207
https://doi.org/10.3414/ME17-02-0011
received: July 28, 2017
accepted: July 6, 2018
Funding
The present work was partially funded by the govern-
ment of Galicia (Xunta de Galicia, Spain), which cover-
ed the travel expenses to participants‘ homes during
the pilot study (grant # 2016/236), and also by ‘Rede
Galega de Investigación en Demencias’ (IN607C-
2017/02) funded by Axencia Galega de Innovación
GAIN – Xunta de Galicia.
197
Pervasive Health
1. Introduction
Cognitive functions make it possible to
Downloaded by: Trinity College Dublin. Copyrighted material.
navigate the world around us, actively con-
trolling the reception, selection, processing,
storage, elaboration and retrieval of in-
formation. Presently, their evaluation is
supported by validated tests or neuropsy-
chological batteries, which will be referred
as “classical tests” [1] in the rest of this
paper. The usual format for this type of
tests consists of a collection of tasks or ex-
ercises to be solved using pen and paper. At
the end of the test, a measurable outcome
or score on the subject’s cognitive status
can be computed from the subject’s re-
sponses.
Classical tests are time consuming as
they require a personalized and compre-
hensive attention from the professionals
responsible for administering such tests.
Besides, some subjects perceive classical
tests as intrusive tools, and more specifi-
cally as an exam of their intelligence, which
may influence their motivation, alertness
or effort, and thus influence the outcome.
In addition, classical tests are dependent on
confounder variables [2] that may alter the
final result. Age, educational level [3] or
geo-cultural aspects may have relevant im-
pact on the outcomes. Finally, they are ad-
ministered when the symptoms of cogni-
tive decline are evident for the person or
their relatives [4] and too late for efficient
rehabilitation or palliative measures.
Alternatives to classical tests are there-
fore very much in demand. The introduc-
tion of information and communication
technologies (ICT) for neuropsychological
Methods Inf Med 4/2018
 198 S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
evaluation brings some relevant benefits
[5], such as a better standardization of ad-
ministration; increased accuracy of timing
presentation and response latencies; an ea-
Pervasive Health
sier administration and collection of data;
the possibility of capturing and monitoring
dynamic indicators (e.g., cognitive markers
such as reaction time latency[6] or other
not cognitive indicators such as gait pat-
terns[7]); and better randomization of the
presentation of stimuli through repeated
administrations.
A promising approach is the use of seri-
ous games. They are “games that do not
have enjoyment, entertainment or fun as
their primary purpose” [8]. They are being
introduced in many application areas, such
as education, rehabilitation, or military
training among many others. Serious
games include a playful component as a
motivation element, and have a non rec-
reational aim or motivation such as learn-
ing, evaluation or training. Moreover, for
older adults playing serious games, differ-
ent game aspects must be taken into ac-
count, including positive effect, fun, and
challenge [9]. Thus, the serious game con-
cept encompasses approaches such as
simulations, educational games, or even
books. The next paragraphs provide addi-
tional insight into this approach to cogni-
tive evaluation.
Several contributions exploring the in-
troduction of serious games to assess dif-
ferent cognitive areas can be found in the
literature [10]. For example, regarding the
evaluation of episodic memory, VR Driv-
ing [11] emulates a route through a city or
town that partcipants are requested to
complete. They also have to memorize de-
tails of the scenes they encounter while
driving the route. Another example is the
VR-based home [12], which displays a vir-
tual apartment. Participants are distributed
into two groups, active and passive. Partici-
pants in the passive group just watch a pre-
viously recorded video of the apartment,
while active participants are requested to
actively explore each room and carefully
look at the objects in order to try to recall
them. Finally, as a last example, Jebara et al.
[13] propose a virtual reality environment
that displays a town with buildings, people,
cars, street objects such as lampposts, and
several intersections. The environment
Methods Inf Med 4/2018
comprises 16 different scenes and relevant
events happening at those places. Subjects
have to carry out cognitive tasks such as
free recall, visuospatial recall, delayed free
recall and recognition.
With respect to the assessment of atten-
tion, the videogame AULA [14, 15, 16], re-
produces a virtual environment depicting a
classroom, which is targeted to the diag-
nosis of the Attention Deficit Hyperactivity
Disorder (ADHD). The cognitive tasks in-
volved are omission and commission er-
rors, reaction time, motor activity, differ-
entiation between No-X and X tasks, com-
parison of sensorial modalities, and quality
of attention focus.
We can also find some videogames tar-
geting the assessment of working memory.
The Virtual Radial Arm Maze (VRAM)
[17], reproduces a virtual room that con-
tains a maze with six arms, three of which
contain a treasure at their ends. The goal of
the game is to find such treasures. As an-
other example, Shapebuilder [18] displays a
4×4 grid and four sets of basic colored
shapes at the sides of the grid. The goal is
to replay the series of stimuli presented
(i.e., ordering and spatial position of a set
of colored shapes).
There are some proposals to evaluate
executive functions. The Scavenger Hunt
[19] displays a set of randomly placed dots
that the player connects by clicking them
in a specific sequence. Data captured in-
clude recall time, search time, motor time
and number of steps. Finally, the Virtual
Action Planning Supermarket (VAP-S)
[20] reproduces a virtual environment dis-
playing a supermarket. Data captured in
this case include the total distance tra-
versed, total time, number of items pur-
chased, number of correct actions, number
of incorrect actions, number of pauses,
combined duration of pauses and time to
pay.
Attending the visuospatial ability’s
evaluation, a relevant contribution is the
Geoshooter application [21], where partici-
pants are requested to plan routes in four
different conditions in a virtual environ-
ment (i.e., real, hybrid, virtual, and vir-
tual+condition). Another example is
NeuroVirtual 3D [22], which uses virtual
reality for assessing the abilities in en-
coding, storing and syncing different spa-
tial representations.
Finally, some contributions supporting
the evaluation of prospective memory are
Virtual Reality Shopping Task [23], Virtual
Week [24] and Banville et al. [25]. The first
example reproduces a virtual reality en-
vironment in which participants have to
perform an ongoing task (i.e., purchasing a
list of items) and three event-based pros-
pective memory tasks. The second one is a
computerized board game that embeds
event-based and time-based prospective
memory tasks. And finally the last
example, involves the assessment of pros-
pective memory by requesting the player to
remember to perform some actions such as
feeding a fish and switching off a fan at
points of time in the future.
Downloaded by: Trinity College Dublin. Copyrighted material.
The examples above illustrate how seri-
ous games, and specifically those based in
virtual reality environments, are being in-
troduced in the field of cognitive evalu-
ation. These solutions are presented as
better alternatives to classical tests, as they
are more ecological because they typically
reproduce real-life situations, and they are
not perceived as an intrusive tool. How-
ever, more research is still needed to build
reliable and valid serious game-based in-
struments ready to be used in clinical neur-
opsychological assessment [10]. The main
challenge to be addressed is the lack of an
adequate psychometric validation that
would confirm their potential as a serious
alternative to classical neurophsychological
tests, following the Standards for Edu-
cational and Psychological Testing [26].
For example, the works referenced lack a
comprehensive validation of all properties
related to their psychometric validity. Fur-
thermore, another relevant limitation
found is the limited introduction of state-
of-the-art machine learning techniques for
the prediction of cognitive states, as they
are based on mainstream statistical tech-
niques only. The Discussion section below
offers a comparative study of the work in-
troduced in this paper and previous con-
tributions in the field.
This paper presents the design and pre-
liminary validation of Panoramix, a suite of
serious games to detect MCI and AD in
senior adults. Unlike previous approaches
[10], Panoramix has been designed accord-
© Georg Thieme Verlag KG 2018

ing to the standard recommendations [26]
to guarantee the required psychometric
properties that will pave the way for a fu-
ture clinical validation solution to comple-
ment or even replace classical tests. In
order to classify adults in two groups (i.e.,
healthy or MCI/AD), we introduced a
novel application of machine learning [10]
in this field. This process is discussed in
Sect. Methods where the design and devel-
opment methodology is introduced from
both a technological and psychometric per-
spective.
Panoramix is composed of six serious
games addressing the following relevant
areas to assess the cognitive status [27]: epi-
sodic memory, attention, semantic mem-
ory, working memory, procedural memory
and gnosias. This suite aims to reproduce
the abilities of classic neuropsychological
batteries to assess cognition abilities in sen-
ior adults and to detect potential cognitive
impairment. Cognition abilities addressed
include early markers of cognitive alter-
ations [28, 29], particularly episodic mem-
ory and attention, driving to MCI and po-
tentially AD. Due to the central role played
by episodic memory evaluation, a previous
work [30] addressed the design procedure
and a preliminary validation of that spe-
cific area. The research discussed in this
paper complements and completes a first
psychometric validation of Panoramix
aimed to overcome the limitations detected
in related work. The Panoramix tool is in-
troduced in Sect. Results.
Our final aim is to provide an answer to
the following research question: Can seri-
ous games and machine learning algo-
rithms be used to detect MCI or AD? As a
first step towards the validation of this hy-
pothesis we carried out a first experiment
with 16 older adults, including MCI and
AD patients, and healthy adults. The out-
comes of this pilot study, focusing in con-
struct, criterion and external validation (cf.
Sect. Results), are presented in Sect. Dis-
cussion. Finally, conclusions are presented
about the potential of using digital serious
games and classification algorithms for the
early detection of dementia processes.
© Georg Thieme Verlag KG 2018
S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
2. Methods
2.1 Design and Development of
Serious Games
The design process was centred on the end
user. For this, two focus group sessions [31]
were conducted with senior adults to
gather information on the specific needs
that should cover such games. Further-
more, the research team had a multidisci-
plinary composition by including experts
in the field of cognitive neuropsychology
and computer technologies. For each seri-
ous game, design was carried out according
to (1) the selection of cognitive area to
be assessed; (2) identification of the most
appropriate design approach, and finally
(3) identification of variables to be col-
lected throughout game playing. Finally,
from a technological perspective, we chose
Unity, which has a broad background in
the development of 3D applications to be
run in multiple systems and devices. More-
over, games were designed to run on main-
stream computers and touch devices sup-
porting accessible control interfaces such as
joysticks, gaming mice or gestural remotes
when required. The Panoramix suite offers
muti-language support and all stimuli are
also presented in textual and audio formats
to enhance accessibility. In other words,
these serious games provide additional ver-
satility, interactivity, and personalization
capabilities when compared with pen-and-
paper tests, as they are software systems de-
signed under such premises.
Table 1 Test subjects’ characteristics (general and cognitive).
Parameters AD (N=5)
Age (M±SD) 75.8±5.36
Women/Men (n) 4/1
aSchooling years (M±SD) 7.6±3.13
bFreq. Games (M±SD) 1.4±0.54
cFreq. ICT (M±SD) 1±0.00
dMMSE (M±SD) 21±4.95
dMAT (M±SD) - 44±1.41
aSchoolingyears: adapted from ISCED, UNESCO [32]
b”Frequency of use of video games” and
c”Frequency of use of technologies”: 5-point Likert scale [1(never) to 5 (very frequently)]
dMMSE score
dMAT score. It was not administered to AD group to avoid frustration.
199
2.1.1 Test Subjects
A first pilot study was carried out with 16
subjects (12 women and 4 men) from Gali-
Pervasive Health
cia, Spain, in May and June, 2016. The co-
hort study (cf. ▶ Table 1) was composed by
(1) 8 individuals with no cognitive impair-
ment as a control group (average age
68.3±8.88 years); (2) 5 AD patients (aver-
age age 75.8±5.36 years, and (3), 3 MCI
patients (average age 75±6.08 years). All
participants signed informed consent be-
fore participation, in accordance with the
Declaration of Helsinki. The pilot study
was approved by the Autonomic Research
Ethics Committee of Galicia (Spain).
Participants provided data through (1)
an initial interview where several classical
tests were administered to gather golden
Downloaded by: Trinity College Dublin. Copyrighted material.
standard data and to make sure that they
were correctly classified according the
standard clinical practice; (2) a perception
survey on the use of technology in general
and serious games in particular; (3) data
collected from Panoramix games and (4) a
final questionnaire on their experiences
during the pilot study. The pilot was car-
ried out at participants’ homes in two
sessions. With this, a relaxing, non-intru-
sive and close to real-life environment was
guaranteed. In other words, cognitive
evaluation was performed in the most eco-
logical environment possible.
The size of the sample for this prelimary
study was estimated according to main-
stream statistical methods to calculate
MCI (N=3) HC (N=8)
75±6.08 68.3±8.88
2/1 6/2
10±8.54 12.12±5.89
2.7±0.58 1.37±0.74
2.7±1.53 3.3±1.58
25.33±0.58 28.37±1.60
46±3.40
Methods Inf Med 4/2018
 200 S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
Pervasive Health
Figure 1 Methodology: cross-sectional and leave-one-out validation strategy.
sample sizes in small populations, and also,
according to existing recommendations
[33; 34] for the estimation of Spanish adult
population. According to existing data
about the Spanish adult population and the
prevalence of dementia in Spain, the
sample selected —n=16— is a represen-
tative sample of the Spanish senior popu-
lation with a 95% confidence interval and
an error of 7.5%.
2.1.2 Psychometric Validation
As the main goal of Panoramix is to emu-
late a classical neuropsychological battery,
the same validation process of such bat-
teries shall be followed to guarantee psy-
chometric validity. As a reference for this
procedure, we relied on the Standards for
Educational and Psychological Testing
[26]. Besides, from a methodological per-
spective a cross-sectional or prevalence
study [35] was performed, combined with a
one-left-out validation strategy [36] (cf.
▶ Figure 1). In our case, data were ob-
tained to assess construct, criterion and ex-
ternal validity.
Methods Inf Med 4/2018
2.1.2.1 Construct Validity
Construct validity is the degree to which a
test measures what it claims to be measur-
ing. There are several ways to assess con-
struct validity, including the correlation of
a measure of the construct and designated
variables; factor analysis; among others. In
our study, correlations were computed be-
tween the classical tests administered –
Mini Mental State Examination (MMSE)
[37]; Memory Alteration Test (MAT) [38];
Spanish version of California Verbal Learn-
ing Test (CVLT) [39] and Informant Ques-
tionnaire on Cognitive Decline in the
Elderly (IQCODE) [40] –, the correspond-
ing games, and static general variables –
gender, age, schooling years, socialize (i.e.,
level of interaction with other people), fre-
quency of use of video games, frequency of
use of technologies, etc.. Note that classical
tests provided our golden standard truth
data. Despite their limitations, these tools
have clinical validity and are widely ac-
cepted in medical practice. Thus, they con-
stitute our ground truth in order to develop
and to validate an improved cognitive digi-
tal tool based on serious games.
Downloaded by: Trinity College Dublin. Copyrighted material.
2.1.2.2 Criterion Validity
Criterion validity reflects the extent to
which a measure is related to an outcome, a
classification into health, MCI or AD sub-
jects in our case. It is typically measured
through concurrent validity (i.e., with re-
spect to diagnostic categories) and predic-
tive validity (i.e., predicting performance or
behaviour). In this case, data captured
from games only was processed using ma-
chine learning techniques to classify par-
ticipants in healthy individuals or people
with cognitive impairments (i.e., MCI or
AD in our case). In other words, we aim to
compute a cognitive assessment or cogni-
tive status of senior adults, only from their
interaction with the games in the Panora-
mix suit.
2.1.2.3 External Validity
Finally, external validity indicates whether
the test and its results can be generalized to
different subjects, people, places, experi-
menters, etc. For this, we designed both the
digital games and the pilot experiment ac-
cording to these parameters: (1) we used
sufficiently heterogeneous sample, even
© Georg Thieme Verlag KG 2018

though it was small, to guarantee popu-
lation validity; (2) the experimental en-
vironment was as natural as possible (i.e.,
to reinforce the environmental validity) by
taking place at participants’ homes; and
finally (3) we conducted the experiment at
different times for each user in order to
avoid temporary/daily aspects that may in-
terfere with the administration of digital
games. Finally, an adaptation of the Game
Experience Questionnaire (GEQ) [9, 41],
was completed by those participating in the
pilot study, to capture data about this issue.
Particularly, a questionnaire – based on a
5-point Likert scale – about perception of
Panoramix (cf. ▶ Online Appendix A), fa-
cilitated the assessment of players’ feelings
(i.e., perception extracted from Q4, Q5 and
Q6) and thoughts (i.e., perception extract-
ed from Q3 and Q7), that is, some game
experience factors.
2.2 Classification Algorithms
From an analytics perspective, we relied on
Logistic Regression (LR), as it is a tradi-
tional statistical classification method ex-
tensively used in medical classification
problems. Additionally, we applied other
machine learning classifiers such as Sup-
port Vector Machines (SVM) and Classifi-
cation and Regression Trees (CART) [42],
which were recently introduced to support
medical prediction and classification tasks
[43, 44].
For each algorithm, we used a training/
testing ratio based on the one-left-out
strategy [36] – each algorithm is trained
with the whole sample except one, and
then the same algorithm is used to predict
the classification of the one left out; repeat-
ing this process as many times as there are
samples. We could apply the algorithm to
all variables or just to a particular subset of
the most informative of them. To do it, we
used internally the Random Forest algo-
rithm [45, 46] to automatically select the
most informative variables.
On the other side, the classification cri-
terion was modelled as a dichotomous
variable: healthy controls or subjects with
cognitive problems – MCI and AD pa-
tients. Moreover, neuropsychological pre-
dictors included demographic variables
(i.e., age, gender, schooling years, frequen-
© Georg Thieme Verlag KG 2018
S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
Figure 2 Screenshot of pilot experiment playing Attentix game.
cy of use of video games, frequency of use
of technologies, etc.) and all data collected
from interacting with the serious games. In
addition, we applied the classification met-
rics below [43, 44], to offer an accuracy
value for each algorithm:
• True Positive (TP): rate of people with
cognitive problems correctly identified
as people with MCI or AD.
• True Negative (TN): rate of healthy
people correctly identified as healthy.
• False Positive (FP): rate of healthy
people incorrectly identified as people
with MCI or AD.
• False Negative (FN): rate of people with
cognitive problems incorrectly ident-
ified as healthy.
• F-measure (β): β-weighted harmonic
average of precision and recall. Its opti-
mal value is β = 1 and the worst value is
β = 0. Parameter β determines the
weight of precision in the combined
precision-recall score. In our study, re-
sults were computed for β = 1 and addi-
tionally for β = 1.5. In the later case,
more weight is given to the β par-
ameter to reduce the influence of false
negatives. Note that this is a common
criterion in medical studies.
F-measure (β) = (1 + β2) × precision
× recall / (β2 × precision) + recall
201
Pervasive Health
Downloaded by: Trinity College Dublin. Copyrighted material.
3. Results
3.1 The Panoramix Suit
3.1.1 Episodix Serious Game
Episodic memory is a cognitive system that
allows human beings to recall past experi-
ences [47]. The evaluation of episodic
memory is paramount to MCI or AD de-
tection, as it is the most reliable indicator
of the likelihood of conversion of MCI into
AD [29, 48, 49, 50]. Episodix (cf. ▶ Figure
2) is based on the gamification of the Cali-
fornia Verbal Test (CLVT) [39], which is
widely used to assess this type of memory.
In order to obtain a more ecological tool,
we introduced some changes concerning
CVLT’s administration procedure. Firstly,
instead of reproducing an imaginary shop-
ping list verbally, a virtual walk is carried
out where the objects’ list to learn is nat-
urally present on the street. Secondly, we
redefined the categories and words, so that
they are easier to recognize visually, but al-
ways maintaining the same frequency of
use as in the original test. For example, cat-
egories of vehicles and stores are included,
instead of herbs and tools. At the end, the
variables captured from Episodix (cf.
▶ Online Appendix A) are similar to the
ones in the classical test (e.g., yes/no recog-
nition, free recall, short/long delayed recall,
Methods Inf Med 4/2018
 202 S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
recency, primacy, semantic clustering, re-
sponse to inhibitions, etc.), and also, hits,
repetitions, guesses, omissions, errors, and
total time spent in each phase (e.g.,
Pervasive Health
RCL_CP2: long-term recall with 2nd sem-
antic clues; RCL_CP: long-term recall with
clues; RL_CP: long-term recall; R1_B1: im-
mediately recall, etc.).
3.1.2 Attentix Serious Game
This game assesses the attention capabil-
ities of players, as the way they select in-
formation for its conscious processing and
action, maintaining the state of alertness
required for attentive processing at the
same time. We used the popular Simon
Says game as an inspiration. Like in the
original game, Attentix reproduces a color
sequence that the player must repeat. The
length of the sequence is incremented by
one after a correct answer, for a maximum
length of 10 colors. There is also a select-
able difficulty for the color palette (4 or 5
different colors).
Finally, the variables captured to com-
pute a score value are average response
time, total time of play, hits, errors, and
level of difficulty of the sequence to be rep-
licated (cf. ▶ Online Appendix A). Note
that it will not be mandatory to perform all
phases to avoid frustration in some players.
3.1.3 Semantix Serious Game
Semantix is based on the gamification of
the Pyramids and Palm-trees test [51]. It
consists of 52 sets of chips with three
images: the stimulus given, the target
stimulus, and the distraction stimulus. The
player must try to correctly match the
stimulus given to one of the other two op-
tions. The assessment of the state of sem-
antic memory is obtained from the analysis
of the total playing time, number of hits,
number of errors, number of omissions,
and the total number of right answered
chips (cf. ▶ Online Appendix A).
3.1.4 Workix Serious Game
Working memory, also known as short-
term memory, “registers and retains in-
coming information in a highly accessible
form for a short period of time after the
Methods Inf Med 4/2018
input” [52]. One of the classical methods to
evaluate this cognitive area is the Corsi’s
Cubes test [53]. This test involves placing
nine cubes stuck randomly on a board. The
examiner plays a sequence of cubes in a
specific order and the subject under study
must reproduce that sequence (i.e., two se-
quences of 3, 4, 5 and 6 span). In the case of
assessing working memory, the input se-
quence must be repeated in reverse order.
If we replied in the same order, attention
and not working memory would be the
cognitive area involved. In other words,
this test – and therefore the designed seri-
ous game – supports the evaluation of
working memory or attention span, ac-
cording to the order in which the subject
must reproduce the sequence. Workix
gamifies this test to be run from a digital
environment. Variables considered to com-
pute a score include the total length of the
game, average response time, execution
mode, sequence reproduced indicating the
number of successful cubes (cf. ▶ Online
Appendix A).
3.1.5 Procedurix Serious Game
The procedural system is instrumental in
performing activities of daily living, that is,
it learns and retrieves all schemas or rou-
tines of thought and action [54], without
which people would not be able to develop
a normal life. Like in the case of Semantix
and Workix, design was based on the ga-
mification of a classical test by embedding
it in a computerized environment. In par-
ticular, the Pursuit Rotor Task test [32] was
used as a foundation. In this test, the sub-
ject has to track a rotating circle as accu-
rately as possible, using the mouse or other
integrated control element in the digital
version. Thus, this game measures pro-
cedural memory (cf. ▶ Figure 2) along with
motor coordination and executive func-
tions.
For this, the variables of interest that
eventually allowed us to obtain a cognitive
assessment of procedural memory include
total playing time, deviation from the el-
ement to be tracked (i.e., percentage of
deviation during the first trial), initial re-
sponse time, and total time on track (i.e.,
total time over the ball during the second
trial; idem for others trials). Each session
consists of 4 trials lasting 15 seconds where
different aspects of the rotor are modified,
including the rotational speed and the
radius of the different elements composing
the rotor (cf. ▶ Online Appendix A).
3.1.6 Gnosix Serious Game
Gnosia [55] is defined as the human capac-
ity to develop, interpret and assign mean-
ing to the information captured by the
senses. There are different categories of
gnosia, depending on the sense through
which information is captured: visual,
auditory, tactile, olfactory, gustatory, and
body scheme.
In particular, the Gnosix game focuses
on visual gnosias, that is, the ability to vis-
ually recognize different elements to assign
Downloaded by: Trinity College Dublin. Copyrighted material.
a meaning to them. To do this, four activ-
ities are proposed (i.e., recognition of
shapes, colors, expressions or feelings, or a
random combination of them). The vari-
ables captured to compute a score include
the number of correct sequences, number
of sequences failed, task type (i.e., shapes,
colors, feelings or random), average re-
sponse time, total time spent, response
time and number of attempts (cf. ▶ Online
Appendix A).
3.2 Preliminary Validation
Outcomes
3.2.1 Construct Validity
The correlation study performed between
variables collected from serious games and
classical tests according to each cognitive
group shows that:
• Episodix: best correlations are found
for: MMSE (e.g., time duration for both
HC – time duration RCL_CP2 (r=0.78,
ρ=0.01) – and MCI – time duration
RCL_CP (r=0.64, ρ=0.25); guesses –
guesses RL_CP (r=0.78, ρ=0.02) – and
omissions – omissions RL_CP (r=−0.78,
ρ=0.02) – for AD; and finally socialize –
socialize (r=0.91, ρ=0.03) for MCI – and
age – (r=0.86, ρ=0.01) for AD – as
examples of static variables); MAT (e.g.,
time duration for HC – time_du-
ration_RCL_CP2 (r=0.78, ρ=0.01) –
and MCI; guesses and schooling years
for MCI; etc.); IQCODE (e.g., time du-
ration for HC – time duration RL_CP
© Georg Thieme Verlag KG 2018

(r=0.62, p=0.46) – and socialize –
(r=0.77, ρ=0.51) – for AD); and finally
CVLT (e.g., time duration for HC –
time_duration_R1_B1 (r=0.73, ρ=0.03)
and time_duration_RCL_CP2 (r=0.96,
ρ=0)).
• Attentix: the higher correlations are ob-
served between the following game vari-
ables and MMSE, MAT and CVLT tests
results: average time – (r=−0.76, ρ=
0.01) – and failures – (r=−0.69, ρ=0.05).
Idem for IQCODE, but with a positive
correlation.
• Semantix: in this case, the best corre-
lation is observed for MCI patients:
average time (r=−1, ρ=0), time duration
(r=−1, ρ=0), socialize (r=−1, ρ=0) and
age (r=−1, ρ=0) for both MMSE and
MAT. A similar correlation is main-
tained with CVLT (r=−0.98, ρ=0). Be-
sides, we encountered proper positive
correlation with number of points
(r=−0.73, ρ=0), percentage of points
over the total (r=−0.73, ρ=0), and so-
cialize (r=0.77, ρ=0) for AD people and
IQCODE.
• Workix: best correlations are obtained
for MMSE and MAT. Specifically, the
time duration (r=0.60, ρ=0.001) for HC.
By contrast, the correlation with IQ-
CODE test is poor. On the other hand,
we found a good correlation with
gender (r=−0.91, ρ=0) and schooling
years (r=0.88, ρ=0).
• Procedurix: best correlations are found
for the time that the cursor stays on the
ball for the second – it2_TOB (r=0.80,
ρ=0.001) for MCI subjects and MMSE
and it2_TOB (r=0.89, ρ=0.001) for AD
patients and IQCODE; and the devi-
ation with respect to the center of the
ball, for the first and second iterations
(e.g., it1_deviation (r=−0.82, ρ=0.001)
for HC people and MAT). In the case of
static variables, socialize (r=0.83,
ρ=0.001) presents best correlation for
the HC group and MMSE.
• Gnosix: variables having a larger posi-
tive correlation in this game are the
average general time (r=−0.90, ρ=0.003)
for MAT and MCI people; the reaction
time in the second task (r=0.60, ρ=0.03)
for both MMSE and MAT, and inverse
correlation for IQCODE. With regard
to static data, the best results are with
© Georg Thieme Verlag KG 2018
S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
frequency of use of games (r=−1, ρ=0),
frequency of use of ICT (r=−1, ρ=0),
and subjects’ age (r=−1, ρ=0), for HC
subjects and MMSE and IQCODE.
Idem for MAT but direct correlation.
3.2.2 Criterion Validity
Several trials were run using all game vari-
ables, or just a subset of them including
those deemed as most representative (e.g.,
100%, the best 50% or the best percentage
for each method). As a preliminary step to
apply the algorithm, we constructed the
dataset and preprocessed it by interpolat-
ing median values for the missing features,
as well as normalizing them as required by
SVM.
Due to the limitations caused by fast
input stimuli, mainly in the case of AD
subjects, Semantix, Procedurix and Episo-
dix were the only games played by all 16
users in the sample. Thus, the criterion/
prediction study focused on the analysis of
these three games. The main results ob-
tained are summarized in ▶ Table 2.
Considering the most sensitive par-
ameter (i.e., false negatives) and after ap-
plying all algorithms to the games indepen-
dently and using all the variables, the most
relevant result is a FN rate of 6.25% for
Semantix using LR and CART; 0% for
Procedurix using SVM; and 9.09% for Epi-
sodix using SVM. Repeating the same
analysis but using only 50% of the most in-
formative variables, the false negative rate
is reduced to 6.25% for Semantix with LR
and SVM; 0% for Procedurix with CART
and SVM; and finally, 4.55% for Episodix
using SVM. In addition, using the optimal
percentage of most informative variables,
similar results are obtained (i.e., 6.25% –
Semantix using the best 32% of data; 0% –
Procedurix using the best 42%; and 9.09%
– Episodix using only the best 5.5.%).
Finally, the algorithms were applied
jointly to Procedurix, Semantix and Episo-
dix, using all the variables captured from
the interaction with the three games – a
data set of 89 instances/triplets. In this case,
prediction is 100% correct for all algo-
rithms: LR, CART and SVM. In other
words, there were no false negatives.
203
3.2.3 External Validity
The core module of the GEQ questionnaire
[9, 41]was utilized to collect actual feelings
Pervasive Health
and thoughts during game playing. In
addition, we applied the Cronbach’s alpha
algorithm [56] to measure the internal con-
sistency of results from this questionnaire
(cf. ▶ Table 3 and ▶ Online Appendix B,
Table 7 as supplementary information).
The main results obtained are:
• About their feelings – α=.85±1.79 for all
sample and α=.89±2.41 for participants
with cognitive problems –, a 93.4% per-
ceived the games as entertaining; a
93.4% as not boring; and a 93.4% as not
being irritating or unpleasant.
• Regarding their thoughts – α=.61±1.8 in
general and α=.40±1.49 for subjects
Downloaded by: Trinity College Dublin. Copyrighted material.
with MCI and AD –, a 53.4% consider-
ed they played properly; and a 35.7%
considered they had to work hard to
play games.
Finally, we collected subject’s perception
before and after interacting with the games
developed. Firstly, the motivation to use vi-
deogames increases from 6.3% to 60%, and
secondly, the percentage of participants as-
signing the highest score, to consider vi-
deogames as a proper tool for cognitive
evaluation, increases from 12.5% to 40%.
4. Discussion and
Conclusions
This work discusses the design and evalu-
ation of the Panoramix serious game bat-
tery, which is aimed to assess the main
cognitive areas considered the most rel-
evant diagnostic indicators of cognitive im-
pairments – episodic memory, attention,
semantic memory, working memory, pro-
cedural memory and gnosias.
Firstly, the design approach followed –
gamification of existing classical tests or
the emulation in the digital realm of tasks
clearly linked to the cognitive area to be
studied –, guarantees that the games evalu-
ate exactly the areas for which they were
designed. Furthermore, the focus group
sessions led us to gather information on the
specific needs by seniors that should be
covered by the Panoramix suite. Thus, de-
Methods Inf Med 4/2018
 204 S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML

Table 2 Classification of individuals using Panoramix.

Games Variables Algorithm TP+TNa FNb FPc F-measured F-measured
Accuracy (β=1) (β=1.5)
Semantix 100% LR 81.25% 6.25% 12.5% 0.86 0.87
Pervasive Health

Data set 100% CART 75% 6.25% 18.75% 0.82 0.85

(16 triples)
100% SVM 75% 12.5% 12.5% 0.80 0.80
Best 50% LR 75% 12.5% 12.5% 0.80 0.80
Best 50% CART 81,25% 6,25% 12,5% 0.86 0.87
Best 50% SVM 87.5% 6.25% 6.25% 0.90 0.90
Best % (32%) LR 81,25% 6,25% 12,5% 0.86 0.87
Best % (32%) CART 81,25% 6,25% 12,5% 0.86 0.87
Best % (32%) SVM 87.5% 6.25% 6.25% 0.90 0.90
Procedurix 100% LR 73.33% 11.67% 15% 0.73 0.74
Data set 100% CART 95% 1.67% 3.33% 0.95 0.96
(60 triples)
100% SVM 93.33% 0% 6.67% 0.94 0.96

Downloaded by: Trinity College Dublin. Copyrighted material.

Best 50% LR 90% 3.33% 6.67% 0.90 0.91
Best 50% CART 95% 0% 5% 0.95 0.97
Best 50% SVM 93.33% 0% 6.67% 0.94 0.96
Best % (42%) LR 80% 10% 10% 0.79 0.79
Best % (42%) CART 96.67% 0% 3.33% 0.97 0.98
Best % (42%) SVM 93.33% 0% 6.67% 0.94 0.96
Episodix 100% LR 50% 27.27% 22.73% 0.56 0.55
Data set 100% CART 50% 22.7% 22.3% 0.62 0.62
(22 triples)
100% SVM 81.82% 9.09% 9.09% 0.85 0.85
Best 50% LR 50% 27.27% 22.73% 0.56 0.55
Best 50% CART 50% 22.7% 22.3% 0.62 0.62
Best 50% SVM 90.91% 4.55% 4.55% 0.92 0.92
Best % (5.5%) LR 86.36% 9.09% 4.55% 0.88 0.87
Best % (5.5%) CART 77.27% 9.09% 13.64% 0.81 0.83
Best % (5.5%) SVM 77.27% 9.09% 13.64% 0.81 0.83
E+S +Pe 100%, Best 50% LR 100% 0% 0% 1.00 1.00
Data set & Best %
(89 triples) CART 100% 0% 0% 1.00 1.00
SVM 100% 0% 0% 1.00 1.00
aTP (True Positive) and TN (True negative) = rate of num. correct classifications or Accuracy
bFN (False Negative rate)
cFP (False Positive rate)
dF-measure (β) = is the weighted harmonic mean of precision and recall – its optimal value at 1 and its worst value at 0
eEpisodix + Semantix + Procedurix

Note: different subsets of most informative data are obtained using internally Random Forest (RF)

sign was carried out to adequately cover
the cognitive spectrum needed to detect
MCI or AD.
With respect to the validation of this re-
search, the pilot study gave a preliminary
assessment of the construct, criterion and
external validity of the battery. With re-
spect to the first one, that is, the ability to
measure cognitive areas that Panoramix is
intended to measure, all games had statisti-
cally significant correlations with classical
tests (e.g., MMSE, MAT, IQCODE and
CVLT – mainly with Episodix, but partially
with other games, such as Attentix or Pro-
cedurix. This is due to the existing inter-
relations among cognitive areas, which the
games would also make apparent. In gen-
eral terms, all games showed good corre-

Methods Inf Med 4/2018 © Georg Thieme Verlag KG 2018

 S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML 205

lations – correlation factor r close to unity Table 3 Cronbach’s alpha scores for GEQ [9, 41].
and significance ρ<0.05. Especially relevant Parametersc Game experience factora α±SD α±SD α±SD
is the correlation obtained for MCI patients (all) (HC) (MCI/AD)b
using the Semantix game. In sum, the cor- Feelings Positive affect, Negative 0.85±1.75 0.38±0.71 0.89±2.41

relation analysis shows that the serious
games implemented cover the cognitive
areas for which they were designed in the
same way as their corresponding classical
tests do (i.e., construct validity).
Regarding the ability of classifying par-
ticipants according their cognitive status as
healthy controls or people with cognitive
impairment (i.e., criterion validity), we
conducted a separate analysis on the pre-
diction capabilities of Episodix, Semantix
and Procedurix, as all participants played
these three games. As a first result, classifi-
cation offered by each game using all vari-
ables is correct in more than 75% of cases.
Pervasive Health
affect & Annoyance
Thoughts Competence & Challenge 0.61±1.8 * 0.4±1.49
aImmersion and flow answers were discard, due to we have detected difficulties to understand by
senior participants
bSubjects with cognitive problems were grouped, due sample size
▶
c Please refer to Online Appendix B for further details about the perception questionnaire and
game experience factors assessment (e.g., feelings and thoughts)
the classification accuracy of the Pano- as no incidences were detected related to
ramix suite, in line with previous studies the characteristics of the sample (i.e., genre,
[43, 44]. At the same time, most inform- educational level, training on ICT, etc.), or
ative variables present a high correlation, environmental issues (i.e., natural location
which led us to confirm both construct and at participants’ homes), or behavioral ones
criterion validity. For instance, time du- (i.e., not detecting any intrusive feeling in

Downloaded by: Trinity College Dublin. Copyrighted material.

Particularly relevant is the case of Pro-
cedurix using SVM, which predicts cor-
rectly 93.33% of subjects’ cognitive status,
resulting excellent F-values above 0.94,
both for equal precision and recall weights,
and and for recall weight > precission
weight.
In general, prediction of each game im-
proves when we perform the analysis with
a set of most informative variables. Using
the best 50%, classification results improve,
in particular for the three games using
CART. In addition, results already provide
promising prediction/classification levels
using just the best specific subset of vari-
ables for each method (i.e., Semantix = 32%,
Procedurix = 42% and Epiosdix = 5.5%), in
all cases but specially for CART and SVM.
To sum up, individually, the best classifi-
cation is performed by Procedurix, fol-
lowed by Semantix and finally Episodix.
Finally, using all the variables captured
from the three games simultaneously, a
total correct classification is achieved.
Those results are supported by the
F-measure values obtained, particularly
when limiting the influence of false
negatives (β=1.5).
We can conclude that a traditional
analysis method such as LR offers a good
prediction, but it can be improved by ap-
plying emerging machine learning meth-
ods such as CART and specially SVM,
which offers some better results. Moreover,
the different analysis approaches carried
out show consistent results, which stresses
ration RCL_CP2 (5th most informative
variable) and time duration RCL_CP4 (8th
most informative variable) and socialize
(1st); average time (1st) and time duration
(2nd) for Semantix; it2_TOB (2nd) and so-
cialize (1st) for Procedurix.
Concerning the external validity of
Panoramix, the results obtained through
the assessment of the experience of the par-
ticipants were very positive, as a very high
percentage perceived the games as enter-
taining, not boring, and not irritating (i.e.,
93.4% agree or strongly agree). Besides,
about GEQ’s internal consistency, the best
findings were related to feelings, both in
general and in subjects with cognitive
problems (α = 0.89). By contrast, thoughts,
mainly in the HC group, show a low inter-
nal consistency, but we consider that they
are influenced by some difficulties detected
in understanding and expressing opinions
about competence in video games; and
also, an excessive negative perception
about their technological abilities. Even so,
general results show a proper internal con-
sistency and GEQ led us to obtain adequate
information about external validity of
Panoramix. Other very relevant inform-
ation was the subject’s perception about
technology. For that, we asked them about
motivation to use games and also to use
them as a tool to perform cognitive assess-
ment. Both opinions dramatically improve
after playing the games. Finally, there is
promising evidence about this tool being
applicable to the general senior population,
their daily activities).
On the other hand, results obtained
about the Panoramix battery, and taking as
a reference the previous contributions dis-
cussed in Sect. 1, contribute to the advance
the state of art in this field. Note that Pano-
ramix supports the concurrent evaluation
os several cognitive areas, while previous
contributions are focused on some specific
area. Furthermore, the main limitation of
previous studies was the lack of compre-
hensive psychometric validation. In our
case, evidence of the spychometric validity
of Panoramix is provided from several per-
spectives such as construct validity (i.e., ad-
dressed in 57.14% of the works analysed
except [19], [21], [24] and [25]); criterion
validity (i.e., addressed only in [18], [21],
[24] and [25]); and finally, external validity
(addressed only in the 14.29% of studies
[20, 57]).
With respect to the introduction of ma-
chine learning techniques to detect cogni-
tive impairment as an innovative method
to study criterion validity, Panoramix pro-
posess a novel approach beyond main-
stream statistical techniques based on clas-
sifier algorthims such as Support Vector
Machines (SVM) and Classification and
Regression Trees (CART). Besides, prelimi-
nary content, face and ecological validation
was addressed in a previous work [30].
Thus, we can conclude that Panoramix
has been subject to a comprehensive psy-
chometric validation process. Besides the
validation aspects addressed, this research

© Georg Thieme Verlag KG 2018 Methods Inf Med 4/2018

 206 S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
extends our previous work [30] with addi-
tional parameters such as the Pearson cor-
relation for all games, the introduction of
additional quality metrics (e.g., accuracy;
Pervasive Health
weighted harmonic mean of precision and
recall, for both β=1 and β=1.5; and the
complete confusion coefficient matrix),
and an assessment of the gaming experi-
ence using the Cronbach’s alpha scores for
GEQ.
Some limitations were detected during
the pilot study. Firstly, the sample size is
evidently small, no matter its design has
been rigorous from a methodological (i.e.,
cross-sectional study), morfological (i.e.,
sample’s heterogeneity), and ethical/legal
perspectives (i.e., it was approved by the
Galician institutional ethics committee and
was conducted in accordance with the
provisions of the Declaration of Helsinki,
as revised in Seoul 2008). However, the
sample size is representative from a re-
search perspective [34], taking into con-
sideration the Spanish adult population.
Particularly, the selected sample represents
the target Spanish population within a
aceptable error margin for reserarch works
(i.e., 0.05 < ε=0.075 < 0.10) [33, 34] and a
confidence interval of 95%, that can be
deemed as acceptable for a preliminary
study.
On the other hand, regarding usability,
and no matter participants’ perceptions are
positive in general, in most cases they
would prefer tactile control devices, instead
of keyboard and mouse. The velocity of
stimulus presentation should also be re-
vised, mainly in the case of Attentix and
Workix, to facilitate their use by AD sub-
jects. On the other hand, some data incon-
sistencies were found in the answers for
immersion and flow in one participant.
Eventually, both questions were discarded
due to the difficulties detected to under-
stand them by senior participants. Finally,
additional research is needed to evaluate
the complete psychometric properties of
this novel system of neuropsychological
evaluation, mainly in order to provide
clinical-grade evidence by means of an
statistically significant population sample.
More specificall, the analysis of criterion
validity shall be extended to the complete
Panoramix battery, as only three games
(i.e., Episodix, Semantix and Procedurix)
Methods Inf Med 4/2018
were played by all users in the sample. No
matter the introduced machine learning
algorithms (i.e., LR, CART and SVM) pro-
vided promising results, ,this future re-
search may also be extended to additional
machine learning models. A larger data set
consequence of a larger population sample
may facilitate the investigation of addi-
tional machine learning approaches that
better adapt to the new data model.
Finally, the first pilot study provided
evidence of the construct and external psy-
chometric validity of the Panoramix bat-
tery, and also encouraging evidence about
the criterion validity of a combination of
Semantix, Procedurix and Episodix, as data
obtained from playing with these three
games correctly discriminated all healthy
subjects from those already suffering AD
or MCI in the context of this first pilot
study. Thus, no matter the limited sample,
the potential of serious games and machine
learning algorithms to support the early
detection of dementia processes was con-
firmed. Furthermore, discrimination was
performed in a nonintrusive, non-frustrat-
ing way, which is especially relevant when
targeting elder adults that may suffer cog-
nitive impairments.
Acknowledgment
The authors would like to thank the valu-
able comments and suggestions provided
by experts during design phase of digital
games: Dr. Arturo X. Pereiro Rozas and
Dr. Cristina Lojo Seoane from the Depart-
ment of Developmental Psychology at the
University of Santiago de Compostela, and
M.D. M. Jose Moreno Carretero from the
Neurology service at the Alvaro Cunqueiro
University Hospital (Vigo, Galicia, Spain).
We also appreciate the support of the As-
sociation of Relatives of Alzheimer’s Pa-
tients and other Dementias of Galicia
(AFAGA), which facilitated the selection of
participants among its members.
References
1. Selnes OA. A Compendium of Neuropsychological
Tests: Administration, Norms, and Commentary.
Oxford: Oxford University Press; 1991.
2. Williams W, Graham DP, McCurry K, Sanders A,
Eiseman J, Chiu PH, King-Casas B. Group psycho-
therapy’s impact on trust in veterans with PTSD: a
pilot study. Bull Menninger Clin 2014; 78(4):
335–348.
3. Cordell CB, Borson S, Boustani M, Chodosh J,
Reuben D, Verghese J, Thies W, Fried LB, Medi-
care Detection of Cognitive Impairment Work-
group. Alzheimer’s Association recommendations
for operationalizing the detection of cognitive im-
pairment during the Medicare Annual Wellness
Visit in a primary care setting. Alzheimer’s De-
ment 2013; 9(2): 141–150.
4. Holtzman DM, Morris JC, Goate AM. Alzheimer’s
disease: the challenge of the second century. Sci
Transl Med 2011; 3(77): 77sr1.
5. Parsons TD. Virtual Teacher and Classroom for
Assessment of Neurodevelopmental Disorders. In:
Brooks A, Brahnam S, Jain L, editors. Technologies
of Inclusive Well-Being. Berlin, Heidelberg:
Springer; 2014. p. 121–137.
6. Luciana M. Practitioner review: computerized as-
sessment of neuropsychological function in
children: clinical and research applications of the
Cambridge Neuropsychological Testing Auto-
Downloaded by: Trinity College Dublin. Copyrighted material.
mated Battery (CANTAB). J Child Psychol Psy-
chiatry 2003; 44(5): 649–663.
7. Hsu Y-L, Chung P-C, Wang W-H, Pai M-C, Wang
C-Y, Lin C-W, Wu H-L, Wang J-S. Gait and bal-
ance analysis for patients with Alzheimer’s disease
using an inertial-sensor-based wearable instru-
ment. IEEE J Biomed Health Inform 2014; 18(6):
1822–1830.
8. Michael DR, Chen SL. Serious games: Games that
educate, train, and inform. Muska & Lipman/Pre-
mier-Trade; 2005.
9. Nap HH, Diaz-Orueta U, González MF, Lozar-
Manfreda K, Facal D, Dolničar V, Oyarzun D,
Ranga M-M, de Schutter B. Older people’s percep-
tions and experiences of a digital learning game.
Gerontechnology 2014; 13(3): 322–331.
10. Valladares-Rodriguez S, Pérez-Rodriguez R,
Anido-Rifón L, Fernández-Iglesias M. Trends on
the application of serious games to neuropsycho-
logical evaluation: A scoping review. J Biomed In-
form 2016; 64: 296–319.
11. Plancher GG, Barra J, Orriols E, Piolino P. The in-
fluence of action on episodic memory: a virtual
reality study. Q J Exp Psychol (Hove) 2013; 66(5):
895–909.
12. Sauzéon H, Arvind Pala P, Larrue F, Wallet G,
Déjos M, Zheng X, Guitton P, N’Kaoua B. The use
of Virtual Reality for episodic memory assessment:
Effects of active navigation. Exp Psychol 2012;
59(2): 99–108.
13. Jebara N, Orriols E, Zaoui M, Berthoz A, Piolino P.
Effects of enactment in episodic memory: a pilot
virtual reality study with young and elderly adults.
Front Aging Neurosci 2014; 6: 338.
14. Díaz-Orueta U, Garcia-López C, Crespo-Eguílaz
N, Sánchez-Carpintero R, Climent G, Narbona J,
Diaz-Orueta U, Garcia-Lopez C, Crespo-Eguilaz
N, Sanchez-Carpintero R, Climent G, Narbona J.
AULA virtual reality test as an attention measure:
Convergent validity with Conners Continuous
Performance Test. Child Neuropsychol 2014;
20(3): 328–342.
15. Iriarte Y, Diaz-Orueta U, Cueto E, Irazustabarrena
P, Banterla F, Climent G. AULA – Advanced vir-
tual reality tool for the assessment of attention:
© Georg Thieme Verlag KG 2018

Normative study in Spain. J Atten Disord 2016;
20(6): 542–568.
16. Shriki L, Weizer M, Pollak Y, Weiss PL, Rizzo AA,
Gross-Tsur V. The utility of a continuous perform-
ance test embedded in virtual reality in measuring
the effectiveness of MPH treatment in boys with
ADHD. Harefuah 2010; 149(1): 18–23, 63.
17. Lee J-Y, Kho S, Yoo H Bin, Park S, Choi J-S, Kwon
JS, Cha KR, Jung H-Y. Spatial memory impair-
ments in amnestic mild cognitive impairment in a
virtual radial arm maze. Neuropsychiatr Dis Treat
2014; 10: 653–660.
18. Atkins SM, Sprenger AM, Colflesh GJ, Briner TL,
Buchanan JB, Chavis SE, Chen SY, Iannuzzi GL,
Kashtelyan V, Dowling E, Harbison JI, Bolger DJ,
Bunting MF, Dougherty MR. Measuring working
memory is all fun and games: A four-dimensional
spatial game predicts cognitive task performance.
Exp Psychol 2014; 61(6): 417–438.
19. Hagler S, Jimison HB, Pavel M. Assessing execu-
tive function using a computer game: computa-
tional modeling of cognitive processes. IEEE J
Biomed Health Inform 2014; 18(4): 1442–1452.
20. Werner P, Rabinowitz S, Klinger E, Korczyn AD,
Josman N. Use of the Virtual Action Planning
Supermarket for the Diagnosis of Mild Cognitive
Impairment. Dement Geriatr Cogn Disord 2009;
27: 301–309.
21. van der Ham IJM, Faber AME, Venselaar M, van
Kreveld MJ, Löffler M. Ecological validity of vir-
tual environments to assess human navigation
ability. Front Psychol 2015; 6: 637.
22. Serino S, Morganti F, Di Stefano F, Riva G. Detect-
ing early egocentric and allocentric impairments
deficits in Alzheimer’s disease: An experimental
study with virtual reality. Front Aging Neurosci
2015; 7: 88.
23. Canty AL, Fleming J, Patterson F, Green HJ, Man
D, Shum DHK. Evaluation of a virtual reality pros-
pective memory task for use with individuals with
severe traumatic brain injury. Neuropsychol Reha-
bil 2014; 24(2): 238–265.
24. Henry JD, Terrett G, Altgassen M, Raponi-
Saunders S, Ballhausen N, Schnitzspahn KM, Ren-
dell PG. A Virtual Week study of prospective
memory function in autism spectrum disorders. J
Exp Child Psychol 2014; 127: 110–125.
25. Banville F, Nolin P, Lalonde S, Henry M, Dery
M-P, Villemure R. Multitasking and prospective
memory: Can virtual reality be useful for diag-
nosis? Behav Neurol 2010; 23(4): 209–211.
26. AERA, APA, NCME. The Standards for Edu-
cational and Psychological Testing. AERA; 2014.
27. Petersen RC. Mild cognitive impairment as a diag-
nostic entity. J Intern Med 2004; 256(3): 183–194.
28. Facal D, Guàrdia-Olmos J, Juncos-Rabadán O.
Diagnostic transitions in mild cognitive impair-
ment by use of simple Markov models. Int J Ge-
riatr Psychiatry 2015; 30(7): 669–676.
29. Juncos-Rabadán O, Pereiro AX, Facal D, Rodri-
guez N, Lojo C, Caamaño JA, Sueiro J, Boveda J,
Eiroa P. Prevalence and correlates of cognitive im-
pairment in adults with subjective memory com-
© Georg Thieme Verlag KG 2018
S. Valladares-Rodriguez et al.: Detection of Cognitive Impairment Using Games and ML
plaints in primary care centres. Dement Geriatr
Cogn Disord 2012; 33(4): 226–232.
30. Valladares-Rodriguez S, Perez-Rodriguez R, Facal
D, Fernandez-Iglesias MJ, Anido-Rifon L, Mour-
iño-Garcia M. Design process and preliminary
psychometric study of a video game to detect cog-
nitive impairment in senior adults. PeerJ 2017; 5:
e3508.
31. Liamputtong P. Focus group methodology: Prin-
ciple and practice. Sage Publications; 2011.
32. UNESCO Institute for Statistics [Internet]. Inter-
national Standard Classification of Education:
ISCED 2011. UIS, Montreal, Quebec; 2012. Avail-
able from: http://uis.unesco.org/sites/default/files/
documents/international-standard-classification-
of-education-isced-2011-en.pdf.
33. Hajian-Tilaki K. Sample size estimation in diag-
nostic test studies of biomedical informatics. J
Biomed Inform 2014; 48: 193–204.
34. Suresh KP, Chandrashekara S. Sample size esti-
mation and power analysis for clinical research
studies. J Hum Reprod Sci 2012; 5(1): 7.
35. Barnett K, Mercer SW, Norbury M, Watt G, Wyke
S, Guthrie B. Epidemiology of multimorbidity and
implications for health care, research, and medical
education: a cross-sectional study. Lancet 2012;
380(9836): 37–43.
36. Soyka F, Giordano PR, Barnett-Cowan M, Bulthoff
HH. Modeling direction discrimination thresholds
for yaw rotations around an earth-vertical axis for
arbitrary motion profiles. Exp Brain Res 2012;
220(1): 89–99.
37. Cockrell JR, Folstein MF. Mini-Mental State
Examination (MMSE). Psychopharmacol Bull
1987; 24(4): 689–692.
38. Rami L, Bosch B, Sanchez-Valle R, Molinuevo JL.
The memory alteration test (M@ T) discriminates
between subjective memory complaints, mild cog-
nitive impairment and Alzheimer’s disease. Arch
Gerontol Geriatr 2010; 50(2): 171–174.
39. Delis DC, Kramer JH, Kaplan E, Thompkins BAO.
CVLT: California Verbal Learning Test-Adult Ver-
sion: Manual.San Antonio, Texas: Psychological
Corporation; 1987.
40. Jorm AF. The Informant Questionnaire on cogni-
tive decline in the elderly (IQCODE): a review. Int
Psychogeriatrics 2004; 16(3): 275–293.
41. IJsselsteijn WA, De Kort YAW, Poels K. The Game
Experience Questionnaire: Development of a self-
report measure to assess the psychological impact
of digital games. Forthcoming.
42. De’ath G, Fabricius KE. Classification and regres-
sion trees: a powerful yet simple technique for eco-
logical data analysis. Ecology 2000; 81(11):
3178–3192.
43. Maroco J, Silva D, Rodrigues A, Guerreiro M,
Santana I, de Mendonça A. Data mining methods
in the prediction of Dementia: A real-data com-
parison of the accuracy, sensitivity and specificity
of linear discriminant analysis, logistic regression,
neural networks, support vector machines, classifi-
cation trees and random forests. BMC Res Notes
2011; 4(1):299.
207
44. Lehmann C, Koenig T, Jelic V, Prichep L, John RE,
Wahlund L-O, Dodge Y, Dierks T. Application and
comparison of classification algorithms for recog-
nition of Alzheimer’s disease in electrical brain ac-
tivity (EEG). J Neurosci Methods 2007; 161(2):
Pervasive Health
342–350.
45. Archer KJ, Kimes R V. Empirical characterization
of random forest variable importance measures.
Comput Stat Data Anal 2008; 52(4): 2249–2260.
46. Pal M. Random forest classifier for remote sensing
classification. Int J Remote Sens 2005; 26(1):
217–222.
47. Tulving E. Episodic memory: from mind to brain.
Annu Rev Psychol 2002; 53(1): 1–25.
48. Albert MS, DeKosky ST, Dickson D, Dubois B,
Feldman HH, Fox NC, Gamst A, Holtzman DM,
Jagust WJ, Petersen RC, Snyder PJ, Carrillo MC,
Thies B, Phelps CH. The diagnosis of mild cogni-
tive impairment due to Alzheimer’s disease: rec-
ommendations from the National Institute on
Aging-Alzheimer’s Association workgroups on
diagnostic guidelines for Alzheimer’s disease. Alz-
heimers Dement 2011; 7(3): 270–279.
Downloaded by: Trinity College Dublin. Copyrighted material.
49. Serino S, Pedroli E, Cipresso P, Pallavicini F, Alba-
ni G, Mauro A, Riva G. The role of virtual reality
in neuropsychology: The virtual multiple errands
test for the assessment of executive functions in
Parkinson’s disease. Intell Syst Ref Libr 2014; 68:
257–274.
50. Campos-Magdaleno M, Díaz-Bóveda R, Juncos-
Rabadán O, Facal D, Pereiro AX. Learning and ser-
ial effects on verbal memory in mild cognitive im-
pairment. Appl Neuropsychol Adult 2015; 23(4):
1–14.
51. Howard D, Patterson KE. The Pyramids and Palm
Trees Test: A test of semantic access from words
and pictures. Thames Valley Test Company; 1992.
52. Schacter DL, Tulving E. Memory systems 1994.
ilustrada. Mit Press; 1994.
53. Kaplan E, Fein D, Morris R, Delis D. WAIS-R NI
Manual. San Antonio: Psychological Corporation;
1991.
54. Schank RC, Abelson RP. Scripts, plans, goals, and
understanding: An inquiry into human knowledge
structures. Psychology Press; 2013.
55. Lezak MD. Neuropsychological assessment. Ox-
ford: Oxford University Press; 2004.
56. Gliem JA, Gliem RR. Calculating, interpreting,
and reporting Cronbach’s alpha reliability coeffi-
cient for Likert-type scales. In: Midwest Research-
to-Practice Conference in Adult, Continuing, and
Community Education; The Ohio State University,
Columbus: 2003.
57. Plancher G, Tirard A, Gyselinck V, Nicolas S,
Piolino P. Using virtual reality to characterize epi-
sodic memory profiles in amnestic mild cognitive
impairment and Alzheimer’s disease: influence of
active and passive encoding. Neuropsychologia
2012; 50(5): 592–602.
Methods Inf Med 4/2018