Journal of Biomedical Informatics 62 (2016) 202–209

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier.com/locate/yjbin

Assessing the user experience of older adults using a neural network

trained to recognize emotions from brain signals
Victoria Meza-Kubo ⇑, Alberto L. Morán, Ivan Carrillo, Gilberto Galindo, Eloisa García-Canseco
Facultad de Ciencias, Universidad Autónoma de Baja California, Km. 103 Carretera Tijuana-Ensenada, Mexico

a r t i c l e i n f o a b s t r a c t

Article history: The use of Ambient Assisted Living (AAL) technologies as a means to cope with problems that arise due to
Received 1 March 2016 an increasing and aging population is becoming usual. AAL technologies are used to prevent, cure and
Revised 22 June 2016 improve the wellness and health conditions of the elderly. However, their adoption and use by older
Accepted 4 July 2016
adults is still a major challenge. User Experience (UX) evaluations aim at aiding on this task, by identify-
Available online 5 July 2016
ing the experience that a user has while interacting with an AAL technology under particular conditions.
This may help designing better products and improve user engagement and adoption of AAL solutions.
Keywords:
However, evaluating the UX of AAL technologies is a difficult task, due to the inherent limitations of their
EEG signals
Emotion recognition
subjects and of the evaluation methods. In this study, we validated the feasibility of assessing the UX of
Neural networks older adults while they use a cognitive stimulation application using a neural network trained to recog-
User experience nize pleasant and unpleasant emotions from electroencephalography (EEG) signals by contrasting our
Evaluation results with those of additional self-report and qualitative analysis UX evaluations. Our study results pro-
Older adults vide evidence about the feasibility of assessing the UX of older adults using a neural network that take as
input the EEG signals; the classification accuracy of our neural network ranges from 60.87% to 82.61%. As
future work we will conduct additional UX evaluation studies using the three different methods, in order
to appropriately validate these results.
Ó 2016 Elsevier Inc. All rights reserved.

1. Introduction differences in the circumstances during and after an interaction

Inherent problems due to a world wide increasing and aging
population require the development of solutions to cope with
them. These problems include the physical and cognitive problems
of the elderly [1]. Solutions include Ambient Assisted Living (AAL)
technologies that can be used to prevent, cure, and improve well-
ness and the health conditions of older adults [2]. However, due to
a decline in the elderly’s physical and cognitive skills, assessing
their perception regarding the use, acceptance and adoption of
these technologies is of great interest. Technology designed for
elderly users should not only focus on usability issues, but also
seriously investigate their motivation to engage them on the use
of the new technology [3]. These assessments are usually per-
formed through user experience (UX) evaluations.
Although, there is no clear consensus about the definition of UX
[4], it is generally understood as inherently dynamic, given the
ever-changing internal and emotional state of a person and the
⇑ Corresponding author.
E-mail addresses: mmeza@uabc.edu.mx (V. Meza-Kubo), alberto.moran@uabc.
edu.mx (A.L. Morán), ivan.carrillo@uabc.edu.mx (I. Carrillo), eloisa.garcia@uabc.edu.
mx (G. Galindo), gilberto.galindo.aldana@uabc.edu.mx (E. García-Canseco).
with a technological product [5,6]. The main effect of a good UX
is user engagement. This is characterized by attributes of chal-
lenge, positive affect, endurability, aesthetic and sensory appeal,
attention, feedback, variety/novelty, interactivity, and perceived
user control [7]. All these attributes are measured through UX eval-
uations, where understanding the users’ experiences that a product
evokes, either pleasant or unpleasant, is crucial [8].
In the literature, several UX evaluations have been reported to
assess the perception of the elderly concerning the use of technol-
ogy [9], nevertheless, this kind of assessments may be little reliable
due to the inherent limitations of the subjects, and also the limita-
tions of the evaluation methods. For example, Arhippainen con-
cluded ‘‘in self-report based techniques, participants tend to
respond what the researcher wants to hear, or tend to be insincere
and improve their perception of the results because they felt
assessed, or because they have forgotten the details of their expe-
rience” [10]. On other hand, the use of qualitative analysis tech-
niques include coding the gestures, verbal and non-verbal
expressions and other subject data as an indicator of human expe-
rience, however, this technique requires a lengthy and rigorous
process [11].

http://dx.doi.org/10.1016/j.jbi.2016.07.004
1532-0464/Ó 2016 Elsevier Inc. All rights reserved.
 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209
An alternative to these methods is the neurophysiological
method, which infers emotions during the use of a product by
measuring and interpreting physiological signals [11]. These sig-
nals include respiration and hearth rate (electrocardiography), gal-
vanic skin response, electric muscle (electromyography) and brain
signals (electroencephalography). In particular, the use of elec-
troencephalography (EEG) signals is more reliable because of its
high accuracy and objective evaluation in comparison with other
external appearance clues like facial expressions and gestures
[12]. Various psychophysiological studies have demonstrated the
correlations between human emotions and EEG signals [13–15].
Nonetheless advances in EEG algorithms creation, simplification
of devices and real life applications, many features of emotional
states still require to be improved as to obtain accurate interpreta-
tions of emotions from EEG signals in the human brain.
In this study, we aim at validating the feasibility of assessing the
UX of older adults while they use a cognitive stimulation (CS)
application using a neurophysiological technique, namely a neural
network trained to recognize pleasant and unpleasant emotions
from EEG signals.
In the first part of the study, we constructed, trained and vali-
dated a Patternnet neural network to identify pleasant and
unpleasant emotions. The neural network was trained using EEG
signals recorded during the presentation of visual stimuli that
induced emotions known a priori. The accuracy obtained was of
93.3% for 600 characteristics of the preprocessed signals [16].
In the second part of the study, we validated the use of the
trained neural network to assess the UX of the older adults while
they use a CS application. A summary of the main results of the
first study, along with the details of the second study and its main
results are presented here.
The remainder of the paper is organized as follows: Section 2
presents a summary of related work concerning emotion recogni-
tion and UX evaluation using EEG signals. Sections 3 and 4 present
the process of recognition of emotions from EEG signals. Section 3
summarizes the construction, training and validation of the neural
network used to differentiate between pleasant and unpleasant
emotions, while Section 4 uses the previously trained neural net-
work to classify the brain signals recorded during the use of a CS
application. Section 5 presents the evaluation results, while Sec-
tion 6 discusses these results. Finally, concluding remarks and
future work are presented in Section 7.
2. Emotions and EEG signals
Regardless of the context of the situation, people experience a
range of emotions whether positive (e.g. joy, gratefulness, sympa-
thy, happiness, love, etc.) or negative (e.g. displeasure, irritability,
disgust, anger, sadness, etc.) [17]. Positive emotions are associated
with the activation of regions in the left hemisphere of the brain
while negative emotions are related to the activation of regions
in the right hemisphere [18,19].
In the frequency domain, the spectral power in various fre-
quency bands has been implicated in the emotional state. The
asymmetry analysis of the alpha power is recognized as a useful
procedure for the study of emotional reactivity [20]. Furthermore,
it is common to find asymmetries in the frontal region of the brain,
which may be perceived on a subject since childhood [21]. In a
study conducted in [22] a spectral analysis of the electrical activity
obtained through an EEG demonstrated that the alpha power var-
ies depending on the emotion present (positive or negative).
In recent years, a growing number of efforts to recognize a per-
son’s emotion in real time using EEG have been reported in litera-
ture. For instance, Ishino & Hagiwara [23] proposed a system that
estimated subjective feeling using neural networks to categorize
203
emotional states based on EEG features, with 54–67% of accuracy
for four emotional states. Petrantonakis [24] used neural networks
to classify EEG signals into six emotions based on emotional
valence and arousal, with a 61% success rate. Zhang and Lee [25],
proposed an emotion understanding system that classified users’
status into two emotional states with accuracy of 73% during
image viewing. Chai [26] evaluated three mobile phone applica-
tions using a self-report technique and recording brain activity
using EEG signals from participants in order to determine the pos-
itive and negative states in the UX of the applications used. Gürkök
[27] evaluated the UX of a brain-computer interface game based on
the expectation of users; they used an evoked visual response as
stimuli, and an EEG to register physiological data. [28] used
machine-learning algorithms to categorize EEG dynamics accord-
ing to subject self-reported emotional states during music learning,
identifying four emotional states with 82.9% of accuracy.
Although some of the aforementioned studies suggest that EEG
signals can be used to identify the users’ emotional states, there are
issues that remain open and should be still explored. In this work,
we will present the results of using a neural network, trained to
recognize emotional states evoked from visual stimuli from EGG
signals (Stage 1), to infer the UX of older adults while they conduct
a cognitive stimulation (CS) activity (Stage 2); that is, we will try to
recognize emotional states evoked from visual, auditory and cogni-
tive processing stimuli.
3. Stage 1: Assessing pleasant and unpleasant emotions from
visual stimuli via EEG signals
This section summarizes the procedure performed in [16] to
design, train and validate the neural network used to recognize
pleasant and unpleasant emotions.
3.1. Introduction
The EEG signals were recorded using the EMOTIV EPOC + Head-
set1 (Fig. 1), which is a neuro-signal acquisition and processing wire-
less neuroheadset with 14 saline sensors that is able not only to
detect brain signals but also users facial expressions. The EMOTIVE
EPOC + Headset provides access to dense array, high quality, raw
EEG data by software.
In order to evoke emotions from participants, we used a set of
the International Affective Picture System (IAPS),2 which is a free
access database of pictures designed to provide a standardized set
of pictures for studying emotion and attention that has been widely
used in psychological research. During the first stage of the study, we
presented a set of the selected IAPS pictures to stimulate known a
priori emotions while recording the EEG response from participants
with the EMOTIV EPOC + headset. Then, the EEG signals were filtered
and processed to design and train a neural network that we used to
identify two basic emotional states, namely, pleasant or unpleasant.
Participants were eight older adults, 2 male and 6 female, aged
60–83 years old (AVG 72.3 years, SD 8.46 years). Inclusion criteria
were: older adults aged over 60 years, not having suffered a head
trauma, absence of moderate or severe cognitive problems, and
absence of visual problems.
3.2. Procedure
Phase 1. First, participants were introduced to the experiment,
and the EMOTIV EPOC + Headset device was calibrated prior to
use in order to set base lines signals. The calibration process was
1
http://emotiv.com/.
2
http://csea.phhp.ufl.edu/Media.html.
204 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209
Fig. 1. EMOTIVE EPOC + Headset is a low cost 14 channel wireless EEG.
realized through the EMOTIV Control Panel Application,3 where
participant’s facial expressions were imitated by a virtual robot,
and participants practiced mental commands control to move a
floating cube.
Second, a set of IAPS pictures were presented to each partici-
pant as follows: pleasant, fear, unpleasant and neutral for 6 s each,
and immediately after, each participant was asked to indicate what
was his/her impression upon seeing the picture, and selecting one
of the following categories for the picture: pleasant, unpleasant,
neutral and fear respectively (see Fig. 2). In this manner, the col-
lected brain signals were automatically labeled with the corre-
sponding participant’s answers.
Phase 2. In this phase the EEG signals were processed. First,
these signals were preprocessed and their feature signals were
extracted using the Fast Fourier Transform. Second, the signal pow-
ers were classified into brain waves: Alpha, Beta and Theta. And
finally, these records where used as input for a Patternnet neural
network with a 151-neuron hidden layer. The neural network
was trained using the brain signals from pleasant and unpleasant
emotions and the participants’ verbal responses. The network clas-
sification accuracy of the brain signals in the corresponding emo-
tions of pleasant or unpleasant obtained was of 93.3% for 600
characteristics of the preprocessed signals during the validation
process (refer to [16] for a more detailed explanation).
4. Stage 2: Assessing the UX of older adults conducting CS
activities by means of a neural network that recognizes
emotions from EEG signals
In this section we report on a study to evaluate the feasibility of
using the neural network obtained in the previous stage to assess
the UX of older adults conducting CS activities. It should be high-
lighted that the neural network was trained to recognize pleasant
and unpleasant emotional states from EEG signal evoked by visual
stimuli using IAPS pictures. However, in this stage, the target emo-
tional states will be evoked by the UX of older adults while using a
CS application that generates not only visual stimuli, but also audi-
tory and cognitive processing stimuli.
In order to validate the recognition accuracy of the neural net-
work, we conducted additional self-report and qualitative analysis
by observation UX evaluations of the CS activity to contrast the
obtained results.
For the self-report evaluation participants were asked to answer
a questionnaire immediately after concluding the CS activity. For
the qualitative analysis evaluation, the CS activity sessions were
3
http://emotiv.com/product/xavier-control-panel/.
Fig. 2. Presentation scheme of IAPS pictures.
video recorded, and analyzed to identify verbal and non-verbal
expressions of older adults during their conducting the activity.
4.1. Participants
Participants were 23 older adults, 10 male and 13 female, aged
60–83 years old (AVR 65.5 years, SD 6.227 years). Inclusion criteria
were: aged over 60 years, not having suffered a head trauma,
absence of moderate or severe cognitive problems, not having
motor impairments and absence of visual problems (i.e., being able
to see without glasses at a distance of 30–50 cm).
4.2. Materials
To carry out the study we used:
 CS application. We used Abueparty [29], a cognitive wellness
system that uses a game board metaphor similar to the tradi-
tional ‘‘Snakes and Ladders” game (Fig. 3). The main game (up
to 4 players) integrates several mini-games that implement
cognitive challenges that users have to complete in order to
advance through the board to the finish line. The application
uses a tactile screen and a custom tangible control to ease inter-
action for the elderly.
 EMOTIV EPOC + Headset to register EEG signals through 14
electrodes (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8,
AF4) distributed according to the International System 10–20
[30].
 EMOTIV Control Panel for calibrating the headset and training.
 Video camcorder for registering the activities carried out during
the study for future qualitative analysis.
 Questionnaires (Likert TAM-Like) for assessing user experience
[31].
4.3. Procedure
Fig. 4 depicts the different evaluation phases. A brief description
of these follows.
4.3.1. Introduction
Firstly, participants where individually introduced to the exper-
iment and to the main features of the EMOTIV EPOC + Headset so
that they could get familiar with its use. Participants were also
asked to sign a consent form, and the Mini Mental State Examina-
tion (MMSE) was applied [32]. The MMSE is the most commonly
used test for complaints of problems with memory or other mental
abilities. It consists of a series of questions and tests, each of which
scores points if answered correctly. After the introduction, the
device was calibrated for each participant by using the EMOTIV
Control Panel Application. Finally, participants were introduced
to Abueparty, its use and interaction with its mini-games. Also
the instructions regarding the use of the tactile screen and the tan-
gible control were provided.
 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209 205

Fig. 3. Abueparty’s main screen.

Fig. 4. Emotion recognition process using EEG signals.

4.3.2. Data collection
The activity was performed in pairs, competing among them to
reach the goal. At this stage, each participant wore the EMOTIV
EPOC + headset, so that the brain activity may be recorded using
the Test-Bench software. During the game, an observer inserted a
mini-game label that served for future signal segmentation. The
complete interaction of the participants was video recorded for
later analysis.
4.3.3. Data processing
The recorded EEG signal included some artifacts (e.g., electrical
signals due to facial movements, eye blinking, etc.), which must be
206 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209
removed in order to enhance the EEG signal. To remove these arti-
facts we applied a pre-processing procedure consisting of average
elimination and best linear fit of the signal mean, a Hamming win-
dow and a Finite Impulse Response (FIR) bandpass filter (1–30 Hz)
were used. After that, a Fast Fourier Transform (FFT) was applied to
each brain wave (Alpha, Beta and Theta), in order to extract signal
features such as magnitude, frequency and spectral power. Further
detail about the data processing procedure is provided elsewhere
[16].
4.3.4. Data classification
Previous to classification the processed EEG signals must be
segmented in order to fit as input data to the neural network.
Finally, we used the Patternnet neural network previously trained
for signal classification. The neural network provides an output in
the form (P, U), where P is the percentage of inputs classified as
‘‘pleasant” and U is the percentage of inputs classified as
‘‘unpleasant”.
At the end of the game, participants were asked to answer an
on-exit survey to obtain their perception on the intention of use
and on their UX perception.
5. Results
In this section we present the results from the self-report, qual-
itative assessment, and neurophysiological evaluations in order to
compare and validate them.
5.1. Self-reporting UX evaluation
To obtain the older adults’ perception on the use and on the UX
of the application by self-report we used a TAM-Like 5 point Likert
scale questionnaire used in [31]. Although the questionnaire com-
prises several UX aspects, for this study we focused on the percep-
tion of enjoyment (pleasant) and anxiety (unpleasant) aspects.
Regarding the ‘‘Felt anxiety while using the system” (unpleas-
ant) question, participants reported being little anxious during
the game (see Fig. 5; data were normalized 0 to 100). On average,
Fig. 5. Results reported by participants in the self-report UX evaluation.
Fig. 6. Results reported by the expert observer in the qualitative analysis UX evaluation.
participants reported being a little anxious (9.78/100). In particu-
lar, participants P3, P4 and P7 indicated that they had been a little
more anxious during the activity (37.5/100, 50/100, and 50/100,
respectively).
Regarding the ‘‘I enjoyed playing Abueparty” (pleasant) ques-
tion, all participants reported having enjoyed (AVG 99.63/100)
the activity and the application (see Fig. 5).
This result would be usually interpreted and reported as ‘‘all
participants (23/23, 100%) had a pleasant experience conducting
the CS activities; although some of them (3, 13.04%) reported being
a little anxious also”.
5.2. Qualitative analysis UX evaluation
As described in the previous section, the activity of the partici-
pants was videotaped during the study for later analysis. The
videos were analyzed to assess the UX using the qualitative tech-
niques described in [33]. This method suggests observing verbal
and non-verbal expressions that may indicate the emotional states
of participants. Through a quantification of the frequencies of these
expressions and a qualitative analysis of them, an UX expert deter-
mined the overall user experience classifying it as pleasant, neutral
or unpleasant (see Fig. 6).
The results of the qualitative evaluation concluded that most
older adults mainly showed expressions and attitudes of pleasure
during the activity (15/23, 65.21%). Further, there were some par-
ticipants (P4, P7, P9 and P17) that mainly showed expressions and
attitudes of displeasure (4/23, 17.39%). In addition, there was a
participant that showed both types of expressions almost equally
(P2, 1/23, 4.34%) and some participants that mainly were inexpres-
sive (P1, P10 and P11); all of them were classified as Neutral (4/23,
17.39%).
Participant P1, although focused, remained serious throughout
the activity and showed no expression indicating either pleasant
(enjoy) or unpleasant (anxiety) emotions; he did not even inter-
acted much with his activity partner. It was the same case for par-
ticipants P10 and P11, whom conducted the activity together.
Participants P2 and P17 played together. They remained
focused and participating in the activity, although both of them
 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209
Fig. 7. Results obtained by the neural network in the neurophysiological evaluation.
Table 1
Summary of results of the UX assessment studies, by emotion.
Emotion Self-report Qualitative analysis EEG signal classification
Pleasant 23 (100%) 15 (65.22%) 17 (73.91%)
Unpleasant 0 4 (17.39%) 6 (26.09%)
Neutral 0 4 (17.39%) 0
looked nervous at the beginning. There was very little interaction
between them. P17 looked very serious throughout the activity;
for this reason, he was considered as having unpleasant emotions.
P2 was considered as being neutral.
Participants P9 and P14 played together. During the activity
participant P14 was more active than participant P9. However, par-
ticipant P14 criticized P9’s plays, causing him to get bothered and
make comments such as ‘‘It can’t be, the same mini-game again!”,
‘‘I do not want to play anymore”, ‘‘It can’t be, another wrong
answer”, ‘‘Better luck next time, I better leave”. Thus, although
P14 enjoyed the game, P9 did not.
Participants P4 and P7 played together, and were evaluated as
having an unpleasant UX during their participation. The reason
for this was because they looked very serious during their partici-
pation, and because it was observed that in one occasion during
the ‘‘Puzzle” mini-game, participant P7 gave instructions to P4 on
how to move the puzzle piece on the screen, which apparently
annoyed P4, who replied in an annoyed tone ‘‘Do not tell me what
to do!”.
This result would be usually interpreted and reported as ‘‘most
participants (15/23, 65.21%) had a pleasant experience conducting
the CS activities; others (4/23, 17.39%) had an unpleasant experi-
ence, and the rest (4/23, 17.39%) were considered as having a neu-
tral experience”.
5.3. Neurophysiological UX evaluation
The results concerning the neural network classification of the
EEG signals into pleasant or unpleasant emotional states reported
indicate that (17/23, 73.91%) of the participants showed EEG sig-
nals classified with pleasant as a predominant emotion. The
remaining participants (P1, P2, P9, P11, P17, P19, 6/23, 26.09%)
were classified as predominantly showing unpleasant emotions.
Participants P1, P11 and P17 showed a little difference between
the signals classified as pleasant and unpleasant, with a slight pre-
dominance of the latter. Further, participants P2, P9 and P19
showed a higher percentage of unpleasant emotions (greater than
67%). Details of the results of the classification of EEG signals into
pleasant and unpleasant emotions by the neural network are sum-
marized in Fig. 7.
207
This result would be usually interpreted and reported as ‘‘most
participants (17/23, 73.91%) had a pleasant experience conducting
the CS activities, while the remaining participants (6/23, 26.09%)
had an unpleasant experience”.
6. Discussion
6.1. UX of the older adults while conducting the CS activity
Regarding the meaning of these results from the point of view
of the UX of older adults, as can be seen in Table 1, their user expe-
rience during their conducting the CS activity using the Abueparty
application could be considered as pleasant (from 65.22% to 100%).
Only 4–6 participants (from 17.39% to 26.09%) considered that
they had an unpleasant experience. It should be noted that this
value could increase up to 34.78% if those participants identified
as having a neutral experience turned out to have an unpleasant
one.
Regarding the meaning of these results from the point of the
accuracy of the proposed EEG signal neural network based classifi-
cation method; it is necessary to compare and take a more in-detail
look at the particular results from each of the used methods. To do
so, we will group the obtained emotions and analyze them as
ordered triplets (SR, QA, EC), with the results from the Self-report
(SR), Qualitative Analysis (QA), and EEG signal Classification (EC),
as shown in the first column of Table 2.
Firstly, we could consider valid results from the EEG signal clas-
sification method when the three methods arrive to the same
result. The only case where it happens is when the three methods
coincide in (P, P, P – Pleasant, Pleasant, Pleasant) emotions. Sec-
ondly, we could also consider valid results from the EEG signal
classification method when two methods arrive to the same result.
On the one hand, this occurs when the results of the EEG signal
classification coincides with those of qualitative analysis (either
(? P P) or (? U U)). The ? symbol in the triplet is a placeholder that
stands for any value. On the other hand, it happens when the
results of EEG signal classification coincides with those of self-
report (P ? P).
Additionally, we will also consider valid results from the EEG
signal classification method when its results coincide with the
results of any of the other methods. That is, this occurs when the
results of EEG signal classification coincides with those of qualita-
tive analysis (either (? P P) or (? U U)) and with those of self-report
(P ? P).
Accuracy results for the EEG signal classification method, con-
sidering the previous analysis, are presented next.
208 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209
Table 2
Detailed summary of results of the UX assessment studies, as triplets of emotions (X,
Y, Z).
Recognized emotions N %
PPP 14/23
PUP 2/23
PNP 1/23
PUU 2/23
PPU 1/23
PNU 3/23
P = Pleasant, U = Unpleasant, N = Neutral.
6.1.1. Coincidence among the three methods (P P P)
As can be seen in Table 2, there are 14 cases out of 23 where the
results are (P, P, P). This means that (i) participants expressed that
they had a pleasant experience, (ii) the expert observer noticed
that participants presented verbal expressions and gestures that
denoted a pleasant experience, and (iii) that the neural network
classified the EEG signals as a pleasant emotional state for that par-
ticipant. Then this represents an accuracy of 60.87% for the neural
network used in the neurophysiological UX evaluation.
6.1.2. Coincidence between qualitative analysis and EEG signal
classification (? P P) or (? U U)
As can be seen in Table 2, there are 16 cases where the results
from the neural network coincide with those of the qualitative anal-
ysis of the expert observer (14 (? P P) and 2 (? U U)). In this case, it
could be considered that the accuracy of the neural network for the
recognition of pleasant or unpleasant emotional states is 69.56%.
6.1.3. Coincidence between self-report and EEG signal classification (P
? P)
As can be seen in Table 2, there are 17 cases where the results
from the neural network coincide with those of the Self-report
evaluation (P ? P). In this case, it could be considered that the accu-
racy of the neural network for the recognition of pleasant or
unpleasant emotional states is 73.91%.
6.2. Coincidence between EEG signal classification and either self-
report or qualitative analysis (? P P) or (? U U) or (P ? P)
As can be seen in Table 2, there are 19 cases where the results
from the neural network coincide with those of either additional
method (17 (P ? P) or (? P P)) plus 2 more in (? U U). In this case,
it could be considered that the accuracy of the neural network for
the recognition of pleasant or unpleasant emotional states
increases to 82.61%.
6.2.1. Difference between EEG signal classification and either self-
report or qualitative analysis results (P P U) and (P N U)
The remaining 4 cases where the EEG signal classification
results does not coincide with any of the other methods represent
a 17.39%.
From these, there is 1 case (P P U) where the results of self-
report coincide with those of qualitative analysis but not with that
of EEG signal classification, and where from the qualitative analysis
we know that the EEG signal classification result is wrong. In the
video recording of the session of P19, s/he sees focused, active
and enjoying the activity (Pleasant). From here, we are certain that
in this case the neural network did not correctly recognized the
emotions. A possible explanation for this could be that the EEG sig-
nal was incorrectly captured, for instance, due to a bad placement
of the EEG headset in the head of the participant at capture time.
Finally, in the remaining 3 cases (P N U), we do not have enough
information as to assert whether or not the results coincide
between methods, as in self-report the participant stated that it
was a Pleasant experience, in EEG signal classification it was clas-
sified as Unpleasant, and the observer classified their emotions as
neutral (that is, the participants remained inexpressive during
60.87 their conducting the CS activity).
13.04
4.35
8.70 6.3. Validity of UX evaluation results vs. Cost of achieving the results
4.35
13.04 As a final point of the discussion, and considering the cost and
benefit of conducting an UX evaluation with any of the three pro-
posed methods, our results suggest that it would be advisable to
complement the results of the three methods in order to increase
their validity.
The rationale behind our suggestion includes:
i Although a qualitative analysis by an expert observer could
be the method that provides the most informed results about
the UX of a participant while conducting a certain activity, it
is clear that it is also the most effort expensive method, and
that the observer may introduce a bias due to his/her own
beliefs or previous experiences [10];
ii Although a self-report could be the least effort expensive
method to conduct an UX evaluation, we could not blindly
believe in its results, as it is well known that participants tend
to ameliorate the reported perception of the results because
they feel evaluated or because they say what they think the
researcher wants to hear [10];
iii Although a neurophysiological UX evaluation could be the
most objective method to conduct an UX evaluation, we can-
not blindly believe in its results (as there could be cases
where neurophysiological data is erroneously captured), and
completely ignore what the users could say in a self-report
or during an observation [12].
Further, based on the evidence obtained from this study, we
could suggest conducting firstly both self-report and EEG signal
classification evaluations and contrast the results (considering
them as the least expensive methods). Our data suggests that the
accuracy of these results would be around 70%.
Also, it would be advisable to consider the cases where the
results do not coincide between self-report and EEG signal classifi-
cation, and then conduct a qualitative analysis by an expert obser-
ver on these cases, not only to validate the data, but to be informed
about the causes of those differences. Based on the results of this
study, the evidence suggests that with an effort of 30% during
the qualitative analysis (on the results that do not coincide with
the other methods), the accuracy of the data could be increased
up to 80%.
Finally, it should be clear that it is necessary to conduct further
complementary UX studies using the three different methods, in
order to appropriately validate these suggestions no matter how
promising the evidence could be.
7. Conclusions and future work
In this work we report the results of a study to evaluate the fea-
sibility of using a neural network previously obtained to assess
pleasant and unpleasant emotions of older adults. The neural net-
work was trained to recognize known a priori emotional states
from EEG signals evoked by visual stimuli using IAPS pictures.
The challenge in this study was that the target emotional states
will be evoked by the UX of older adults while using a CS applica-
tion, that is considering not only visual stimuli, but also auditory
and cognitive processing stimuli.
 V. Meza-Kubo et al. / Journal of Biomedical Informatics 62 (2016) 202–209
The results from our study suggest that it is feasible to assess
the UX of older adults while they use a CS application using the
neural network trained to recognize pleasant and unpleasant emo-
tions from EEG signals. The recognition accuracy of the proposed
method ranges from 60.87% to 82.61%, which is in the same range
of results from previous related studies [23–26,28].
Also, our study results suggest that it is feasible to conduct UX
evaluations following a complementing approach among the 3 pre-
sented methods, in order to increase the accuracy of the results
while reducing the costs associated to actually conducting the
three studies. To do so, researchers may conduct complete (least
cost demanding) Self-report and Neurophysiological UX evalua-
tions, contrast results, and conduct (most cost demanding) qualita-
tive analysis by expert observer only on the results that do not
coincide between the two former evaluations (a possible reduction
of 70% on the effort required to conduct the latter evaluation).
In both cases, it is necessary to conduct further, and larger, com-
plementary UX studies using the three different methods, in order
to appropriately validate these results.
For this reason, as future work, we propose to retrain the neural
network using not only EEG signals evoked with visual stimuli, but
also including auditory and cognitive processing generated stimuli.
Then, we will use the trained neural network to recognize pleasant
and unpleasant emotional states to evaluate specific segments of
interaction (for instance, by single mini-game). Furthermore, we
propose to reuse these techniques on other CS and rehabilitation
applications that we have developed, to validate the proposed
complementing approach.
Conflict of interest
None declared.
Acknowledgements
We acknowledge the support of UABC, in the form of Programa
de Servicio Social 212 and Internal project number 231; and of
CONACYT in the form of scholarship number 538130 for the third
author. We also acknowledge the elderly participants from Ense-
nada and Mexicali, B.C., Mexico for their support and participation
in the study.
References
[1] R.C. Petersen, R. Doody, A. Kurz, R.C. Mohs, J.C. Morris, P.V. Rabins, K. Ritchie,
M. Rossor, L. Thal, B. Winblad, Current concepts in mild cognitive impairment,
Arch. Neurol. 58 (12) (2001) 1985–1992, http://dx.doi.org/10.1001/
archneur.58.12.1985.
[2] P. Rashidi, A. Mihailidis, A survey on ambient-assisted living tools for older
adults, IEEE J. Biomed. Health Inform. 17 (3) (2013) 579–590, http://dx.doi.org/
10.1109/JBHI.2012.2234129.
[3] W. Ijsselsteijn, H.H. Nap, Y. de Kort, K. Poels, Digital game design for elderly
users, in: Proceedings of the 2007 Conference on Future Play, ACM, 2007, pp.
17–22, http://dx.doi.org/10.1145/1328202.1328206.
[4] C. Lallemand, G. Gronier, V. Koenig, User experience: a concept without
consensus? Exploring practitioners perspectives through an international
survey, Comput. Human Behav. 43 (2015) 35–48, http://dx.doi.org/10.1016/j.
chb.2014.10.048.
[5] M. Hassenzahl, User experience (UX): towards an experiential perspective on
product quality, in: Proceedings of the 20th International Conference of the
Association Francophone d’Interaction Homme-Machine, ACM, 2008, pp. 11–
15, http://dx.doi.org/10.1145/1512714.1512717.
[6] E.L.-C. Law, V. Roto, M. Hassenzahl, A.P. Vermeeren, J. Kort, Understanding,
scoping and defining user experience: a survey approach, in: Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2009,
pp. 719–728, http://dx.doi.org/10.1145/1518701.1518813.
[7] H.L. O’Brien, E.G. Toms, What is user engagement? A conceptual framework for
defining user engagement with technology, J. Am. Soc. Inform. Sci. Technol. 59
(6) (2008) 938–955, http://dx.doi.org/10.1002/asi.20801.
[8] M. Hassenzahl, N. Tractinsky, User experience – a research agenda, Behav.
Inform. Technol. 25 (2) (2006) 91–97, http://dx.doi.org/10.1080/
01449290500330331.
209
[9] A.P. Vermeeren, E.L.-C. Law, V. Roto, M. Obrist, J. Hoonhout, K. Väänänen-
Vainio-Mattila, User experience evaluation methods: current state and
development needs, in: Proceedings of the 6th Nordic Conference on
Human–Computer Interaction: Extending Boundaries, ACM, 2010, pp. 521–
530, http://dx.doi.org/10.1145/1868914.1868973.
[10] L. Arhippainen, M. Tähti, Empirical evaluation of user experience in two
adaptive mobile application prototypes, in: Proceedings of the 2nd
International Conference on Mobile and Ubiquitous Multimedia, 2003, pp.
27–34.
[11] R.L. Mandryk, K.M. Inkpen, T.W. Calvert, Using psychophysiological techniques
to measure user experience with entertainment technologies, Behav. Inform.
Technol. 25 (2) (2006) 141–158, http://dx.doi.org/10.1080/
01449290500331156.
[12] G.L. Ahern, G.E. Schwartz, Differential lateralization for positive and negative
emotion in the human bra, in: EEG spectral analysis, Neuropsychologia 23 (6)
(1985) 745–755, http://dx.doi.org/10.1016/0028-3932(85)90081-8.
[13] D. Sammler, M. Grigutsch, T. Fritz, S. Koelsch, Music and emotion:
electrophysiological correlates of the processing of pleasant and unpleasant
music, Psychophysiology 44 (2) (2007) 293–304, http://dx.doi.org/10.1111/
j.1469-8986.2007.00497.x.
[14] G.G. Knyazev, J.Y. Slobodskoj-Plusnin, A.V. Bocharov, Gender differences in
implicit and explicit processing of emotional facial expressions as revealed by
event-related theta synchronization, Emotion 10 (5) (2010) 678–687, http://
dx.doi.org/10.1037/a0019175.
[15] D. Mathersul, L.M. Williams, P.J. Hopkinson, A.H. Kemp, Investigating models
of affect: relationships among EEG alpha asymmetry, depression, and anxiety,
Emotion 8 (4) (2008) 560–572, http://dx.doi.org/10.1037/a0012811.
[16] I. Carrillo, V. Meza-Kubo, A.L. Morán, G. Galindo, E. García-Canseco, Processing
EEG signals towards the construction of a user experience assessment method,
in: Ambient Intelligence for Health, Lecture Notes in Computer Science, vol.
9456, Springer, 2015, pp. 281–292, http://dx.doi.org/10.1007/978-3-319-
26508-7_28.
[17] B.L. Fredrickson, M.F. Losada, Positive affect and the complex dynamics of
human flourishing, Am. Psychologist 60 (7) (2005) 678–686, http://dx.doi.org/
10.1037/0003-066X.60.7.678.
[18] E. Harmon-Jones, P.A. Gable, C.K. Peterson, The role of asymmetric frontal
cortical activity in emotion-related phenomena: a review and update, Biol.
Psychol. 84 (3) (2010) 451–462, http://dx.doi.org/10.1016/j.
biopsycho.2009.08.010.
[19] I. Winkler, M. Jäger, V. Mihajlovic, T. Tsoneva, Frontal EEG asymmetry based
classification of emotional valence using common spatial patterns, World
Acad. Sci. Eng. Technol. 45 (2010) 373–378.
[20] R.J. Davidson, P. Ekman, C.D. Saron, J.A. Senulis, W.V. Friesen, Approach-
withdrawal and cerebral asymmetry: emotional expression and brain
physiology: I, J. Personality Social Psychol. 58 (2) (1990) 330–341.
[21] R.J. Davidson, N.A. Fox, Frontal brain asymmetry predicts infants’ response to
maternal separation, J. Abnormal Psychol. 98 (2) (1989) 127.
[22] M. Kostyunina, M. Kulikov, Frequency characteristics of EEG spectra in the
emotions, Neurosci. Behav. Physiol. 26 (4) (1996) 340–343, http://dx.doi.org/
10.1007/BF02359037.
[23] K. Ishino, M. Hagiwara, A feeling estimation system using a simple
electroencephalograph, IEEE International Conference on Systems, Man and
Cybernetics, 2003, vol. 5, IEEE, 2003, pp. 4204–4209, http://dx.doi.org/
10.1109/ICSMC.2003.1245645.
[24] P.C. Petrantonakis, L.J. Hadjileontiadis, Emotion recognition from EEG using
higher order crossings, IEEE Trans. Inform. Technol. Biomed. 14 (2) (2010)
186–197, http://dx.doi.org/10.1109/TITB.2009.2034649.
[25] Q. Zhang, M. Lee, Analysis of positive and negative emotions in natural scene
using brain activity and GIST, Neurocomputing 72 (4) (2009) 1302–1306,
http://dx.doi.org/10.1016/j.neucom.2008.11.007.
[26] J. Chai, Y. Ge, Y. Liu, W. Li, L. Zhou, L. Yao, X. Sun, Application of frontal EEG
asymmetry to user experience research, in: Engineering Psychology and
Cognitive Ergonomics, Lecture Notes in Computer Science, vol. 8532, Springer,
2014, pp. 234–243, http://dx.doi.org/10.1007/978-3-319-07515-0_24.
[27] H. Gürkök, G. Hakvoort, M. Poel, Evaluating user experience with respect to
user expectations in brain-computer interface games, in: Proceedings of the
5th International Brain-Computer Interface Conference, BCI 2011, Verlag der
Technischen Universität Graz, 2011, pp. 348–351.
[28] Y.-P. Lin, C.-H. Wang, T.-P. Jung, T.-L. Wu, S.-K. Jeng, J.-R. Duann, J.-H. Chen,
EEG-based emotion recognition in music listening, IEEE Trans. Biomed. Eng. 57
(7) (2010) 1798–1806, http://dx.doi.org/10.1109/TBME.2010.2048568.
[29] V. Meza-Kubo, A. Morán, Abueparty: an everyday entertainment system for
the cognitive wellness of the worried-well, in: 5th International Symposium of
Ubiquitous Computing and Ambient Intelligence, UCAMI, 2011.
[30] H. Jasper, Report of the committee on methods of clinical examination in
electroencephalography, Electroencephalogr. Clin. Neurophysiol. 10 (2) (1958)
370–375.
[31] V. Meza-Kubo, A.L. Morán, UCSA: a design framework for usable cognitive
systems for the worried-well, Personal Ubiquitous Comput. 17 (6) (2013)
1135–1145, http://dx.doi.org/10.1007/s00779-012-0554-x.
[32] J. Cockrell, M. Folstein, Mini-mental state examination (MMSE),
Psychopharmacol. Bull. 24 (4) (1988) 689–692.
[33] A.L. Morán, V. Meza-Kubo, Evaluating the user experience of a cognitive
stimulation tool through elders’ interactions, in: Intelligent Environments
(Workshops), IOS Press, 2012, pp. 66–77, http://dx.doi.org/10.3233/978-1-
61499-080-2-66.