See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/323935725

Analyzing User Emotions via Physiology Signals

Article · December 2017

CITATIONS READS

13 3,981

6 authors, including:

Bohdan Myroniv Cheng-Wei Wu

National Chiao Tung University 46 PUBLICATIONS   1,950 CITATIONS   
2 PUBLICATIONS   14 CITATIONS   
SEE PROFILE
SEE PROFILE

Yi Ren Albert Christian

University of East Anglia National Chiao Tung University
67 PUBLICATIONS   415 CITATIONS    7 PUBLICATIONS   19 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Eye On You: Fusing Gesture Data from Depth Camera and Inertial Sensors for Person Identification View project

Queueing and Performance Evaluation View project

All content following this page was uploaded by Albert Christian on 27 March 2018.

The user has requested enhancement of the downloaded file.

 Data Science and Pattern Recognition 2017
c ISSN 2520-4165
Ubiquitous International Volume x, Number 0x, March 2017

Analyzing User Emotions via Physiology Signals

Bohdan Myroniv ∗ , Cheng-Wei Wu ∗

, Yi Ren †
, Albert Christian ∗
, Ensa Bajo ∗
and Yu-Chee Tseng ∗

∗
Department of Computer Science
National Chiao Tung University, Hsinchu, Taiwan, ROC
cww0403@nctu.edu.tw
†
School of Computing Science
University of East Anglia, Norwich, UK

Abstract. Recently some studies have shown that the major influential factor of our
health is not only physical activities, but the states of our emotion that we experience
through our daily life, which continuously build our behavior and affect our physical health
significantly. Therefore, emotion recognition draws more and more attention for many
researchers in recent years. In this paper, we propose a system that uses off the-shelf
wearable sensors, including heart-rate, galvanic skin response, and body temperature sen-
sors, to read physiological signals from the user and apply machine learning techniques
to recognize emotional states of the user. These states are key steps, toward improving
not only the physical health but also emotional intelligence in advance human-machine
interaction. In this work, we consider three types of emotional states and conducted ex-
periments on real-life scenarios, achieving highest recognition accuracy of 97.31%.

Keywords: Emotion recognition, Machine learning, Physiological signals, Wearable

devices.

1. Introduction. The Internet of Things (IoT) [30] have rapidly become one of the
most popular research topics due to its wide range of application scenarios. In general,
these applications can be divided into three categories: (1) smart home [32], (2) smart
transportation [33], and (3) e-health [34], which relies on wearable sensors. Wearable tech-
nology [31] is often touted as one of the important applications of the IoT, which consists
of low-cost and low-power sensors available in markets. The existing wearable applications
are in their early phase and currently dominated by physical activities tracking devices.
So-called smart-band can monitor some of our daily activities, such as walking, running,
cycling, and swimming. It also can track when and how long we sleep. Some of these act
as personal trainers and count how many calories was burned. Although physical health
monitoring applications are booming, only few studies have been conducted on emotional
health.

Emotion is an integral part of our health and has a huge impact on our bodies, mental
health and behavior. Weak emotional can affect our immune system, making us more
likely to get colds and other minor infections. In reference to American Psychological As-
sociation (APA) [10], more than 53% of Americans report personal health problems as a
source of stress. This issue of stress which has left unchecked can contribute many health
problems, such as high blood pressure, heart disease, obesity and diabetes. According to
a study conducted by the American College Health Association (ACHA) [11], a consid-
erable proportion of students said that mental health problems affected their academic

1
2 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

performance, causing them to dropping courses, or receiving low grades in the classes.

Most of the existing studies of emotion recognition are based on visual camera-based
approaches [2, 3, 6], audio speech-based approaches [1, 4, 5, 7] and physiology-based emo-
tion recognition [27, 28]. The visual camera-based approaches that uses image processing
technologies to detect users’ emotions. However, the approach requires the users to be
well seen by the cameras. Moreover the approaches recognition may fall significantly.
With regards to the audio speech-based approaches [1, 4, 5, 7], the main idea of these
approaches is to use the speakers’ loudness, tone and intonation patterns in a speech to
detect their emotions. However, conversational properties may vary in different cultures
and nationalities, which may lead to worse recognition results from the system. Recently
many studies have shown that less attention was paid to physiological signals analysis
for emotion recognition using wearable devices[27, 28]. However, detecting users’ emo-
tions using physiological signals required collecting reliable data. In the sense that the
autonomous nervous system cannot be controlled consciously by users themselves. Such
data will not be affect by the factors, such as light requirements in video-based approaches
or cultural peculiarities in audio-based approaches.

Figure 1. a)Heart Rate sensor; b)Galvanic skin response sensor; c) Tem-

perature sensor

In this paper, we use three non-invasive sensors as shown in Fig. 1: Heart Rate sensor
to sense user’s pulse, Beats Per Minute (BPM), Galvanic Skin Response (GSR) sensor to
sense the skin conductivity to modulates the amount of sweat secretion from sweat glands,
and body Temperature sensor (T) to sense high and low temperate values of the user’s
body. Then, we apply sliding window-based segmentation method to segment collected
data and extract candidate features from the segments. After that, we feed extracted
features to classifiers to identify the type of users’ emotional states. We implement the
prototype of the proposed methods on Arduino platform and evaluate the performance
of six classification algorithms to train emotion classification models: Random Tree, J48,
Nave Bayes, SVM, KNN and Multilayer Perceptron Neural Network.
 Analyzing User Emotions via Physiology Signals 3

We collected data from 10 participants using Geneva Affective Picture Database (GAPED)
and International Affective Picture System (IAPS) as triggering mechanism for experi-
mental investigations of emotion. These databases consist of a set of labeled pictures
which were designed to stimulate particular emotions. The results show high accuracy
above 97% of user’s emotional conditions recognition using low-cost sensors that are avail-
able in current wearable devices. In recognition stage, we consider three general emotional
states: Negative, Neutral and Positive.
The rest of this paper is organized as follows: Section 2 reviews the related work.
Section 3 introduces the details of the proposed method algorithms and experimental
methodology. Data analysis, performance evaluation and prototype implementation of
the proposed system are described in Section 4. Finally, Section 5 gives conclusion and
future works.

2. Related Works. In this section we discuss about existing studies which fall into three
categories: emotion recognition based on video, audio and physiology approaches:

2.1. Video-based emotion recognition. The approaches in this category can be di-
vided into two types: emotion recognition (ER) based on facial expression [18, 19, 20]
and body movements [17, 21]. Both of them rely on cameras and image processing tech-
nologies to detect emotional states. However, in order to detect emotional states, user
has to be in a range of camera vision and even if so, user has to be well seen. In other
words, such system will output an unreliable results or even will not work in a poorly lit
environment. Besides, the user can consciously control facial expression to false classifi-
cation results. In addition, camera-based emotion recognition systems are stationary and
cannot be properly used as a day-long wearable solution.

2.2. Audio-based emotion recognition. Besides information that we intend to say,

our speech contain information about our emotions [22]. Different methods and algorithms
were designed for audio-based emotion recognition, such as [23, 24]. These methods
rely on microphones and based on speakers loudness, tone and intonation. However,
conversational properties vary in different cultures, nationalities or even differ from human
being to human being [7], so the universality of such systems be low and may tight
to a specific area with a specific conversation properties. Besides, in order to detect
users’ emotional states, users have to constantly or continuously talking, whics is very
inconvenient in some applications.

2.3. Physiology-based emotion recognition. Using biosensors to detect emotional

states becomes more popular among researchers in recent years. These biosensors are
monitoring physiological signals generated by an autonomic nervous system of human,
which produces visceral sensations that shape emotional experience. Autonomous nervous
system cannot be controlled consciously and it allows to collect reliable data, which is
not affected by factors, such as cultural and national diversities in audio-based approach
or range of vision and light requirements in video-based approach. Besides, both of
these approaches require users to do some actions to know their emotional state. Studies
[27, 28] take advantage of using biosensors for emotion recognition. These studies, use
high cost and precision sensors and achieve high recognition accuracy of 96.6% [26]. The
system developed in [28] achieves even higher accuracy. They use electrocardiogram
(ECG) sensor to read heart’s electrical activity, electromyograph (EMG) sensor to read
electrical activity produced by skeletal muscles, electroencephalogram (ECG) sensor to
detect electrical activity in brain and respiration rate sensor to detect how fast and deep
4 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

we are breathing. However, they are costly and cannot be properly integrated in smart-
watches and fitness bands.

3. The Proposed Method. In this section, we introduce the proposed methods and
explain the procedure of physiological signals processing for obtaining user’s emotional
states.

3.1. System Architecture Overview. In this subsection, we explain the procedure

of physiological signals processing for obtaining user’s emotional state. In our research,
we use three sensors which are connected to Arduino Yun micro controller to collect
users’ data. The user’s data is collected with 10Hz sampling rate frequency, that further
processed three physiological signals collected from non-invasive sensors below:
1. Heart Rate (Grove Ear-clip Heart Rate Sensor) is used to measure amount of heart
beats per minute. Heart rate changes while different emotions [13] or even listening
to different music styles [14].
2. Galvanic Skin Response (Grove GSR sensor) is used to measure electrical conduc-
tance of the skin. The purpose of the usage of GSR sensor because emotion resulting
the stimulus to our sympathetic nervous system. The stimulus itself has impact on
our body will become sweaty which caused by the secretions of the sweat glands [15].
3. Temperature sensor (MLX90614) is used to measure body temperature. Body tem-
perature changes while different emotions are triggered [16].

Figure 2. The Input-Process-Output diagram of the proposed system

The collected sensor data are sent to a server via Bluetooth for further processing. Fig.
2 shows the Input-Process-Output diagram of the proposed system. In this work, we
consider the recognition of three states of emotion; positive, neutral and negative. Data
generated from the sensors go through four phases before finally recognizing an individ-
ual user’s emotional state, which are pre-processing, segmentation, feature extraction and
emotion classification.

Pre-processing Phase: In this phase, we observe that collected data tend to have
missing values during a short period (i.e., one to two seconds) just after the connec-
tion stability between the devices are established. In order to address such problem,
we therefore cut off the first two seconds’ data from the collected data to avoid the
interference of the missing data. Such issue may happen due to the ongoing estab-
lishment of the Bluetooth connection.
 Analyzing User Emotions via Physiology Signals 5

Figure 3. Labeling process for sensor data

Table 1. The extracted candidate features

ID Feature Formula

1
Pn
1 Mean µ(A) = n i=1 ai

1
Pn
2 Variance V ar(A) = n i=1 (ai − µ(A))2

1
Pn 2
3 Energy Eng(A) = n i=1 (ai )

1
Pn
4 Average Absolute Difference Aad(A) = n−1 i=2 | ai−1 − ai |

1
Pn
5 Average Absolute Value Aav(A) n−1 i=1 | ai |

1 Pn 3
i=1 (ai −µ(A))
6 Skewness Skew(A) = n
1 Pn
[ n i=1 (ai −µ(A)) ]3/2
2

1 Pn
(a −µ(A))4
7 Kurtosis Kurt(A) [ 1n Pni=1(a i−µ(A))2 ]2
n i=1 i

1
Pn
8 Zero Crossing Rate ZCR = 2 i=2 | sign(ai ) − sign(ai−1 ) |
Pn
i=2 |sign(ai −µ(A))−sign(ai−1 −µ(A))|
9 Mean Crossing Rate M CR = 2

q P
1 n
10 Root Mean Square Rms(A) = n i=1 a2i

Feature Extraction Phase: In this phase, we extract candidate features from the
sensor signals in each segment. For instance, let A =< a1 , a2 , ..., an > be a time
series data of a sensor in a segment, where ai (1 ≤ i ≤ n) is the ith sample in A. The
extracted candidate features are shown in Table 1.

• Mean - It measures the central tendency of the data.

6 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

• Variance - It represents the expected squared deviation from the mean.

• Energy - It measures to find out the part of data where the activities may occur.
• Average Absolute Difference- It is a measure of statistical dispersion equal
to the average absolute difference of two independent values drawn from a prob-
ability distribution.
• Average Absolute Value- It is the average of the absolute value or the average
of modulus | x | of a real number x is the non-negative value of x without regard
to its sign. Namely, | x | = x for a positive x, | x | = x for a negative x (in which
case x is positive), and | 0 | = 0. For example, the absolute value of 3 is 3, and
the absolute value of 3 is also 3. The absolute value of a number may be thought
of as its distance from zero.
• Skewness It measures the symmetry, or more precisely, the lack of symmetry.
• Kurtosis - It measures the tailedness, whether the data are heavy-tailed or
light-tailed relative to a normal distribution.
• Zero Crossing Rate (ZCR)- It is a measure that reflects the times that signs
of two adjacent values in A change from positive to negative or vice versa where
sign() returns 1 if one of the two inputs is positive and another one is negative;
otherwise it returns 0.
• Mean Crossing Rate(MCR)- It is a measure that reflects how many times
the sign of two adjacent values in A cross mean.
• Average Absolute Value - It measures data’s absolute deviations around the
data’s mean.
Above all the most important feature we consider is zero-crossing rate feature,
which define the change stage of our emotional state from positive to negative to
neutral or back.

Figure 4. Data segmentation and overlapping sliding window method

Segmentation Phase: Fig. 4 describes how we apply overlapping sliding window-

based segmentation to segment the collected sensor data. The segmented window
size is set to θ(θ > 0) seconds and the size of overlapping is set to δ(0 ≤ δ ≤ θ)
seconds. In the experiments, we evaluate the effect of varied window sizes and over-
lapping sizes on the recognition results of the proposed system.

Emotion Classification Phase: In this phase, the extracted candidate features are
used to build classifiers using classification algorithms in Weka [12]. We consider
the following six different types of classification algorithms: K -Nearest Neighbor
(abbr. KNN), J48 Decision Tree (abbr. J48), Nave Bayes (abbr. NB), Random
Tree (abbr. RT), Support Vector Machine (abbr. SVM) and Multilayer Perceptron
Neural Networks (abbr. MP).
3.2. Experiment Methodology. In experiment, we recruit ten participants, which are
divided into two groups: seven males and three females. Their ages are between twenty-
two and thirty. Fig. 3 shows how we label sensor data using emotional triggering database.
 Analyzing User Emotions via Physiology Signals 7

Table 2. The accuracy of the group models under different sizes of over-
lapping windows

RT J48 NB KNN SVM MP

0.5 sec. 84.68 81.92 39.63 78.37 49.54 77.92
1.0 sec. 79.51 84.33 38.95 79.11 42.57 80.72
1.5 sec. 85.31 86.01 38.11 77.62 44.40 76.92
2.0 sec. 85.58 85.88 44.44 84.08 48.34 83.18
2.5 sec. 86.53 90.77 44.38 85.28 46.38 83.54
3.0 sec. 87.41 89.27 41.38 84.58 45.28 85.15
3.5 sec. 89.82 88.62 49.11 87.27 49.85 88.47
4.0 sec. 93.11 92.31 46.30 92.11 48.41 88.92
4.5 sec. 94.22 95.46 52.14 97.31 48.41 91.78

All the experiments are conducted in separate room. We request the participants to leave
their phones out during the experiments. Emotion triggering is a very important part of
the experiment. In the emotional triggering, we use two affective emotion photo databases
in our experiments. The first one is Geneva Affective Picture Database (GAPED) [8] pro-
vided by Swiss Center for Affective Sciences, and the second one is International Affective
Picture System (IAPS) [9] provided by Center for The Study of Emotion and Attention.
We choose twenty photos for each type of emotional states (i.e., positive, neutral, and
negative) from the above mentioned databases. Each photo was displayed for five sec-
onds. We have tried different orders of data collection, i.e. different emotion stimuli were
shown in different days, and different consistency. Such approach was applied to minimize
dependencies of one emotional condition from another one.

We held our experiments three times. The data collected from the first experiment
was never used in the processing. This is because, we realize that participants themselves
comments that they felt nervous and strange due to the bunch of sensors and cables
connected to their arms, which would influence participants’ emotion and lead to noise
data. Therefore, the data collected from the first experiment is not used. However after
the first experiment, our participants were more familiar with the sensors setup and it
was not disturbing for them to wear these sensors in the next data collection rounds.
Therefore, the data collected from the second and third experiments are used in the
processing.

4. Performance Evaluation.

4.1. Experimental Evaluation. In this subsection, we test the analysis and recognition
performance of the proposed sensing system using different types of classifiers, including
KNN, J48, NB, RT, SVM, and MP. During the testing process, we apply 10 fold cross-
validation to do the performance evaluation. We consider two kinds of models called group
model and personalized model. In group model, the emotion classifier is constructed
by considering all the participants’ physiology signals in the training phase. While in
8 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

Table 3. Comparing results of six classification algorithms and five seg-

ment sizes for group model

Group Dataset
Accuracy of each segment size
Classifiers
1 sec. 2 sec. 3 sec. 4 sec. 5 sec.
RT 83.69% 82.90% 82.58% 73.29% 80.52%
J48 90.35% 84.49% 86.48% 87.25% 80.55%
NB 39.46% 39.16% 45.34% 41.62% 34%
KNN 72.76% 74.75% 76.57% 80.45% 79.56%
SVM 47.11% 42.94% 42.64% 44.84% 49%
MP 86.28% 82.30% 79.87% 77.28% 72%

personalized model, the classifier is constructed by considering individual participants’

physiology signals in the training phase.

Figure 5. The accuracy of each personalized model.

4.2. Data Analysis. Table 2 describe the accuracy analysis of the group models under
the different size of segmentation. In this experiment, the overlapping size δ is set to 0
seconds. In Table 2, we can see that the J48 classifier performs better than the other five
classifiers. Besides, when the window size θ is set to 1 second, the J48 classifier achieves
90.35% recognition accuracy. In Table 2, the SVM and NB classifiers are not very effective,
their average accuracies are 45.59% and 39.91%, respectively. Then, we test the effect of
δ on the overall accuracy of the proposed system. In this experiment, the window size θ
 Analyzing User Emotions via Physiology Signals 9

Table 4. Detailed accuracy by class of the best combination of classifier,

segment size and overlapping window size
Classifier: Group model / 5sec.segm / 4.5 sec. overlap
KNN Training Testing
Class Precision Recall F-Measure Accuracy Precision Recall F-Measure Accuracy
Positive 0.984 0.980 0.982 98.08 0.973 0.974 0.974 97.31
Neutral 0.974 0.979 0.976 98.08 0.961 0.969 0.965 97.31
Negative 0.985 0.983 0.984 98.08 0.987 0.976 0.981 97.31

is set to 5 seconds. Table 2 shows the accuracy of the group models when the overlapping
size is varied from 0.5 seconds to 4.5 seconds. As shown in Table 3, overlapping size is
set to 4.5 seconds, the KNN classifier has the best recognition accuracy, which is up to
97.31%. In the next experiment, we fix the parameters θ and δ to 5 seconds and 4.5
seconds, respectively, and evaluate the precision, recall and F-Measure of the proposed
system using KNN classifier for each class. Table 4 shows the experimental results. As
shown in Table 4, the system using KNN classifier has good recognition result for each
class. Furthermore, we train model for each participate to test the recognition accuracy
for personalized models. Because there are ten participants, ten personalized models were
built. In this experiment, we set the parameters θ and δ to 5 seconds and 4.5 seconds,
respectively, and consider the KNN classifier. Fig. 4 shows the classification results for
each personalized model of each user. As shown in Fig. 4, the average accuracy of all
the personalized models is 97.78%. Moreover, the maximum and minimum accuracies are
98.2% and 96.8%, respectively.
Table 4 shows the accuracy of the group models under the different segmentation size,
while Fig. 5 shows the accuracy of each personalized model. In this experiment, the
overlapping size δ is set to 0 seconds. In Table 3, the J48 classifier performs better than
the other five classifiers. Besides, when the window size θ is set to 1 second, the J48
classifier achieves 90.35% recognition accuracy. In Table 3, the SVM and NB classifiers
are not very effective, the average accuracies are 45.59% and 39.91%, respectively.

Figure 6. System Prototype Components

Fig. 7, shows pattern rule for emotion classification. It describe that accuracy grows
higher as the overlapping size increases. Hence, the decision of combining more segments
and overlapping size is made as it helps in generating higher accuracy among the classifiers.
Table 5 shows the improvement of accuracies over 10 seconds segment in relation to
10 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

Figure 7. The improve four classifiers accuracy in overlapping size dependency

Table 5. Accuracy over 10 segment and overlapping size

Segment Size (sec.) Overlapping Size (sec.) Accuracy (%)

1 0.5 78.51
2 1.5 87.97
3 2.5 93.53
4 3.5 96.46
5 4.5 97.31
6 5.5 97.75
7 6.5 98.61
8 7.5 99.11
9 8.5 98.81
10 9.5 99.65

overlapping size. In the above table, it presents that when the dataset is small then the
prediction will be inaccurate. On the contrary, when the dataset is huge then inaccurate
predictions are observe due to the long transition between emotional states. Therefore,
in the presented system the optimal combination in 5 seconds segment along with 4.5
seconds overlapping size to achieve an accurate prediction. Fig. 8 shows the comparison
of accuracy between four best classification algorithms: Random Tree (RT), Decision Tree
(J48), k-Nearest Neighbor (KNN), Multilayer Perceptron (MP) and those with sliding
window (SW) method. The usage of sliding window method increases accuracy among
all classifiers averagely for 5.5% as shown in table 6. The performance of classifiers with
each of five segments and overlapping windows is shown on Table 7 for segment size
θ = 1, 2, 3, 4and5 respectively.
4.3. Prototype Implementation. In this sub-section, we introduce the prototype re-
sult of the proposed system. Fig. 6 shows the detail of system prototype components. We
developed an Android application to recognize users’ emotional state online. It consists
of Arduino side and Android side. On the Arduino side, heart rate, body temperature
and GSR sensors will continuously collect users’ physiology signals and send them to
the smart-phone via Bluetooth. On the Android side, the collected data will go through
 Analyzing User Emotions via Physiology Signals 11

Table 6. Average accuracy improvement among four classifiers with and

without using sliding window method

No SW Method With SW method Improvement

MP 79.55% 84.07% 4.52%
KNN 76.82% 85.08% 8.26%
RT 80.59% 87.35% 6.76%
J48 85.82% 88.29% 2.47%

Figure 8. Comparison of improve accuracy between four classifiers with

and without sliding window method

pre-processing, segmentation, feature extraction and emotion classification phases. Fig.

9 shows the prototype of the implemented App. On the first screen of the App user can
click ”Connect” button to connect the App with the Arduino device.

Figure 9. The User Interface of the implemented system prototyping

12 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

Table 7. Average, minimum and maximum accuracies of group dataset

over six classifiers, five segment and without overlapping

Classifier Min Average Max

NB 34% 39.92% 45.34%
SVM 42.64% 45.31% 49%
KNN 72.76% 76.82% 80.45%
RT 73.29% 80.59% 83.69%
MP 72% 79.55% 86.28%
J48 80.55% 85.82% 90.35%

When Bluetooth connection is that established, the App will go to the second screen,
where user can see three boxes shown their detected heart beats per minute (show. as
Heart rate in the interface), GSR (abbr. as Skin resp. in the interface) and temperature
(abbr. as Tempr. in the interface) data from the interface in a real-time fashion. Detected
user’s emotional state is shown in the upper right side of the App. Online recognition is
done by using the pre-trained classifier inside the smart-phone.

4.4. System Performance Testing. We perform the testing of our system in real-life
scenario. We consider different actions to see what emotions will be triggered and see if
they match our expected results. Fig. 10 demonstrate a user doing different actions, such
as: reading paper, scrolling Facebook, watching funny video, reading article, watching
negative pictures.In our performance testing, we can see a user reading article about
newly released laptop, and on left side of the screen shows the screen cast from the
prototype application in on-line fashion. Here the output of the App shows ”Positive”,
which means user’s emotional state is positive.
Furthermore, we would like to point out that while transition between different emo-
tional states. Our usually takes approximately 45 - 60 second to be perfectly steady and
accurate on a given emotional state. For that been the case, since heart rate, tempera-
ture and other physiological signals cannot change quickly, which might lead to a situation
where It takes some time for our bodies to increase / decrease its characteristics.

5. Conclusions and Future Work. In this work, we consider the heart rate, body
temperature and galvanic skin response sensors to design a wearable sensing system for
effective recognition of users’ emotional states. The proposed system allows recognizing
three types of emotions, including positive, neutral and negative, in an on-line fashion.
We apply the machine learning technology to process the physiology signals collected from
the user. The process consists of four main phases: data pre-processing, data segmen-
tation, feature extraction, and emotion classification. We implement the prototype of
the system on Arduino platform with Android smart-phone. Extensive experiments on
real-life scenarios show that the proposed system achieves up to 97% recognition accuracy
when it adopts the k -nearest neighbor classifier. In the future work, we will consider more
types of user’s emotional states and consider to find a correlation between these emotions
and physical activities for more diverse and novel applications. Below is the demo video,
source code and dataset of our system (https : //goo.gl/KhhcyA).
 Analyzing User Emotions via Physiology Signals 13

Figure 10. A User is reading an article about newly released laptop

Acknowledgment. This work was support in part by the Ministry of Science and Tech-
nology of Taiwan, ROC., under grant MOST 104-2221-E-009-113-MY3, 105-2221-E-009-
101-MY3, 105-2218-E-009-004.

REFERENCES

[1] Amin S., Andriluka M., Bulling A., Mller M. P., Verma P.,“Emotion recognition from embedded bod-
ily expressions and speech during dyadic interactions,” IEEE International Conference on Affective
Computing and Intelligent Interaction, pp. 663–669, 2015.
[2] Bulut M., Busso C., Deng Z., Kazemzadeh A., Lee C.M., Lee S., Narayanan S., Neumann U., Yildirim
S.,“Analysis of emotion recognition using facial expres-sions, speech and multimodal information,”
International Conference on Multimodal Interfaces, pp. 205–211, 2004.
[3] Bilakhia S., Cowie R., Eyben F., Jiang B., Pantic M., Schnieder S., Schuller B., Smith K., Val-
star M.,“The continuous audio/visual emotion and depression recognition challenge,” International
Workshop on Audio/Visual Emotion Challenge, pp. 3–10, 2013.
[4] Aswathi E., Deepa T. M., Rajan S., Shameema C. P., Sinith M. S.,“Emotion recognition from
audio signals using Support Vector Machine,” IEEE Recent Advances in Intelligent Computational
Systems, pp. 139–144, 2015.
[5] Dai K., Fell J. H., MacAuslan J.,“Recognizing emotion in speech using neural net-works,” Interna-
tional Conference on Telehealth/Assistive Technologies, pp. 31–36, 2008.
[6] Chakraborty A., Chakraborty U. K., Chatterjee A., Konar A.,“Emotion Recognition From Facial
Expressions and Its Control Using Fuzzy Logic,” IEEE Trans-actions on Systems, Man, and Cyber-
netics - Part A: Systems and Humans, Vol.39, Issue. 4, pp. 726–743, 2009.
[7] Pao T. L., Tsai Y. W., Yeh J. H.,“Recognition and analysis of emotion transition in Mandarin speech
signal,” IEEE International Conference on Systems Man and Cybernetics, pp. 3326–3332, 2010.
[8] Dan-Glauser E. S., Scherer K. R.,“The Geneva affective picture database (GAPED): a new 730-
picture database focusing on valence and normative significance,” Behavior Research Methods, Vol.
43, pp. 468-477, 2011.
[9] Bradley, M.M., Cuthbert, B.N., Lang, P.J.,“International Affective Picture System (IAPS): Technical
Manual and Affective Ratings. ,” NIMH Center for the Study of Emotion and Attention, 1997 ,
http : //csea.phhp.uf l.edu/media.html.
[10] American Psychological Association, http : //www.apa.org/index.aspx
[11] Consequences of Poor Mental Health, www.campushealthandsaf ety.org/mentalhealth/consequences/
[12] Weka 3: Data Mining Software in Java, http : //www.cs.waikato.ac.nz/ml/weka/
14 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

[13] Ralf Kbele, Mandy Koschke, Steffen Schulz, Gerd Wagner, ShravyaYeragani, Chaitra T. Ra-
machandraiah, Andreas Voss, Vikram K. Yeragani,Karl-Jrgen Br,“The influence of negative mood
on heart rate complexity measures and baroreflex sensitivity in healthy subjects,” Indian Journal of
Psychiatry, pp.4247, 52(1):. 2010.
[14] Gregory D. Webster, Catherine G. Weir. ,“Emotional Responses to Music: Interactive Effects of
Mode, Texture, and Tempo,” Motivation and Emotion,Volume 29, Issue 1, pp 1939, 2005.
[15] Wolfram Boucse . ,“Electrodermal Activity,” University of Wuppertal , 2012.
[16] Lauri Nummenma, Enrico Glereana, Riitta Harib, Jari K. Hietanend,“Bodily maps of emotions,”
Developmental science, pp 1-8, 2016.
[17] Ginevra Castellano, Santiago D. Villalba, Antonio Camurri,“Recognizing Human Emotions from
Body Movement and Gesture Dynamics,” ACII, pp. 71-82, 2007.
[18] Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul MinLee, Abe Kazemzadeh,
Sungbok Lee, Ulrich Neumann, Shrikanth Narayanan,“Analysis of emotion recognition using facial
expressions, speech and multi modal information,” ICMI, pp 205-211, 2004.
[19] G.U. Kharat, S.V. Dudul,“Neural network classifier for human emotion recognition from facial ex-
pressions using discrete cosine transform,” First International Conference on Emerging Trends in
Engineering and Technology , pp.653-658, 2008.
[20] R.Cowie, E.D. Cowie, J.G. Taylor, S.Ioannou, M.Wallace, S. Kollias,“An Intelligent system for facial
recognition,” IEEE International Conference on Multimedia and Expo, 2005.
[21] A. Cammuri, I. Lagerlof, G. Volpe,“Recognizing emotion from dance movement: comparison of
spectator recognition and automated techniques,” International journal of human-computer studies,
vol. 59, pp. 213-255,2003.
[22] J.H.Jeon, R.Xia, Y.Liu .,“Sentence level emotion recognition based on decisions from subsentence
segments,” IEEE, ICASSP, pp.4940-4943, 2011.
[23] A.A. Razak, M.I.Zainal Abidin, R. Komiya,“Comparison between fuzzy and NN method for speech
emotion recognition,” International conference on Information Technology and Applications, 2015.
[24] S.G.Koolagudi, R.Reddy, K.S. Rao,“Emotion recognition from speech signal using epoch parame-
ters,” SPCOM, 2010.
[25] T.L.Pao, J.H.Yeh, Y.W.Tsai,“Recognition and analisis of emotion transition in Mandarin speech
signal,” IEEE, SMC, pp.3326-3332, 2010.
[26] Andreas Haag, Silke Goronzy, Peter Schaich, Jason Williams,“Emotion Recognition Using Biosen-
sors: First Steps towards an Automatic System,” Affective Dialogue Systems, Tutorial and Research
Workshop, ADS, Kloster Isree, 2004.
[27] Eun-Hye Jang, Byoung-Jun Park, Sang-Hyeob Kim, Myoung-Ae Chung,Mi-Sook Park, Jin-Hun
Sohn,“ Classification of Human Emotions from Physiological signals using Machine Learning Algo-
rithms,” IARIA, pp.395-400, 2013.
[28] Eun-Hye Jang, Byoung-Jun Park, Sang-Hyeob Kim,“Emotion Classification based on Physiological
Signals inducted by Negative emotions,” IEEE, ICNSC, 2012.
[29] Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, and Yu Chee Tseng,“Analysis of Users’ Emotions Through
Physiology,” The 11th International Conference on Genetic and Evolutionary Computing, 2017.
[30] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash,“Internet of Things: A
Survey on Enabling Technologies, Protocols, and Applications,” IEEE Communications Surveys and
Tutorials, Vol. 17, No.4, pp.2347-2376, 2015.
[31] Oscar D. Lara and Miguel A. Labrador,“A Survey an Human Activity Recognition Using Wearable
Sensors,” IEEE Communications Surveys and Tutorials, Vol. 15, No. 3, 2013.
[32] Smart Home, https : //en.wikipedia.org/wiki/Homea utomation
[33] Smart Transportation, https : //en.wikipedia.org/wiki/V ehiculara utomation
[34] Wearables / e-Health, https : //en.wikipedia.org/wiki/EHealth
 Analyzing User Emotions via Physiology Signals 15

Bohdan Myroniv received his BSc. degree in Information Security

from Yuriy Fedkovych Chernivtsi National University, Ukraine in 2015
and M.S. degree in National Chiao Tung University, Taiwan, in 2017.
His research interests include Machine Learning, Networking, and Data
Mining.

Cheng-Wei Wu received the Ph.D. degree in Department of Com-

puter Science and Information Engineering from National Cheng Kung
University, Taiwan, in 2015. Currently, he is hired as a assistant re-
searcher in College of Computer Science, National Chiao Tung Univer-
sity, Taiwan. His research interests include data mining, utility mining,
pattern discovery, machine learning and big data analytics.

Yi Ren received the Ph.D. degree in information communication and

technology from the University of Agder, Norway, in 2012. He was
with the Department of Computer Science, National Chiao Tung Uni-
versity (NCTU), Hsinchu, Taiwan, as a Postdoctoral Fellow, an Assis-
tant Research Fellow, and an Adjunct Assistant Professor from 2012 to
2017. He is currently a Lecturer in the School of Computing Science at
University of East Anglia (UEA), Norwich, U.K. His current research
interests include security and performance analysis in wireless sensor
networks, ad hoc, and mesh networks, LTE, and e-health security. He
received the Best Paper Award in IEEE MDM 2012.

Albert Budi Christian received his B.C.S degree from Soegi-

japranata Catholic University, Indonesia, in April 2014 and M.S. degree
from Chang Gung University, Taiwan, in June 2017. Currently, he is
pursuing the Ph.D. degree in the Department of Electrical Engineering
and Computer Science, National Chiao Tung University, Taiwan. His
research interests include Data Mining, Internet of Things(IoT), and
Human-Computer Interaction (HCI).
 16 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

Ensa Bajo received his BSc. degree in Electrical Engineering from the
Republic of China Military Academy, Taiwan. Currently, he is pursuing
a M.Sc. in the Department of Computer Science at National Chiao
Tung University. His research interests include smart-phone sensing
and Data mining.

Yu-Chee Tseng received the PhD degree in Computer and infor-

mation science from the Ohio State University in January of 1994.
He was/is chairman (2005-2009) and dean (2011-present), College of
Computer Science, National Chiao-Tung University, Taiwan. He has
been awarded as NCTU chair professor (2011-present) and Y.Z. Hsu
Scientific chair professor (2012-2013). He received the Outstanding Re-
search Award (National Science Council, 2001, 2003, and 2009), Best
Paper Award (International Conference on Parallel Processing, 2003),
Elite I. T. Award (2004), Distinguished Alumnus Award (Ohio State
University, 2005), and Y. Z. Hsu Scientific Paper Award (2009). His
research interests include mobile computing, wireless communication,
and sensor networks. He served/ serves on the editorial boards of IEEE
Transactions on Vehicular Technology, IEEE Transactions on Mobile Computing, IEEE Trans-
actions on Parallel and Distributed Systems, and IEEE Internet of Things Journal. His h-index
is more than 50. He is a fellow of the IEEE.

View publication stats