Tracking the Eye-Mind Relationship of Positive Emotion Using Eye Tracking Technique

TRACKING THE EYE-MIND RELATIONSHIP OF POSITIVE

EMOTION USING EYE TRACKING TECHNIQUE
S. Thiyagarajan1, S.N. Rosli1, M.K.M. Amin1 and S.Wibirama2
1
Malaysia-Japan International Institute of Technology,
Universiti Teknologi Malaysia, 54100,
Kuala Lumpur, Malaysia.

2
Department of Electrical Engineering and Information Technology,
Faculty of Engineering, Universitas Gadjah Mada,
Yogyakarta 55281, Indonesia.

Corresponding Author’s Email: 1sumita2@live.utm.my

Article History: Received 27 December 2018; Revised 30 April 2019;

Accepted 5 October 2019

ABSTRACT: This article presents an exploration and investigation of the eye

movement features to track the eye-mind relationship through eye tracking
device during video clips of positive emotional stimulation. The relationship
of eyes gazing behavior with human mind is a challenging topic. Many
parameters of the eyes gazing behavior is yet to be explored namely pupil
size and pupil dilation. Since pupil size is one of the indicators of the brain’s
activity studies on eye-mind relationship can be beneficial and added value
in learning human behavior. Prior studies have suggested that the cognitive
processing and affective information affect the size of pupils in humans.
Significantly, this study focus on the behavior of positive emotion by using
eye tracking technique to observe the eye-mind relation. It is hypothesized
from this observation that the visual attention of the gazing behavior will
affect the pupil dilation and will further provide evidence of the human
mind triggered emotion. Ten subjects’ pupil responses were measured while
watching interesting and amusing emotional clips. The results showed that
the fixation duration and pupil dilation significantly different between each
video stimulation. These results suggest that the measurement of eye fixation
is a potential computer input for detecting emotional state.

KEYWORDS: Eye Tracking; Fixation; Pupil Dilation

Journal ofAdvancedManufacturingTechnology (JAMT)

1.0 INTRODUCTION
Emergent of eye tracking devices has attracted both academics and
industries for its application in various field of studies and businesses.
The main interest is to involve the inner self factor of human being to
benefit research findings, commerce and productions. Studies on
individuals with physical and psychological disorders suggest that
eye-tracking techniques have the potential to offer insight into the
downstream difficulties in everyday
eISSN: 2289-8107 social
Special Issue interaction
iDECON 2018 which such
93
individual’s experience [1-2]. Similar objectives of eye tracking
 Emergent of eye tracking devices has attracted both academics and
industries for itsManufacturing
Journal of Advanced applicationTechnology
in various field of studies and businesses.
(JAMT)
The main interest is to involve the inner self factor of human being to
benefit research findings, commerce and productions. Studies on
individuals with physical and psychological disorders suggest that
eye-tracking techniques have the potential to offer insight into the
downstream difficulties in everyday social interaction which such
individual’s experience [1-2]. Similar objectives of eye tracking
application is actively used today in the new field called
Neuromarketing [3-4]. Such studies showed that eye analysis can be
useful not only for diagnostic and therapeutic purpose but can be
generally utilized to understand the human behavior through the
relationship between the pupil responses and its attitudes and
behaviors [5-7]. Benefit research findings, commerce and productions.
Studies on individuals with physical and psychological disorders
suggest that eye-tracking techniques have the potential to offer insight
into the downstream difficulties in everyday social interaction which
such individual’s experience [1-2]. Similar objectives of eye tracking
application is actively used today in the new field called
Neuromarketing [3-4]. Such studies showed that eye analysis can be
useful not only for diagnostic and therapeutic purpose but can be
generally utilized to understand the human behavior through the
relationship between the pupil responses and its attitudes and
behaviors [5-7].

The psychological factor of the emotional part closely correlates to the

to Human-computer interaction [8-9] as the pupil dilation of the eye
complement the viewing of emotionally toned or exciting visual stimuli
[10]. Hence, the emotional stimuli is advantageous in attracting the
response of the Human-computer interaction system [11-12].
Moreover, the visual fixation recording in real-time is beneficial to
examine the positive influences of subjects gazing attention. This study
explored and investigated these challenges to relate the eyes gazing
behavior with human mind through the eye tracking technique. The
eye-mind [13] relation is very complex but very few has reported such
as gazing behavior relation to personality [14]. Many parameters of the
eyes gazing behavior is yet to be explored such as pupil size and pupil
dilation. Since pupil size is one of the indicators of the brain’s activity
[15-17], studies on eye-mind relationship can be beneficial and added
value in learning human
Journal ofAdvancedManufacturingTechnology (JAMT)behavior.

Prior studies have suggested that the cognitive processing and affective
information affect the size of pupils in humans. For example, research
finding had proved that pupil dilates when cognitive load was
increasing [18]. Significantly, this study focus on the behavior of
positive emotion by using eye tracking technique to observe the eye-
mind relation. Observation will be made to the pupil dilation and its
fixation to recognize the emotional state. It is hypothesized from this
observation that the visual attention of the gazing behavior will affect
the pupil dilation and will further provide evidence of the human mind
triggered
94 emotion.
eISSN: 2289-8107 Special Issue iDECON 2018
 increasing [18]. Significantly, this study focus on the behavior of
positive emotion byEye-Mind
Tracking the using Relationship
eye tracking technique
of Positive to observe
Emotion Using Eye Trackingthe eye-
Technique
mind relation. Observation will be made to the pupil dilation and its
fixation to recognize the emotional state. It is hypothesized from this
observation that the visual attention of the gazing behavior will affect
the pupil dilation and will further provide evidence of the human mind
triggered emotion.

2.0 METHODOLOGY

2.1 Experimental Flow Chart

Figure 1 depicts the experiment flow chat.

Prepare subject
Prepare subjectand
andstimuli.
stimuli.

Calibration process.
Calibration process.

Conduct theexperiment.
Conduct the experiment.

Define variables.
Define variables.

Analysis
Analysis ofofdata.
data.

Figure 1: Flow chart of the experiment

Journal ofAdvancedManufacturingTechnology (JAMT)

2.2 Subjects
10 students with the age ranging from 20 to 30 years old were randomly
recruited to take part in this study. The experiment was conducted in a
closed air-conditioned room at the Bio-Cognition Laboratory,
Malaysia-Japan International Institute of Technology.

2.3 Stimuli
Videos on interesting and amusement as shown in Figures 2 and 3 were
chosen for the stimuli in the experiment. Participants were seated
comfortably on a chair and were requested to view these videos.

eISSN: 2289-8107 Special Issue iDECON 2018 95

 Videos on interesting and amusement as shown in Figures 2 and 3 were
chosen for the stimuli in the experiment. Participants were seated
chosen for the stimuli in the experiment. Participants were seated
comfortably on Manufacturing
Journal of Advanced a chair and were requested
Technology (JAMT) to view these videos.
comfortably on a chair and were requested to view these videos.

Figure 2: Video clip on interesting

Figure 2: Video clip on interesting

Figure 3: Video clip on amusement

Figure 3: Video clip on amusement

2.4 Experiment Procedure

2.4 Experiment Procedure
The experiment protocol was briefed and informed consent were
The experiment protocol was briefed and informed consent were
obtained from the participants before starting the experiment.
obtained from the (JAMT)
Journal ofAdvancedManufacturingTechnology participants before starting the experiment.
Participants were instructed
Journal ofAdvancedManufacturingTechnology (JAMT) to sit comfortably to watch the video clips.
Participants were instructed to sit comfortably to watch the video clips.
The
record Tobii TX300
the subjects eye tracking
eyestracking device was
gazing attention. Theused as an equipment
eye tracker to
uses infrared
The Tobii TX300 eye device was used as an equipment to
record
cornealthe subjectstoeyes
reflection measuregazing attention.
point of gazeThe
witheye tracker
data uses
rates of 60 infrared
Hz, and
corneal reflection
accuracy to measure
of 0.5 degrees. point were
Subjects of gaze withunderwent
then data rates of 60 Hz, and
a calibration
accuracybefore
process of 0.5 thedegrees.
start Subjects were then underwent
of each experiment. The devicea calibration
requires a
process before the start of each
calibration of the eye movements before starting experiment. The
thedevice requires
recording. At thea
calibrationof
beginning ofeach
the eye
video movements before starting
the experimenter the recording.
left the room At the
to allow subject
beginning
feel of each
more freely video
while the experimenter
watching left the room to allow subject
the videos.
feel more freely while watching the videos.

2.5 Data Analysis

2.5 Data Analysis
Pupil size analyzed by creating segment using Tobii Studio software
Pupil size
which eachanalyzed
segment by creatingone
represents segment
video.using
TobiiTobii
StudioStudio software
generates raw
which each segment represents one video. Tobii Studio generates
data for each subject. Pupil dilation and fixation duration were raw
data for each subject. Pupil dilation
extracted separately from the raw data. and fixation duration were
extracted separately from the raw data.

96
3.0 eISSN:
RESULTS AND 2289-8107 Special Issue iDECON 2018
DISCUSSION
3.0 RESULTS AND DISCUSSION
 data for each subject. Pupil dilation and fixation duration were
data for each subject. Pupil dilation and fixation duration were
extracted separately from the raw data.
extracted separately from the raw data.
Tracking the Eye-Mind Relationship of Positive Emotion Using Eye Tracking Technique
3.0 RESULTS AND DISCUSSION
The data obtained from the experiment were very huge. Each subject
3.03.0RESULTS
RESULTSAND
ANDDISCUSSION
DISCUSSION
produced thousands of pupil dilation from each video. The data were
The data
therefore
The obtained
data obtainedfrom
averaged and the
from took
the experiment
2 days to
experiment were
were veryhuge.
complete
very huge. Each
processing
Each subject
using
subject
produced
produced
Microsoft thousands
thousands
Excel. ofofpupil
As shown in dilation
pupil dilation
Figure 4, from
from eachvideo.
each
responsesvideo. The
The
of pupil data
data towere
were
size the
therefore averaged
therefore averaged
amusement and
arousal and took 2 days
took 2 roughly
provoked to complete
days to higher
complete processing
thanprocessing using
using
the interesting
Microsoft
MicrosoftThe
arousal. Excel.
Excel. As shown
As shown
subjects’ in Figure
in Figuremean
pupil dilation 4, responses
4, responses of pupil size
of pupil
were obtained to the
size tofrom
ranging the
3.2amusement
amusement arousalthe
and 4.93 arousal
during provoked
provoked
exposure roughly
to thehigher
roughly higher than
thanthe
amusement theinteresting
and interesting
arousal. The subjects’ pupil dilation mean were obtained ranging from
arousal.
video. The subjects’ pupil dilation mean were obtained ranging from
3.2 and 4.93 during the exposure to the amusement and interesting
3.2 and 4.93 during the exposure to the amusement and interesting
video.
video.

Journal ofAdvancedManufacturingTechnology (JAMT)

JournalSimilar process was carriedfor the fixation duration. Figure 5 depicts

ofAdvancedManufacturingTechnology
Figure (JAMT)
Journal ofAdvancedManufacturingTechnology (JAMT) 4:
Figure
Average of pupil dilation
4: Average of pupil dilation
Fixation duration of amusement video was ranged from 0.3 to 0.6.
Similar process
Interesting video was carried
ranged for the fixation
between 0.2 andduration. Figure 5 depicts
0.65. However,
Similar process was carried
Figure for the fixation
4: Average of pupil duration.
dilation Figure this data
5 depicts
Fixation
experiment duration of
did notofshowamusement video was
significant in was ranged
average from 0.3
of fixation to 0.6.
duration
Fixation duration
Interesting amusement
video ranged betweenvideo
0.2 and 0.65.ranged fromthis
However, 0.3 to 0.6.
data
differences between amusement and interesting videos
Interesting video ranged between 0.2 and 0.65. However, this data stimulation.
experiment did not show significant in average of fixation duration
experiment
differencesdid not show
between significant
amusement in average
and interesting of fixation
videos duration
stimulation.
differences between amusement and interesting videos stimulation.

Figure
Figure5:5:Average of fixation
Average of fixationduration
duration

AsAsthethe
pupil
pupildilation
dilationisiscontrolled by the
the autonomic nervous system
eISSN: 5:controlled
2289-8107
Figure Average ofbyfixation
Special Issueautonomic
iDECON nervous
duration2018 system 97
[12], the results confirm that
[12], the results confirm that the autonomic nervous system
autonomic nervous system reacts reacts
 Journal of Advanced Manufacturing Technology (JAMT)
Figure 5: Average of fixation duration

As the pupil dilation is controlled by the autonomic nervous system

[12], the results confirm that the autonomic nervous system reacts
differently to emotionally arousing stimuli. The results also preserve of
those past studies of different emotional association responses to
differential autonomic nervous system activity. There is a possibility
that pupil size is insensitive towards the visual stimuli. Apart from that,
there are some of previous studies of discrete emotions might have
failed to control for the differences in the amount of arousal generated
by the respective emotions. Therefore, differences in autonomic
responses attributed to particular emotions may have resulted from
differences in the amount of arousal generated by the emotions.
Journal ofAdvancedManufacturingTechnology (JAMT)

4.0 CONCLUSION
The results of this study has confirmed empirically on the gazing
behavior when presented with positive emotion stimuli. The
experiment had observed that eye activities in amusement and interest
stimulus can be a feasible mean to classify different affective states.
Hence, emotional detection has a great role to play in future technology
and the usage of eye tracking are expected to be exponentially
increased, give the benefit to the fields like psychological studies and
marketing. This study provides an additional measurement and
benchmark on human gazing behavior during visual emotional
stimulation. The average of pupil dilation and average of fixation
duration were investigated. With the present finding, fixation seems to
hold discriminative emotional effect due to different stimulus. Taken
together, these results suggest that further studies to detect different
emotional states through eye tracking will confirm the eye-mind
relationship.

ACKNOWLEDGMENT
The authors would like to thank members of the Biocognition
Laboratory of Bio-inspired System Technology research group,
Malaysia-Japan International Institute of Technology (MJIIT). This
work was supported under the UTM grant, PY/2017/01721 with the cost
center Q.K130000.2643.15J37.

REFERENCES
98 eISSN: 2289-8107 Special Issue iDECON 2018
[1] C. Izard, S. Fine, D.Schultz, A. Mostow, B. Ackerman and E.Youngstrom,
 work was supported under the UTM grant, PY/2017/01721 with the cost
center Q.K130000.2643.15J37.
Tracking the Eye-Mind Relationship of Positive Emotion Using Eye Tracking Technique

REFERENCES

[1] C. Izard, S. Fine, D.Schultz, A. Mostow, B. Ackerman and E.Youngstrom,

“Emotion knowledge as a predictor of social behavior and academic
competence in children at risk”, Psychological Science, vol. 12, no. 1,
pp. 18–23, 2001.

[2] L. Yan, X. Wen, L. Zhang, and Y. Son, “The application of unascertained

measure to the video emotion type recognition,” in 2nd International
Conference on Signal Processing Systems, Dalian, China, 2010, pp. V2-
Journal ofAdvancedManufacturingTechnology (JAMT)
447-V2-451.
[3] P. Rosa, “What do your eyes say? Bridging eye movements to consumer
behavior”, International Journal of Psychological Research, vol. 8, no. 2, pp.
90-103, 2015.

[4] H. Zamani, A. Abas and M. K. M.Amin, “Eye Tracking Application on

Emotion Analysis for Marketing Strategy”, Journal of Telecommunication,
Electronic and Computer Engineering, vol. 8, no. 11, pp. 87-91, 2016.

[5] G. L. Lohse and E. J. Johnson, “A comparison of two process tracing

methods for choice tasks,” Organizational Behavior and Human Decision
Processes, vol. 68, no. 1, pp. 28-43, 1996.

[6] C. Zong and M. Chetouani, “Hilbert-Huang transform based

physiological signals analysis for emotion recognition,” in International
Symposium on Signal Processing and Information Technology, Ajman,
United Arab Emirates, 2009, pp. 334–339.

[7] S. Alghowinem, M. AlShehri, R. Goecke, and M. Wagner, “Exploring eye

activity as an indication of emotional states using an eye-tracking
sensor,” in Intelligent systems for science and information, Studies in
Computational Intelligence, vol. 542, S. Alghowinem, M. AlShehri, R.
Goecke and M. Wagner, Switzerland AG: Springer International
Publishing, 2014, pp. 261-276.

[8] T. Partala, M. Jokiniemi, and V. Surakka, “Pupillary responses to

emotionally provocative stimuli,” in the Proceedings of the Symposium
on Eye Tracking Research and Applications, Florida, USA, 2000, pp. 123–
129.

[9] C. Sharma and S. K. Dubey, “Analysis of eye tracking techniques in

usability and HCI perspective,” in International Conference on
Computing for Sustainable Global Development, New Delhi, India, 2014,
pp. 607-612

[10] P. Binda, M. Pereverzeva, and S. O. Murray,” Pupil size reflects the focus
of feature-based attention”, Journal of Neurophysiology, vol. 112, no. 12, pp.
3046-3052, 2014.
eISSN: 2289-8107 Special Issue iDECON 2018 99
[11] J. Scheirer, R. Fernandez, J. Klein, and R. W. Picard, “Frustrating the user
 usability and HCI perspective,” in International Conference on
Computing
Journal of Advanced for Sustainable
Manufacturing Global Development,
Technology (JAMT) New Delhi, India, 2014,
pp. 607-612

[10] P. Binda, M. Pereverzeva, and S. O. Murray,” Pupil size reflects the focus
of feature-based attention”, Journal of Neurophysiology, vol. 112, no. 12, pp.
3046-3052, 2014.

[11] J. Scheirer, R. Fernandez, J. Klein, and R. W. Picard, “Frustrating the user

on purpose: a step toward building an affective computer,” Interacting
with computers, vol. 14, no. 2, pp. 93-118, 2002.

[12] T. Partala and V. Surakka, “Pupil size variation as an indication of

affective processing,” International Journal of Human-Computer Studies, vol.
Journal ofAdvancedManufacturingTechnology (JAMT)
59, no. 1, pp. 185-198, 2003.

[13] S. Mathôt and S. Van der Stigchel, “New Light on the Mind’s Eye: The
Pupillary Light Response as Active Vision”, Current Directions in
Psychological Science, vol. 24, no. 5, pp. 374-378, 2015.

[14] J.F. Rauthmann, C.T. Seubert, P. Sachse and M.R. Furtner,”Eyes as

windows to the soul: Gazing behavior is related to personality”, Journal
of Research in Personality, vol. 46, no. 2, pp. 147-156, 2012.

[15] M. M. Bradley, L. Miccoli, M. A. Escrig, and P. J. Lang, “The pupil as a

measure of emotional arousal and autonomic activation,”
Psychophysiology, vol. 45, no. 4, pp. 602-607, 2008.

[16] S. Amemiya and K. Ohtomo,” Effect of the observed pupil size on the
amygdala of the beholders”, Social Cognitive and Affective Neuroscience,
vol. 7, no. 3, pp. 332-341, 2012.

[17] S. M. Berends, S. M. Brouwer and S. A. Sprenger, “Eye-Tracking and the

Visual World Paradigm”, in Designing Research on Bilingual Development,
Behavioral and Neurolinguistic Experiments, M.S. Schmid, S.M. Berends, C.
Bergmann, S.M. Brouwer, N. Meulman, B. Seton, S. Sprenger and L.
Stowe, Switzerland AG: Springer International Publishing, 2016, pp. 55-
80.

100 eISSN: 2289-8107 Special Issue iDECON 2018