2020 13th International Symposium on Computational Intelligence and Design (ISCID)
Exploiting the emotional preference of music for music recommendation
in daily activities
2020 13th International Symposium on Computational Intelligence and Design (ISCID) | 978-1-7281-8446-3/20/$31.00 ©2020 IEEE | DOI: 10.1109/ISCID51228.2020.00085
Hui Zhang1 , Kejun Zhang1, ∗ , Nick Bryan-Kinns2
1 Laboratory
of CAD, CG, Zhejiang University, China
2 School of Electronic Engineering and Computer Science, Queen Mary University of London, the United Kingdom
{huiz, zhangkejun}@zju.edu.cn, n.bryan-kinns@qmul.ac.uk
Abstract—The emotional use of music plays an impor-
tant role in influencing people’s music-searching behaviour.
Many users listen to music for multiple mood regulation
purposes. Therefore, emotional preference should be taken into
consideration especially when building context-aware music
recommendation systems which aim to satisfy users’ short-term
music needs. In addition, cultural differences will affect the way
people experience and value emotion from music. Based on
these aspects, we investigate the emotional preference of music
in daily activities through a cross-cultural survey in China and
the UK, and build emotional maps between activity and music
for the two countries. Furthermore, we introduce EmoMusic,
a novel emotion-based music recommendation service for daily
activities which supports visualisation and control of music
emotion through an interactive interface. A user study is pre-
sented to evaluate the application. Results reveal good usability
of the service and show that visualization and interaction
in a music emotion space can improve acceptance of music
recommendations. Our work helps to understand the emotional
preference of music for various activities and how to exploit
emotional cues of music in a music recommendation service,
paving the way for more user-centered music recommendation
systems.
Keywords-user-centered music recommendation; emotional
preference; cross-cultural
I. I NTRODUCTION
As one of the key factors that affect music listening
behaviors of human, emotion has played an important role
in studies on context-aware music recommendation systems.
Park et al. [1] use a fuzzy Bayesian network to deduce a
user’s mood from context information including weather,
noise, time, gender as well as age, and then recommend
music that matches the deduced user mood using manually
annotated emotion labels of each available song. This is
probably the first research work which proposes the concept
of context-aware music recommendation. Chen-Ya’s work
[2] on context and emotion aware system for personalized
music recommendation also measures the similarity between
a user’s emotion and the emotional character of a song
as the affective indicator. In addition, with the develop-
ment of wearable physiological sensors, some researchers
have tried integrating emotion recognition technique based
on physiological signals into the music recommendation
∗ Kejun Zhang is the corresponding author.
2473-3547/20/$31.00 ©2020 IEEE
DOI 10.1109/ISCID51228.2020.00085
Authorized licensed use limited to: University of Prince Edward Island. Downloaded on May 26,2021 at 11:43:16 UTC from IEEE Xplore. Restrictions apply.
systems to obtain accurate user emotional information [3].
Through reviewing the work of these predecessors, it can
be seen that currently the common processes of research on
emotion-aware music recommendation is to first reason user
emotions and then recommend music with similar emotional
states based on the obtained emotional cues. However, in
the case of self-chosen music listening, people’s emotional
needs for music are not necessarily in line with their real-
time emotional state. For example, when someone gets
up in the morning and still feel sleepy, he may prefer
to have some energetic songs playing instead of drowsy
songs. Mood management and regulation is one of the most
common motivations and functions of self-chosen music
listening behaviour [4], and the emotional uses of music
vary across different daily activities [5]. Considering this
aspect, the authors suggest that the emotion-aware music
recommendation systems should not only consider the real-
time emotional state of users, but should recommend songs
that can prompt users to reach their target emotional state.
It leads to the first research question in this paper:
RQ1. Will the user-expected musical emotion match their
emotional state when users start listening to the song
(initial emotion state), or do users prefer music which
can facilitate achieving another emotional state (target
emotion state)?
Furthermore, cultural differences may exist in emotional
uses of music. Previous studies have shown that emotion is
not only biologically determined, but is also affected by the
social environment. The way people experience and value
the emotion varies significantly across different cultural
backgrounds [6]–[8]. Therefore, it’s necessary to explore
how cultural and national context may affect people’s feel-
ings about the emotional uses and preference of music,
which leads to the following research question:
RQ2. Will British people and Chinese people have different
preferences for musical emotions in each daily activity?
This paper sets out to exploit the emotional use of music
in various daily activities in order to optimize the experience
of user-centered, context-aware music recommendation. The
authors have conducted a cross-cultural study to explore peo-
ple’s music preference in daily activities and implemented
350
a web application built on the results. More specifically, the
major contributions of this paper are:
• Emotional preference of music in daily activities: We
explore emotional uses of music in daily activities both
in China and in the United Kingdom, which allows us
to build the mapping between music emotion and daily
activities as well as the cross-cultural comparison.
• Implementation and evaluation of EmoMusic: We im-
plement EmoMusic, a novel emotion-based music rec- Figure 1. The distribution of the scores of participants in the Gold-MSI
subscale of emotion factor (a) and the number of times each activity was
ommendation service for daily activities, which sup- chosen (b).
ports explanation and control of music emotion through
an interactive interface. In addition, we present a user Table I
study and discuss the results to demonstrate its perfor- R ESULTS OF TOST TESTS ON EXPECTED MUSIC EMOTION AND
TARGETING USER EMOTION
mance and usability.
China UK
II. O NLINE S URVEY
Valence Arousal Valence Arousal
In order to better understand emotional preferences of df 211 211 195 195
music listening in everyday situations and the influence of t(ΔL =−0.3) 3.526 3.403 7.353 5.293
cultural aspects, an online survey in Chinese and English p(ΔL =−0.3) < 0.001 < 0.001 < 0.001 < 0.001
t(ΔU =0.3) −6.309 −7.811 −5.401 −5.116
was designed and released from March to May 2020, in p(ΔU =0.3) < 0.001 < 0.001 < 0.001 < 0.001
China and the UK.
A. Participants
happy, 1 = calm and 5 = excited). There was an additional
A total of 89 participants took part in this survey, includ-
free response field for comments and advice. The average
ing 47 from China (29 females and 18 males, aging from
time to complete the survey for each participant is 17.70
18 to 37) and 42 from the UK (15 females and 27 males,
minute.
aging from 18 to 56). The minimum sample size (38 for two-
tailed paired t-test, dz = 0.5, α = 0.05, and power = 0.85) C. Results and Discussion
of participants in each country was calculated using prospec- Scores of participants for the Gold-MSI subscale of emo-
tive power analysis before the experiment to guarantee the tion factor were calculated and can be seen in Figure 1a.
statistical defensibility of further analysis. Participants in both of China and the UK achieved compar-
B. Procedure atively high average scores (0.779 and 0.817 respectively),
which means the major proportion of participants are used
One page of research brief and a consent form were first
to consciously listening to music to manage their mood
presented online to participants before they decided to attend
in everyday life. Figure 1b shows the distribution of the
the survey. After that, participants were asked to fill in some
number of times each activity was selected. It can be seen
basic demographic information (gender, age and nationality)
that the most of activities share similar popularity except
and six items from the Gold-MSI subscale of emotion
that partying, waiting and social interaction differ greatly
factor (importance of music for emotional functions) [9].
in ranking (> 3) between two countries. Though there is
Participants were next required to select five daily activities
some difference in the number of times choosing activities
when they will listen to music from 13 situations shown in
between the two country, there is no significant difference
Table II. For each activity, participants were asked to answer
in the importance and the frequency of listening to music
questions about the frequency (Q1) and importance (Q2) of
in these activities according to the answers of Q1 and Q2.
listening to music, as well as their emotional preference
It is interesting to find that participants in China and the
of music (Q3) and affective states before (Q4) /during
UK share similar levels of engaging in incidental musical
(Q5) this activity. In this paper, emotion is represented by
listening in everyday activities.
using the valence-arousal model (VA model) proposed by
One-way ANOVA analysis was performed on the user-
Russell [10], which is one of the most popular dimensional
expected musical emotion, initial emotion state and target
emotion conceptualizations and has been adopted by a large
emotion state in each daily activity. In both of countries,
proportion of Music Emotion Recognition (MER) research.
results revealed significant difference across valence of three
Q1 and Q2 were rated on a 5-point Likert scale (respectively,
kinds (fv = 65.663, pv < 0.01 in China, and fv =
1 = never and 5 = always, 1 = not at all and 5 = absolutely
55.873, pv < 0.01 in the UK) as well as arousal (fa =
essential). Valence and arousal in Q3, Q5 and Q6 were rated
31.796, pa < 0.01 in China, and fa = 9.265, pa < 0.01 in
on a 5-point Likert scale (respectively, 1 = sad and 5 =
the UK). In order to further analyze the pairwise difference

351

Authorized licensed use limited to: University of Prince Edward Island. Downloaded on May 26,2021 at 11:43:16 UTC from IEEE Xplore. Restrictions apply.
 Table II
AVERAGED EMOTIONAL VALUES OF MUSIC FOR EACH ACTIVITY

Activity music valence music arousal emotional use

China UK China UK China UK
work/study 3.625 3.906 3.250* 2.531* +V+A +V
falling asleep 3.375 3.667 2.125 2.333 -A +V-A
waking up 4.083 4.000 4.167* 2.400* +V+A
commuting 3.931 4.000 3.759 3.688 +V+A +V+A
social interaction 3.500 3.833 3.500 3.667
housework 4.000 3.765 3.944 3.529 +V+A +V+A
physical activity 4.207 3.968 4.655 4.290 +V+A +V+A
shopping 4.500 3.200 3.500 2.400
having a meal 3.400 3.000 3.000 1.500 Figure 2. Screenshot of the EmoMusic interface. (a: left) pane for selecting
partying 4.571 4.474 4.286 4.211 +V daily activities, (b: centre) visualization of music emotion space, (c: right)
gaming 3.833 3.800 4.083 3.600 +V+A +V+A recommendation song list.
waiting 3.857 3.750 3.286 3.750 +V
relaxation 3.923 3.870 3.231* 2.565* +V +V

* means significant difference on this pair of emotional items of
the two countries.
Note. Emotional use is presented only when there is significant
difference between initial emotion and target emotion for this
activity, e.g, +V means there is a significant increase from initial
valence to target valence.
among the three sets of emotion value, we conducted t-
tests on them in pairs and found that mood change expected
by participants was significant (tv = 12.388, pv < 0.01,
ta = 8.352, pa =< 0.01 in China and tv = 10.234, pv <
0.01, ta = 5.190, pa < 0.01 in the UK), while there
was no significant difference between music emotion and
target emotion. We further conducted an equivalence test
on music emotion and target emotion using two-one-sided
t-tests (TOST). Results suggested that the expected music
emotion and targeting user emotion (both of valence and
arousal) were statistically equivalent (shown in Table I),
which means participants would like to listen to music of
emotion that matches the target emotion instead of their
initial emotional state.
The detailed values of expected music emotion for each
activity are described in Table II, which can be utilized
to build emotional map between music emotion and daily
activities in China and the UK, as well as implement
emotion-based music recommendation system for everyday
life. In addition, a set of repeated t-tests was performed on
initial emotion and target emotion to explore the emotional
use of music listening in each activity. As shown in Table II,
participants in China and the UK share similar expectation of
mood management direction for the majority of daily activi-
ties though some difference exists, e.g., Chinese participants
would like use music to increase both of valence and arousal
when doing ‘work/study’ while UK participants only expect
valence improvement.
III. E MO M USIC S ERVICE
To demonstrate how to utilize the emotional map between
activity and music acquired from the previous survey to build
emotion-based music recommendation service, a prototype
web application, EmoMusic, was developed and deployed
online. The interaction of EmoMusic is designed based on
simplicity and usability and its user interface consists of
a navigation bar and a main operation area which can be
divided into three sections: activity selection (Figure 2a),
emotion visualization (Figure 2b) and recommendation panel
(Figure 2c). Due to the personalization of emotional needs
and the dynamic nature of the listening context, adaptivity of
music recommendation service is very important [11]. Emo-
Music implements the change of user emotional preference
by enabling the movement of the target emotion star within
the visualization panel, and the recommended songs will
be recalculated according to mood-based similarity. Music
collection used in EmoMusic are obtained from the PMEmo
Dataset [12], which provides emotional annotations in both
of valence and arousal dimension for 794 music excerpts.
A. User Study
In order to evaluate the usability and performance of
EmoMusic in recommending music for daily activities, we
conducted a user study in the UK and invited 31 participants
(16 females and 15 males) from the age of 24 to 56.
Participants were first asked to provide demographic related
information in a pre-study, then performed the main task
and finally gave qualitative feedback in a post study. During
the main task, participant were asked to select a daily
activity they would like to do whilst listening to music.
The emotional values of the selected activity were used
to generate a list of 5 recommended music tracks which
were shown on the right side of the web page. Ratings were
collected for 5 tracks in an initial recommendation list. After
that, participants were allowed to interact freely with the
application to generate recommendation lists as they wished.
Once satisfied, they were required to again rate the full list
of tracks for the final list. In addition to rating individual
tracks, participants were also asked to provide overall ratings
for the initial list and the final list.
Distribution of scores rated by participants in this study
is shown in Figure 3. As shown in Figure 3a, the average
rating of individual tracks was 3.323 for the initial playlist

352

Authorized licensed use limited to: University of Prince Edward Island. Downloaded on May 26,2021 at 11:43:16 UTC from IEEE Xplore. Restrictions apply.

Figure 3. Scores rated by participants. (a) RI is rating of individual
tracks in initial playlist, RF is rating of individual tracks in final playlist,
ORI is overall rating of initial playlist, and ORF is overall rating of final
playlist. (b) VBU and VAU denote scores of valence before and after using
EmoMusic respectively, ABU and AAU denote scores of arousal before
and after using EmoMusic respectively. (c) presents scores of level of
satisfaction and difficulty of using EmoMusic. The purple full lines inform
the median value and blue dotted lines inform the mean value.
and 3.884 for the final playlist. Meanwhile, the mean value
of overall rating was 3.226 for the initial playlist and 3.839
for the final list, which indicated that participants were able
to find better tracks through the interactive panel of music
emotion (t = 5.022, p < 0.01 for individual tracks and
t = 2.906, p < 0.01 for overall playlist). In order to detect
the influence of EmoMusic on mood regulation, participants
were asked to report their mood in the pre-study and the
post study (Figure 3b). Participants were reported to have
significantly higher level of valence (t = 2.244, p < 0.05)
and arousal (t = 4.284, p < 0.01) after their experience
on EmoMusic. It should be mentioned that no participant
chose ‘falling asleep’ during this study. In addition, we can
see in Figure 3c that the median level of satisfaction and
difficulty of using EmoMusic were 4 and 2 respectively,
which revealed good usability of the service.
At the end of the study, participants were asked to give
comments on their experience and leave suggestions for
improving the service. On the positive side, many partic-
ipants said the interface was easy to use (e.g, ‘nice and
clean’, ‘intuitive’) and they enjoyed the service. On the other
hand, participants also mentioned that the music collection
was small and lacked music of other genres they like such
as ‘instrumental’. This problem can be addressed in the
future by extending the music collection with tracks of
more genres and incorporating collaborative or content based
recommendation methods for personalization.
IV. C ONCLUSION AND F UTURE W ORK
In this work, we investigated the emotional preference
of music in daily activities for context-aware music rec-
ommendation through a cross-cultural survey. Based on the
emotional map between activity and music acquired from
the survey, we presented EmoMusic - a novel emotion-based
music recommendation service for daily activities which
supports explanation and control of music emotion through
an interactive interface. A user study was presented to eval-
uate the EmoMusic application and reported good feedback
from participants. In future work we will (i) explore further
353
Authorized licensed use limited to: University of Prince Edward Island. Downloaded on May 26,2021 at 11:43:16 UTC from IEEE Xplore. Restrictions apply.
the relation between personal characteristics and emotional
preference of music. (ii) incorporate emotional preference
with collaborative or content based recommendation engine
for better personalization.
ACKNOWLEDGMENT
This project is supported by the Key Project of Na-
tional Science Foundation of Zhejiang Province (No.
LZ19F020002), the National Natural Science Foundation
of China (No. 61672451), Provincial Key Research and
Development Plan of Zhejiang Province (No. 2019C03137).
And the first author would like to thank the financial support
of China Scholarship Council (CSC).
R EFERENCES
[1] H.-S. Park, J.-O. Yoo, and S.-B. Cho, “A context-aware music
recommendation system using fuzzy bayesian networks with
utility theory,” in International conference on Fuzzy systems
and knowledge discovery. Springer, 2006, pp. 970–979.
[2] C.-Y. Wang, Y.-C. Wang, and S.-C. T. Chou, “A context and
emotion aware system for personalized music recommenda-
tion,” Journal of Internet Technology, vol. 19, no. 3, pp. 765–
779, 2018.
[3] D. Ayata, Y. Yaslan, and M. E. Kamasak, “Emotion based
music recommendation system using wearable physiological
sensors,” IEEE transactions on consumer electronics, vol. 64,
no. 2, pp. 196–203, 2018.
[4] T. Schäfer, P. Sedlmeier, C. Städtler, and D. Huron, “The
psychological functions of music listening,” Frontiers in psy-
chology, vol. 4, p. 511, 2013.
[5] Y. Song, “The role of emotion and context in musical prefer-
ence.” Ph.D. dissertation, Queen Mary University of London,
2016.
[6] M. H. Immordino-Yang, X.-F. Yang, and H. Damasio, “Corre-
lations between social-emotional feelings and anterior insula
activity are independent from visceral states but influenced
by culture,” Frontiers in human neuroscience, vol. 8, p. 728,
2014.
[7] Y. Miyamoto, X. Ma, and A. G. Petermann, “Cultural differ-
ences in hedonic emotion regulation after a negative event.”
Emotion, vol. 14, no. 4, p. 804, 2014.
[8] M. Joshanloo, Z. K. Lepshokova, T. Panyusheva, A. Natalia,
W.-C. Poon, V. W.-l. Yeung, S. Sundaram, M. Achoui,
R. Asano, T. Igarashi et al., “Cross-cultural validation of
fear of happiness scale across 14 national groups,” Journal
of Cross-Cultural Psychology, vol. 45, no. 2, pp. 246–264,
2014.
[9] D. Müllensiefen, B. Gingras, J. Musil, and L. Stewart, “The
musicality of non-musicians: an index for assessing musical
sophistication in the general population.” PloS one, 2014.
[10] J. Posner, J. A. Russell, and B. S. Peterson, “The circum-
plex model of affect: An integrative approach to affective
neuroscience, cognitive development, and psychopathology,”
Development and psychopathology, vol. 17, no. 3, pp. 715–
734, 2005.
[11] S. Stober and A. Nürnberger, “Adaptive music retrieval–a
state of the art,” Multimedia Tools and Applications, vol. 65,
no. 3, pp. 467–494, 2013.
[12] K. Zhang, H. Zhang, S. Li, C. Yang, and L. Sun, “The pmemo
dataset for music emotion recognition,” in Proceedings of
the 2018 acm on international conference on multimedia
retrieval, 2018, pp. 135–142.