IEEE TRANSACTIONS ON HUMAN - MACHINE SYSTEMS 1

Human Computer Musical Interfaces: A survey

Roberto Avila - Va´zquez1, Rogelio Bustamante - Bello 1, Arturo Molina 1, Sergio Navarro - Tuch 1,
Javier Izquierdo Reyes 1 and Rafael Ferrer 2
1Tecnolo´gico de Monterrey, School of Science and Engineering, Ciudad de Me´xico, Me´xico
2 Tecnolo´gico de Monterrey, School of Education and Humanities, Ciudad de Me´xico , Me´xico

Human Computer Musical Interfaces (HCMI) is a growing research area closely related to Human and Brain Computer
Interaction (HCI / BCI). These systems have many potential applications, such as, music composition, interactive sonification,
recommendation systems and music therapy. HCMI emerged with a strong epistemic justification but perhaps because of the
assimilation of new technologies borrowed from the most diverse fields, it has become disperse and fragmented. This paper presents a
theoretical framework that combines an analysis of the state of the art with a review about the technological tools used for interface
development during the last fifteen years. In addition, this work includes a discussion about the future development of the discipline
in terms of methodological standardization of devices and experiences, the use of machine learning techniques, and a renewal of the
musical experience for education and entertainment.

Index Terms—Human Computer Musical Interface, music, Human - Computer Interface, sonification, music composition, music
therapy, recommendation systems

I. I NTRODUCTION input, implies particular changes of data processing,

H UMAN - Computer Music Interfaces (HCMI) is a

research field focused in the interaction between computer
machines and humans. It typically includes a device, a
procedure or a way of interaction, related with music hearing
and performance. HCMI core is comprised by a wide range
of multidisciplinary studies that entail music as an input
for modeled systems, being that of cybernetic (i.e.,
computers) and organic (e.g., human beings) nature the
most frequently investigated. A central paradigm consists of
knowing the forms of interaction between machines and
humans, more specifically, understanding the information
exchange. At an international level professionals from
different disciplines produce musical interfaces. Hence, the
fundamental background and evidence of important
developments in the area are distributed along several
sources sustained by diverse disciplines, such as specialized
journals (e.g., computer science, machine learning, acoustics,
medicine and virtual reality). This work represents an effort
to bring together the
most relevant advances and research topics within the area.
understanding of the relationship between input and
output, and an adequate organization of space and tasks
for an effective audio perception or production. Another
fundamental characteristic is the speed of the system,
which must respond in at least 10 milliseconds to
produce an effective impact of stimuli given to the
person that interacts with the device.
• HCMI also focus on very different forms of physical
interactions between devices and subjects. Even subtle
mechanics applied to musical instruments in
performance, can be captured and transported into further
interfaces. Moreover, the special possibilities of this kind
of interfaces allows for experimentation of new kinds of
interaction, such as, unconventional movements, spatial
intervention, and communication with robots by sound.
• HCMI has special ways of storage (e.g., sheet
music, MIDI –Musical Instrument Digital Interface–
format), which implies additional processing for
adequate recording of the results and modifications
performed over the sound material.

II. T HEORETICAL A PPROACHES Laske [3] presented an early proposal about the interfaces that
incorporate music as stimuli or final product. He recognized
A multidisciplinary research area also requires a wide
two central ideas: the musical task –used in interventions
range of definitions borrowed from diverse areas. Some
that involve the creation of a musical play– and the musical
authors have tried to organize a theoretical corpus for
ambient (i.e., the use of music as an element of the
HCMI, centered on different conceptualization of this kind
soundscape [4]) inside the interface. He also identified two
of platforms. For instance, Holland [1] argues that the
types of feedback: stimulus generation and proof, and
techniques, common practices, and languages derived from
heuristic research (i.e., feedback with a test applied to the
the long musical tradition, make HCMI a branch of
subject or the scenario). Finally, the paper mentions another
knowledge different from HCI. Another example, from
basic components of this kind of interfaces: task analysis,
Wessel [2] , outlines the main differences between HCI and
analysis of the basal state (i.e., initial state) of the subject,
HCMI. These are listed next:
recognition and assessment of discrepancies between basal
• HCMI has a different set of peripheral devices in state and competence state of users, and improvement of
comparison to another kind of interfaces. Joining interface design. Keller [5] classified the interactions between
musical instruments, loudspeakers, visualization devices humans and computers with musical interface in four major
for sheet music management, design of haptic devices, types [6]:
and development of multimedia platforms, with audio
 • Natural interaction: It corresponds to usual ways of
interaction between computers (e.g., pressing keys,
giving voice commands, and using a touch screen) and
musical instruments (e.g., blowing, rubbing or hitting).
• Event sequence: It corresponds to sequences of sonic
phenomena related to a particular task or steps in a
complex process. This type of interaction includes alarms
and sonified space.
• Process control: It corresponds to the monitoring
and automatic change of sound, caused by a
specific variable from the scenario (e.g., temperature,
humidity, illumination) or the subject (e.g., position,
gesture, facial expression, electrocardiographic, and
electroencephalographic signal).
• A mixture of two or more of the outlined types of
interaction.
Sodnik [7] distinguishes three main design strategies for this
kind of interfaces:
• Sonification, which is a method for shifting any source
of information into the auditory domain [8]. For instance,
an n-dimensional signal that is parametrized to produce
a specific sound or acoustic feature, hence mapping
every salient pattern identified in the original signal. The
sonified signal may have any source, for example, a
physiological or ambient variable, a spatial coordinate or
trajectory, and a set of parametrized gestures. For Kramer
[9], sonification is the transformation of data relations
into perceived relations in an acoustic signal, for the
purposes of facilitating communication or interpretation.
• Sound clues: Sound clues, points to virtual objects
representing acoustic phenomena, this is, a set of sounds
or acoustic features of a sound that allows for the
reproduction of data associated with the sound clue by
following a specific set of rules. This is also known as
sound icon [10].
• Interface metaphors, are physical objects that interact
with a given platform (i.e., sets of algorithms comprising
a software) in a typical way, such as that of a common
object or musical instrument. This strategy not
necessarily imply that the physical morphology of the
metaphor must coincide with the object that inspired its
function [11].
HCMI had then, special challenges to face in the following
years. For instance, the high diversity of projects in the area
produce incompatibility, which in turn resulted in a need
for standards. This represents a critical trouble that must be
tackled to reach major advances within the area of research.
Furthermore, subjects typically use bad quality headphones
and loudspeakers, thus the adoption of high quality audio
experiences may not always be well accepted by the market
[12].
III. M AIN R ESEARCH A REAS IN THE D ISCIPLINE
HCMI seems to be a wide research area with multiple
research objectives, motivations, tools and results. The subject
can be grouped in five major areas of research (combining
HCI and BCI methodologies), centering the classification on
research objectives or forms of interaction. These major areas
are: music composition interfaces, interactive sonification,
music recommendation systems, music therapy, and emergent
approaches.
A. Music Composition Interfaces
Alternatives of music composition by HCMI platforms
comprise the major group of proposals in the area. This
branch of research includes simple devices like modular
sound generators by analogue electronics [13], and platforms
that include electroencephalographic measures classified by
artificial intelligence that controls musical instruments [14].
This kind of interfaces can be divided in two major
areas: interfaces that incorporate HCI strategies and tools
and interfaces that include BCI strategies. The interfaces
that manage HCI strategies can be further divided in four
main areas: audio synthesis and hyper instruments, musical
composition based in human gestures, affectively - driven
algorithmic music generation, and platforms for collective
musical composition.
Various strategies have been applied for sound synthesis,
from hardware modification to software programming. Some
authors proposed the use of modular units of synthesis that
can be assembled and modified to produce more complex
sounds [13], or the use of microphones in different granular
materials for recording sound with unusual patterns [15].
Software proposals are centered in interactive genetic
algorithms, that can substitute the goal function by user’s
evaluation of the produced musical piece [16], or evaluation
of the mood evoked by the synthesized sound [17]. Another
strategies that does not make use of direct synthesis is the
sound production for musical accompaniment of a human
performer, based on fixed rules and recorded audio [18]. The
emotional state as an interpretation of electro-dermal
activity acquisition [19], or the use of a genetic algorithm for
selecting recorded chords based on the live play of a musical
instrument by the user [20].
Hyper instruments are electroacoustic devices that combine
acoustic and electronic instrument features for enhancing the
sound production [21]. Music composition interfaces is an
active research area since the beginning of the 1980s [22],
as it is centered in the distributed and simultaneous control
of multiple instruments, computer-based sound synthesis, and
keyboard interfaces; this made possible by the advent of MIDI
protocol. At the present time different interface metaphors
are implemented, like violins [23] and sitars [24]. Also,
there are different strategies for music instrument intervention,
like full modification of the instrument in the first case,
or pattern identification of a fully modified instrument –for
minimum intervention of similar instruments– in the second
case. Hyper instruments research is suffering a serious change
that combines other types of sound modification, such as
interactive sonification or haptic device development, which
seriously modify the traditional concepts in the area of
research.
Using human gestures for controlling audio data has its own
tradition and practices. Since the modification of conducting
practice on the Nineteenth century, the use of hand gestures
from the conductor for orchestra performance has developed
its own codification and variations [25]. The identification of
these gestures oriented to manage audio information is an
important research topic in HCMI, which includes proposals
ranging from traditional gestures on orchestra performances
[26], [27] , to motion capture of the whole body while dancing
[28], [29]. These proposals evolved to platforms that amplify
the traditional ways to understand music conduction, like
motion capture of hand gestures on Theremin playing [30],
sound control by feet movement [31], or eye tracking [32]. A
wide range of sensors have been used in this kind of
interfaces, like motion capture by low cost infrared sensor,
accelerometers and eye trackers. Also an increasing number of
strategies have been used to manage the relationship between
the gesture and the sound material. For instance, artificial
neuro-fuzzy networks [33], databases of recorded sounds [32],
and rules for conversion to MIDI format [31].
A special paradigm of music composition by human
computer interaction is the affectively-driven algorithmic
music composition (AAC). The term was developed mainly
by Miranda and colleagues [34], [35], [36], [37], but include
proposals of feeling-based interactive genetic algorithms [18],
[20], and proposals for electroencephalographic modification
of emotion [38], [37]. This kind of interfaces had a relevant
problem: the definition of emotion production, expression
and significance. Eerola and Vuoskoski [39] contrasted two
main models of emotion classified by the type of variable
for emotion characterization: discrete models (i.e., discrete
variables evaluated by Likert scale [40]) and dimensional
models (i.e., Russell’s Circumplex Model [41] with
continuous variables). Both models have been used on HCMI,
meaning that this novel approach combines knowledge of
different physiological and psychological paradigms that tend
to be measured in real time. Possibly, AAC platforms will
have a wide range of applications in music therapy [38] and
automatic musical accompaniment [42].
Musical composition has been understood as an individual
activity, but HCMI aids in obtaining better collective
experiences while elaborating music. These type of platforms
could be used as a way for investigating human interactions
in music production on real stages [43], [44] and also
in virtual environments [45], [46]. This kind of interfaces
combine metaphors and sound clues, mainly in virtual
reality proposals, to foster the interaction between two
or more people in an active process of sound selection
and mixing. A collaborative experience means producing
sonification of collaborative activities of daily life [43], as
well as interaction with uncommon interfaces like a tactile
surface [45] or a circular electro-mechanic music box [44].
Collaborative musical composition is an emergent approach
exploring the process of human interaction and emotions
evoked by creative activities [47]. HCMI that link BCI
strategies can be grouped in two classes: platforms that
sonify the electroencephalographic signal and platforms with
electroencephalographic management of the sound material.
The first type of platform was originally developed by Lucier
and has many purposes along time like artistic expression
[48] and music score transcription [49]. The second approach
has opened more possibilities of development, for example,
management of a musical instrument by disabled population
[50], [51], use of AAC with electroencephalographic signals
[52], and hybrid approaches between sonification and
voluntary modification of audio [53]. These systems have
also relied on diverse forms of data processing, such as
Hjorth analysis [50], bump modeling [49], linear discriminant
analysis (LDA) [52], P300 [54], steady state visually evoked
potentials (SSVEP) identification [42], [51], Laplace filtering,
and artificial neural networks [50]. Noticeable, this corpus
of research is the fundamental base for applications like
composition for quadriplegic population, or research of new
audio material based on electroencephalographic information.
Moreover, this line of research does not only pursue the
enhancement of the natural capabilities of musical expression,
but represents an opportunity for vulnerable population to
improve the quality of their lives through artistic expression.
Two major research lines are focused on this approach:
interfaces for music creation directed to persons with
disabilities, and interfaces for music education. These sort
of applications face complicated design challenges because
of the special needs of children and the wide range of
disabilities requiring optimal design processes. Furthermore,
the techniques used in this kind of populations, comprising the
use of special interface metaphors and methods for retrieval
of physiological variables, could not be applied uniformly to
the whole population.
Most recent efforts are focused on three types of disabilities:
blind people, hemiplegic, and quadriplegics. According to
Haenselmann [55], musical expressivity on blind population
can be reached by three different strategies: by using
devices with braille interfaces, musical sheet conversion
to braille format, and the use of musical keyboards with
braille messages for note identification. Anderson’s proposal
[56] for hemiplegic population uses limited mobility as
a way of interaction for music production; the system
identifies with infrared sensors the position of the subject on
wheelchair and maps the coordinates in terms of pitch and
amplitude of the sound being reproduced by loudspeakers.
Working with quadriplegic population requires a sophisticated
development, such as Miranda’s solution [57]. There, the
electroencephalographic signal is processed by a set of rules
that determinate the sound, played by a piano using MIDI
protocol, and reproduced by a loudspeaker. Naturally, this
proposal was capable of providing a experience of flexible
musical composition without the need, for the user, to produce
a single movement.
B. Interactive sonification
Interactive sonification is the second major area of research
of HCMI. Diniz [58] defines this kind of platforms as
the presentation of multiple non-linguistic sound variables,
for communication with a human being, especially in the
interaction with a space. He proposed two models of
sonification: one way sonification –with a fixed strategy of
sound production, and two way sonification –with a variable
strategy of sound production. Leider [12] discusses some
basic principles for the design of spaces for this kind of
purposes:
• Independence of the space instrumented from the original
recorded space of the sound used for the interface.
• Consistency and repeatability of the sound reproduced
must be assured with special care to the equalization of
sound.
• Feedback strategy not only considers functional
characteristics, but includes also aesthetic considerations
about the sound creation.
Another proposal, by Diniz [58], uses a camera with infrared
movement sensors, image projectors, loudspeakers and a
computer to link the variables. The interface detects the
posture and position of the subject, then reproduces a sound
varying its main features (e.g., timbre, pitch) as a function of
these two variables. Camurri [59] presented a prototype based
on the record of a musical piece for orchestra. In his setting,
every spatial section is related to a specific set of instruments
of the original musical piece. It can be argued that such an
strategy is directed to explore the timbre, given that different
sections of the orchestra are explored using the movement
of the subject. This system also makes use of an infrared
sensor for the movement, and loudspeakers connected with a
computer that map the spatial position and relates it to the
sound that will be reproduced. Furthermore, the space is not
necessarily explored with a direct physical interaction with
the subject, but also an auxiliary device can be used for this
purpose. Livingstone [60] developed a system named Orb3,
which has a plastic sphere with accelerometers, microphones,
and sensors. These are capable of producing sounds to reflect
the position, environmental conditions, and proximity with
another Orb3 devices.
Nevertheless, interactive sonification also refers to proposals
beyond space sonification. Human movement sonification
has been used in multiple applications, for instance, artistic
expression [28], [61], help on arm rehabilitation after stroke
[62], live automatic harmonization of melodies [63], music
conduction training aid [64], sonification of movement for
chronic pain rehabilitation [65] and management of sound
material –as preprocessing– for musical composition [26],
[31], [32], [30], [66]. This kind of sonification means an
interaction with the sound material in uncommon ways, thus
resulting in a new comprehension in contrast with traditional
techniques of musical composition. Also these proposals
introduce music techniques and material on new applications,
like rehabilitation programs or interactive expositions on
museums.
C. Music Recommendation Systems
Recommendation systems are software tools and techniques
providing suggestions for items to be of use to a user
[67]. Recommendation of audio content is a wide area of
research with multiple applications on entertainment industry.
Moreover, it is a topic that uses multiple novel techniques
(e.g., machine learning, artificial intelligence) [68], [69] for
giving better recommendations. Preliminary attempts had
diverse purposes, for instance, better sleep quality [70],
discovery of new music based on user preferences [71], [72],
singer identification [73], mood recognition [74], indexing
of huge audio databases [75], and automatic visualization
of music preferences [76]. Audio recommendation systems
can be grouped in two major areas according to the nature
of the content analyzed by the system: linguistic and
musical. Linguistic analysis systems incorporate the research
in metadata (e.g.,lyrics, tags) linked to the sound object [77],
[78]. Metadata is a rich source of information that can be
used for genre [79] or mood classification. Conversely, music
content analysis can be based on time [80], frequency [81],
and spectral features, graph analysis [82], and features in the
time-frequency domain, such as wavelets, and Mel Frequency
Cepstral Coefficients (MFCC) [83], [84].
Moreover, physiological data is used in the recommendation
to generate an accurate modification of the sonic environment,
in real time, for specific purposes, like relaxation [85];
incorporation of physiological data in recommendation will
become critical to improve the accuracy of recommendation
systems. This procedure could be applied for different
purposes, such as employing electroencephalographic signals
to elucidate musical preferences [86], sonification of
physiological variables [87], and subject mood modification
[37]. Furthermore, Liu [88] has determined subject’s mood
through their electrocardiographic signal, having the
advantage of synchronizing the tempo in recommended music
with users’ heartbeats.
A special mention is worth for audio recommendation systems
used for driver’s assistance. The specific requirements on
attention and commodity need a system with optimal driver
behavior and comfort. Most diverse approaches could be
assessed to accomplish this objective, for instance, automatic
music selection by scenario recognition [89], voice analysis
for mood recognition [90], electroencephalographic
recognition of drowsiness [91], or automatic music
equalization [92]. These systems might result in better driver
performance, but it must be acknowledged that they must pass
a full range of proofs on real performance to validate the
improvement in the subject behavior.
D. Therapeutic Applications
HCMI in therapy has become a central topic of research
during the last decade. Music therapy has increasingly been
accepted as a serious empirical field during the last 30 years,
but a standardized methodology is still far from reaching
a stability point in development. Nevertheless, there are
studies directed at identifying pathologies (i.e., unhealthy
behaviors) that could be palliated by the interaction with
the interface: hypertension and cardiovascular diseases [93],
[94], stress [95], [85], stroke rehabilitation [62], [96] and
depression [38]. The characteristics of the disease selected to
be evaluated, strongly influence the type of instrumentation
required to assure the security and efficiency of the
therapeutic intervention through HCMI.
Cardiovascular diseases (e.g., hypertension and tachycardia)
had demonstrated a positive response to music therapy.
Unfortunately, research about musical genres and audio
features has not reached yet a unified approach for music
therapy. This knowledge is critical for a personalized music
recommendation because it can optimize heartbeat frequency
and blood pressure. The origins of the relationship between
music hearing and cardiovascular behavior is still unclear:
Ho and Chen [93] propose tempo as the central feature
determining pulse, and Kim [97] is focused in the mood
communicated by the musical piece. Both cases regard
biofeedback as the fundamental design approach, but models
does not include yet the personal musical preference –or
personality– as an input for selecting the stimulus given to
the patient.
People use stress management by music in an instinctive
fashion, however, this state has not been defined as an
operative concept capable of being measured easily. Thus,
bioinstrumentation for measuring stress is a fuzzy topic in
the need of further research. Shin [95] proposes measuring
respiratory rate, electrocardiogram, and concentration of
carbon dioxide. And the stress is measured as an index of the
frequency spectrum of the electrocardiographic signal. Chang
[85] uses for this purpose the electroencephalographic signal,
but advances in this direction are still in initial stages.
An important advance is mood modification by musical
stimulus; two main related works have demonstrated, through
careful methodology and solid results, the efficacy of the
procedure. These are, Ram´ırez [38] work against depression,
and Daly’s [37] proposal on mood modification. Ram´ırez
found a relationship between Rusell’s Circumplex Model and
the electroencephalographic waves from specific electrodes.
The approach is an emergent alternative to measure emotions
by using objective and contrasting data. Daly [37] also
includes galvanic skin response and respiration rate, related
with subject’s emotion, estimating a personal rating through a
Support Vector Machine (SVM). After a classification
process, the system composes a melody based on a database
of piano sounds and a genetic algorithm. The system performs
basic accuracy on emotion classification, but it is now the
most relevant advance on this line of research.
The trend in HCMI is to gather the consolidation of
individualized therapeutic applications. This direction implies
a fundamental shift in the planning and execution of the
experiments, and the design of the interfaces, oriented to
fulfill the best standards in bioethics, and satisfy the
preconditions settled by ethics committees. It also implies that
the design can not be oriented to a proof of concept, rather, a
rigorous methodology to attain the efficacy of the interaction
and warrant the security for the clients in every step of the
therapy.
E. Emergent Approaches
Even though HCMI has been presented as a research
area with well defined research lines, still numerous studies
could be classified as emergent because their disciplinary
boundaries do not represent yet a differentiated research
line. This section summarizes the main characteristics of
this sort of HCMI, comprised by machine musicianship,
collective interfaces, virtual reality, and development of haptic
devices. Noticeably, these topics will seemingly represent the
future of the discipline, by increasing the management of
audio information in daily situations through new ways of
interaction.
For example, interaction between robot systems and human
beings should have its own classification, because the
applications tend to emulate social interaction with other
human being. This kind of interfaces were defined by
Hoffman [98] as robot musicianship, systems that could
measure environmental factors, psychological (e.g., emotion,
attention, drowsiness), and physiological variables (e.g., heart
and respiratory rate, galvanic skin response) for planning a
response adapted to individual needs. These responses could
be stored on a database or modified by artificial intelligence
(i.e., genetic algorithms or neural networks). In addition, this
kind of systems use facial recognition [99] and identification
of gestures linked to music performance [98]. Recently, there
are systems with automatic accompaniment based on real
time performance [100] and capable to interact with a human
performer.
Interestingly, it can be argued that the major part of the
interfaces developed in HCMI are oriented to individual
usage. Therefore, collective musical experiences are a novelty
area of development that entail a novel comprehension of the
musical experience, emerging from an introspective to social
sharing activities. This kind of interfaces combine different
variables from the social group: gesture recognition [101],
motion analysis [102], physical variables [103], and video
transmitted on real time [104]. The diversity and novelty of
the experience also brought a new kind of challenges that
must be solved. For example, evaluating the interface as a real
collective experience, assuring a real communication between
the participants and the interface, and management of multiple
input and output devices. These novel approaches on audio
experience have not become widely used devices, but will
increasingly change the understanding and practice of music
performance in different contexts.
This change in the traditional understanding of music hearing
also entails the use of virtual reality scenarios to provide a
novel experience of this activity. Some proposals are centered
on sonification of virtual environments [46], [105], systems
with virtual elements that control real instruments [45], and
the use of virtual interface metaphors for musical home
scenarios [106]. The advances in this area were
conceptualized by Vorla¨nder [107] as Virtual Room Acoustics.
This discipline, based in the traditional work of Room
Acoustics, might establish the basis for simulating the
behavior of sound on a virtual space by trying to emulate
the physical behavior of the space designed on real conditions.
Recent advances on availability on this technology and similar
proposals will be developed to contribute to the development
of virtual reality. Finally, there is the design of haptic devices
for musical composition, and sound reproduction and
management. This kind of devices allow enables a balance
between traditional musical instrument performance and new
ways of interaction through manual manipulation. The devices
are characterized by their creativity on design and purposes,
for example, foam spheres [108], gloves [109], rubber
balls [110], and
cubes [111]. This particular group of interfaces also have
its own purposes, like direct synthesis and playlist browsing
and reproduction. Most proposals are on preliminary stages
of design, but will gradually become the base of commercial
devices and alternatives of choice for music performance and
research.
IV. INSTRUMENTATION AND RELEVANT TOOLS
HCMI is supported by a full set of hardware and software
tools that can be used for development. Usually, development
of devices involved problems related to the compatibility
of the systems with computers and other devices, such as
musical instruments. The enormous variety of applications
emerged from this kind of interfaces does not only need
a theoretical framework, but also requires an explanation
about the main tools that could be used in the design and
construction of these devices. Concerning the particular
characteristics of music, these imply using specific tools
that can modulate the interaction between sounds and users.
Measuring physiological variables is a major matter of
disagreement in HCMI. For instance, the level of precision
–needed for the management of the interface– correlates
positively with the price and the computational demand of
the interfaces.
The capability to recognize every detail in the physiological
variables is needed in the context of a therapeutic interface,
but it is not essential for an interface designed for artistic
expression or audio management. To define the possibilities
and limitations of every kind of HCMI, it is crucial
to create a set of physiological measurements built
by the right physiological variables to be used in the
device. The most basic physiological measures used in
HCMI are those generated by the electrocardiogram, the
electroencephalograph, the galvanic skin response, and the
facial recognition. New devices featuring synchronization
and real time acquisition offer flexible measurements and
processing of the variables as an input for the management
of audio content and other capabilities of the interfaces.
Not unexpectedly, selection of physiological variables
and precision of measurements increase response times
of the system and physical connections with the user.
It is possible, however, to tackle this problem with the
use of new technologies, such as low cost wireless
electroencephalography (e.g., Emotiv TM EPOC), and
electrocardiographic devices (e.g., Zephyr TM Bioharness).
These apparatuses may prove comfortable for the user,
although measurements lack of acceptable precision if
compared with similar devices with clinical standards. Hence,
the use of this sort of devices must always be oriented to
fulfill user’s satisfaction.
Controlling musical instruments by electronic means is
a rich research topic. Communication can be enabled by
direct programming on a common platform (e.g., Matlab,
LabView), or via MIDI. MIDI is an old but robust technical
standard protocol, better understood if analyzed as stacked
layers, very much as the OSI (Open Systems Interconnection)
model. A hierarchy for these layers from top-down would
be, applications, software, and hardware. These layers are
linked to fulfill the purpose of transporting digitized musical
scores and performance subtleties, such as duration of notes
and velocity of attacks, among other things. Thus, through
digital interfaces and connectors, a wide variety of electronic
musical instruments, computers and other related devices can
communicate with one another [112]. Most operating systems
(e.g., Windows, OSX ) have generic MIDI players, which
are capable of producing sound by translating sequences
of orders encoded in a given score. The mechanism is
straightforward, just as it might occur with a musician
reading a sore, MIDI players select sounds from a database
to play them through a given output. No sound is encoded
in MIDI, so timbre and quality depend on how the particular
database was built; some of these might be comprised of
sounds produced by low quality synthesizers, some are
gigantic collections of samples that have reached the level
of industrial standards. Another protocol frequently used in
HCMI is Open Sound Control (OSC), developed at the Center
of New Music and Audio Technologies [113]; this is also
used for controlling synthesizers and multimedia elements
by computer programming. Furthermore, OSC is capable of
managing temporal information and network architecture that
allows for complex development on audio interfacing. Both
protocols are widely used in the control of HCMI.
V. D ISCUSSION
Conceptualization of HCMI represents a challenging
endeavor perhaps because of the asymmetry evoked by
historical baggage. This refers to the fact that music is
a technology with more than forty thousand years of
development, while the first generation of humans
transformed by computing is still alive. Due this circumstance,
a central problem emerge, which is the dispersion of the
theoretical framework at diverse levels, including the semantic
(i.e., jargon). It cannot be considered as a traditional
development in HCI because initial efforts were done in
parallel with traditional application of HCI. Moreover, the
inclusion of the applications of musical BCI on the
classification done by Andujar [114], of Artistic Brain
Computer Interface, does not include the numerous projects
on HCI, and the possibilities of interaction with other
variables like electrocardiography or galvanic skin response
are excluded. Thus, HCMI as an independent
multidisciplinary research topic is the best way to understand
this type of proposals, especially because of the use of the
music language, traditional communication (score and
recordings) and conceptualization of time and space in
interaction with the user.
Figure 1 shows a general framework for the design of HCMI.
This paradigm is a synthesis that reveals three domains in the
design of this kind of devices: the user domain, the computer
domain and the interaction media. This division is relevant
compared with other type of interfaces because the sound
could be modified by the physical or virtual space in which
the interaction is taking place. As the intuition suggest, the
structure of a HCMI is a feedback cycle capable to modify
both interaction elements, but the presence of the objective

Fig. 1. General design framework for a HCMI
in the top of the diagram implies that this feedback process
allows to reach the objective.
This vision of HCMI reflects a central problem in the
design of HCMI: the equilibrium between the user needs and
the possibility to generate new ways of musical expression
and comprehension as the origin of the device design. The
utilitarian vision of design implies a complete study of the
user needs and the technology available in order to satisfy the
need of the subject (or the collective). The radical innovation
principle imply a deep reflection about the experience and the
possibilities of the user and its interaction with the device in a
specific way (in this case the musical hearing and
performance experience). The framework presented allows to
set the main objective as the satisfaction of a user need or
could be a exploration that not necessarily satisfies the
subject. These alternatives could be a sound produced by the
interaction performed to an audience, the behavioral change
of a device
- like an haptic or a robot - that not necessarily give feedback
to the user but affects the media or other person. These
complexities must be carefully afforded in order to evaluate
the reliability of the device in the accomplish of the objective.
A fundamental issue in the design is the conception of the
designer about the user because the music effects on the
human would be evaluated from the conception of the
designer on music as a physical stimulus, a mathematical
structure or a cultural product. The user is understood in the
framework in three dimensions: own characteristics, needs
and physiological variables. These dimensions interact in a
natural way in music hearing and performance: needs could
driven the interaction or a need could be satisfied in the
interaction - but not necessarily it is the central objective of
the device - but are limited by the person’s characteristics
and the possibility to measure the physiological variables
during the feedback process. These dimensions had an
intrinsic conceptual problem to draw a clear division between
them. Thus, a clear reflection about the conception of the
user will clarify an adequate design of the device.
The computer domain also has particularities in HCMI. The
differences between the hardware and the software in the
input and the output domain - and also special interactions
that combine both possibilities - creates special considerations
reflected in the figure. As we see in the section about tools,
the physiological variables measure needs a clear pipeline
of input for a correct noise reduction and feature extraction
(specially in the case of the electroencephalographic record),
and the proper input devices of electronic music (e.g. musical
instruments or MIDI input devices) must be considered as a
whole in the interface device. Also the output pipeline has
devices (e.g. loudspeakers, set of lights,environmental control)
that must be considered in the control and the output pipeline,
with a special focus in crucial features like synchronization of
data, response rate of the device and interaction of different
software platforms for a common control. The essential
division between an output pipeline and a feedback pipeline
is the capabilities of the devices to give feedback to media
without the interaction of the central machine that control the
interaction - like embedded systems or interactive systems.
As it could be observed in the revision presented, the sources
of information of HCMI are heterogeneous, making them
difficult to summarize and understand. Therefore, a significant
effort must be applied for accelerating the progress of the
discipline. Researchers from this discipline should include the
previous work considering this wide range of publications,
to generate a common framework and a vision about the
progress and possibilities of the discipline. A preparation
in different areas like engineering, psychology, acoustics,
ergonomics, and machine learning, will be needed in future
developments to reach correct data interpretation and a correct
comprehension of the relationship between the interface and
the user. Moreover, an extended revision of the literature is
needed, not only focused in the traditional time period of
engineering projects, but an extensive period to improve the
heuristic cycle. To find relevant works that might enhance
the understanding about previous work and analyze new
possibilities for design proposals.
Accordingly, analytic tools for audio retrieval are needed on
HCMI because prior works are centered in music developed
by the user or recommendation systems of music recordings.
Our comprehension of the human response to audio should
be based on our own analytic comprehension of the stimulus
given by the user. Gradually, most recent techniques of
classification could give us a better comprehension of how
the user is affected by specific characteristics of the stimulus.
This phenomena is dependent on the background, personality,
and culture of the user. Hence, a complex taxonomy of the
musical taste must be done to identify common patterns in
people. This effort is especially important for the development
of therapeutic applications because a detailed comprehension
of human physiology in health and disease conditions may
serve to attain a better behavior of human body on disease,
with the use of a HCMI.
Several unclassified emergent proposals are included in their
respective section. This situation is also a consequence of the
technological changes in multimedia systems of our times.
The changes on the devices and methodologies in sound and
multimedia management also extend the possibilities of
HCMI
in the context of ubiquitous music. Progressively, trends on
individualized consumption of multimedia works and novelty
in music performance –for entertainment and health purposes–
may direct the proposals on HCMI as common devices in
daily life. Also these changes open a clear opportunity for
specialists, to challenge their traditional understanding of
music if searching for new forms of interaction to benefit
users through music.
Finally, it is important to acknowledge the social impact of
the proposals of HCMI. The improvement of life quality
of different populations is present on the different research
lines within the area. The work with disabled populations,
the collective applications, and the first therapeutic interfaces,
indicate a social concern about the research involving music
features as central elements. The multidisciplinary nature of
this research area will improve the quality and diversity of
these interfaces that will doubtlessly improve the life quality
of societies through music.
VI. F UTURE WORK
The use of A.I. on HCMI is an area of potential
development of the discipline [115]. As we see in the analysis
of AAC, the incorporation of genetic algorithms in the
composition process could give better results that other types
of approaches. Recent proposals of environmental control
presents the neurofuzzy algorithms that allows to link the user
state with the response of actuators. Also unsupervised
learning algorithms with adequate feature like k-means could
be capable of categorizing the sound material related with the
subject response in order to describe a stable behavior of the
interface with different users. A.I. gradually will be solving
problems about the pattern identification of the human
behavior with an auditory stimulus and it will allow to think in
systems integrated on new spaces like houses or classrooms.
Also this interaction will produce an effective integration with
other machines, like robots or hyperinstruments.
The HCMI composition devices are centered on Western
conventions of music production. Thus, the future
development of HCMI will incorporate other musical
traditions. The research on non Western musical traditions is
centered on the mathematical description of traditional songs,
mainly in Hindustani [116], [117], [118], [119], [120],
Chinese [121],
[122], Greek [123], [124] and Turkish [125], [126] traditions.
Recently, researchers gradually incorporates experiments on
music therapy [127] and analysis of the music effect on cross-
cultural music samples [128], [129]. A central problem in the
dialogue between musical cultures is the concept of
consonance linked to physiological reactions provoked by
this sort of sounds in the human bodies. McDermott [130]
criticizes the Western point of view about consonance,
providing evidence about the similar judgment of consonant
and dissonant chords as pleasant on Amazonian population.
This result contradict previous evidence [131], thus the origin
of the effects of music therapy as a reaction in a cultural
context or a innate response to sound remains open.
The discussion between consonance and dissonance also
implies an enquire about the type of sound that is provided to
the user in the HCMI. Successful applications are studied with
pink noise [132] and binaural beats [133], [134]. The stress
reduction and tinnitus management of both type of sounds is
not well understood, thus the potential application of this kind
of sound material on HCI must be considered as a potential
therapeutic agent applied on this kind of interfaces. Also, a
deep analysis is necessary to learn about the links between
mechanisms of physiological reactions and sound stimuli,
both represented as sets of features. This problem is explored
in recent studies about the relationship between music and the
human voice [135], claim that correlates of the frequency
pattern used for the expression of happy and sad music (major
and minor scales) is similar with typical oral phrases that
express different emotions. This relationship also could
explain the cultural differences between folk music of non
Western musical traditions.
VII. C ONCLUSIONS
This paper presents a survey on Human Computer Music
Interfaces. The work presents a historical framework from the
experiments of the sixties of the past century and theoretical
background on sonification, ways of interaction and design
strategies. Five major areas were identified: music
composition interfaces, interactive sonification, music
recommendation systems, therapeutic application and
emergent approaches. Basic tools for the development of
HCMI were identified: physiological variables acquisition
devices and communication protocols. A multidisciplinary
background and a rigorous revision of the state of the art is
needed for future developments in the area.
REFERENCES
[1] S. Holland, K. Wilkie, P. Mulholland, and A. Seago, Music and human-
computer interaction, Springer, 2013, p.13.
[2] D. Wessel, and M. Wright, ”Problems and prospects for intimate musical
control of computers.” Computer Music Journal, vol. 26, no. 3, pp. 11-
22, 2002.
[3] O.E. Laske. ”Toward a theory of interfaces for computer music systems.”
Computer Music Journal, pp. 53-60, 1977.
[4] B. C. Pijanowski, L. J. Villanueva-Rivera, S. L. Dumyahn, A. Farina, B.
L. Krause, B. M. Napoletano, S. H. Gage, and N. Pieretti, ”Soundscape
ecology: the science of sound in the landscape.” BioScience, vol. 61, no.
3, pp. 203 - 216, 2011.
[5] D. Keller, F.V. Flores, M.S. Pimenta, A. Capasso, and P. Tinajero,
”Convergent trends toward ubiquitous music.” Journal of New Music
Research, vol. 40, no. 3, pp. 265-276, 2011.
[6] L. Flores, E. Miletto, M. Pimenta, E. Miranda, and D. Keller, ”Musical
interaction patterns: communicating computer music knowledge in a
multidisciplinary project.” In Proceedings of the 28th ACM International
Conference on Design of Communication, ACM, 2010, pp. 199-206.
[7] J. Sodnik, and S. Tomazic, Spatial Auditory Human-computer Interfaces,
Springer, 2015, pp. 40-43.
[8] T. Hermann, A. Hunt, and J. G. Neuhoff, ”The sonification handbook,”
Logos Verlag Berlin, 2011, p.10.
[9] G. Kramer ”An introduction to auditory display,” in G. Kramer (Ed.),
Auditory Display: Sonification, Audification, and Auditory Interfaces,
Reading, MA: Addison Wesley, 1994, pp. 178.
[10] J Sa´nchez, & F. Aguayo, ”Mobile inclusion through science learning for
the blind.” Annual Review of CyberTherapy and Telemedicine, vol. 117,
2008.
[11] M. Duignan, J. Noble, P. Barr, & R. Biddle, ”Metaphors for electronic
music production in Reason and Live.” In Asia-Pacific Conference on
Computer Human Interaction Springer Berlin Heidelberg, 2004, pp. 111-
120.
[12] C. Leider, ”Multichannel Audio in Electroacoustic Music: An Aesthetic
and Technical Research Agenda.” In Multimedia and Expo, 2007 IEEE
International Conference on, IEEE, 2007, pp. 1890-1893.
[13] S. Gelineck. and S. Serafin, ”Phoxes-modular electronic music
instruments based on physical modeling sound synthesis.” In 7th Sound
and Music Computing Conference, 2010.
[14] E.R. Miranda, K. Sharman, K. Kilbor, and A. Duncan. ”On harnessing
the electroencephalogram for the musical braincap.” Computer Music
Journal, vol. 27, no. 2, pp. 80-102, 2003.
[15] S. O’Modhrain, and G. Essl, ”PebbleBox and CrumbleBag: tactile
interfaces for granular synthesis.” In Proceedings of the 2004 conference
on New interfaces for musical expression National University of
Singapore. 2004, pp. 74-79.
[16] A. Moroni, F. Von Zuben, and J. Manzolli, ”ArTbitration: Human-
machine interaction in artistic domains.” Leonardo, vol. 35, no. 2,
pp. 185-188, 2002.
[17] H. Zhu, S. Wang, and Z. Wang, ”Emotional music generation using
interactive genetic algorithm.” In Computer Science and Software
Engineering, 2008 International Conference on vol. 1, IEEE, 2008, pp.
345-348.
[18] S. Nakashima, Y. Imamura, S. Ogawa, and M. Fukumoto ”Generation
of Appropriate User Chord Development Based on Interactive Genetic
Algorithm.” In P2P, Parallel, Grid, Cloud and Internet Computing
(3PGCIC), 2010 International Conference on, IEEE, 2010, pp. 450-453. [19]
J.W. Chung, and G.S. Vercoe. ”The affective remixer: Personalized music
arranging.” In CHI06 extended abstracts on Human factors in
computing systems, ACM, 2006, pp. 393-398.
[20] M. Fukumoto ”Creation of Music Chord Progression Suited for Users
Feelings Based on Interactive Genetic Algorithm.” in Advanced Applied
Informatics (IIAIAAI), 2014 IIAI 3rd International Conference on, IEEE,
2014, pp. 757-762.
[21] G. Odowichuk, S. Trail, P. Driessen, W. Nie , and W. Page ”Sensor
fusion: Towards a fully expressive 3d music control interface.” In
Communications, Computers and Signal Processing (PacRim), 2011
IEEE Pacific Rim Conference on, IEEE, 2011, pp. 836-841.
[22] T. Machover, Hyperinstruments: A Progress Report, 1987-1991, MIT
Media Laboratory, p. 1-14, 1992.
[23] F. Bevilacqua, N. Rasamimanana, E. Fle´ty, S. Lemouton, and F. Baschet, ”The
augmented violin project: research, composition and performance report.” In
Proceedings of the 2006 conference on New interfaces for musical expression,
IRCAMCentre Pompidou, 2006, pp. 402-406.
[24] A. Tindale, A. Kapur, and G. Tzanetakis, ”Training surrogate sensors
in musical gesture acquisition systems.” Multimedia, IEEE Transactions
on, vol. 13, no. 1, pp. 50-59, 2011.
[25] J. A. Bowen, The Cambridge companion to conducting. Cambridge
University Press. 2003
[26] J.S. Prasad, G.C. Nandi, and A. Kumar, ”Gesture based music
generation.” In Emerging Trends in Engineering and Technology
(ICETET), 2009 2nd International Conference on, IEEE, 2009, pp. 209-
214.
[27] A. Sarasua, M. de Catalunya, B. Caramiaux, and A. Tanaka, ”Machine
Learning of Personal Gesture Variation in Music Conducting.” In
Proceedings of the 2016 CHI Conference on Human Factors in
Computing Systems, ACM, 2016, pp. 3428-3432.
[28] P.H. Vulliard, J. Larralde, and M. Desainte-Catherine ”Re´troaction
musique-physiologie: une approche selon le paradigme des emotions.” In
Journe´s dInformatique Musicale, 2012.
[29] A.R. Jensenius, Action-sound: Developing methods and tools to study
music-related body movement. PhD Thesis 2007.
[30] C. Geiger, H. Reckter D. Paschke, and F. Schulz ”Evolution of a
Theremin-Based 3D Interface for Music Synthesis” In 3D User
Interfaces, 2008. 3DUI 2008. IEEE Symposium on, IEEE, 2008, pp.
163-164.
[31] M. Fujimoto, N. Fujita, Y. Takegawa, T. Terada, and M. Tsukamoto
Musical B-boying: A Wearable Musical Instrument by Dancing. In
Entertainment Computing-ICEC 2008, Springer Berlin Heidelberg, 2008,
pp. 155-160.
[32] Z. Vamvakousis, and R. Ramirez, ”The eyeharp: An eye-tracking-based
musical instrument.” In 8th Sound and Music Computing Conference,
2011.
[33] J.Y. Lee, J.E. Jung, and H.J. Kim ”A Real-time Hand Gesture
Recognition Technique and Its Application to Music Display System.”
Journal of Automation and Control Engineering vol. 4, no. 2, 2016.
[34] I. Daly, A. Malik, J. Weaver, F. Hwang, S.J. Nasuto, D. Williams, and E.
Miranda ”Towards human-computer music interaction: Evaluation of an
affectively-driven music generator via galvanic skin response measures.”
In Computer Science and Electronic Engineering Conference (CEEC),
2015 7th, IEEE, 2015, pp. 87-92.
[35] A. Kirke, E.R. Miranda, and S. Nasuto ”Learning to make feelings:
Expressive performance as a part of a machine learning tool
for sound-based emotion therapy and control. In Cross-Disciplinary
Perspectives on Expressive Performance Workshop: Proceedings of the
9th International Symposium on Computer Music Modeling and
Retrieval, London, UK, 2012, pp. 19-22.
[36] D. Williams, A. Kirke, J. Eaton, E. Miranda, I. Daly, J. Hallowell,
and S.J. Nasuto. ”Dynamic Game Soundtrack Generation in Response
to a Continuously Varying Emotional Trajectory.” In Audio Engineering
Society Conference: 56th International Conference: Audio for Games,
Audio Engineering Society, 2015.
[37] I. Daly, D. Williams, A. Kirke, J. Weaver, A. Malik, F. Hwang, F. and SJ
Nasuto, ”Affective brain computer music interfacing.” Journal of Neural
Engineering, vol. 13, no. 4, 046022, 2016.
[38] R. Ramirez, M. Palencia-Lefler, S. Giraldo, and Z. Vamvakousis,
”Musical neurofeedback for treating depression in elderly people.”
Frontiers in neuroscience, 9. 2015
[39] T. Eerola, and J.K. Vuoskoski, ”A comparison of the discrete and
dimensional models of emotion in music.” Psychology of Music. vol. 39,
no. 1, pp. 1849, 2010.
[40] M.T. Mitterschiffthaler, C.H. Fu, J.A. Dalton, C.M. Andrew, S.C.
Williams. ”A functional MRI study of happy and sad affective states
induced by classical music.” Human brain mapping, vol. 28, no. 11, pp.
1150-1162, 2007.
[41] J.A. Russell, ”A circumplex model of affect.” Journal of personality and
social psychology, vol. 39, no. 6, pp. 1161, 1980.
[42] J. Eaton, and E. Miranda, ”Expanding on Brain Control: A real
time hybrid BCMI for drum machine combining active (SSVEP) and
passive (affective response) brainwave control.” In 9th Conference on
Interdisciplinary Musicology CIM14, 2014.
[43] E. Morgan, H. Gunes, and N. Bryan Kinns ”Instrumenting the
Interaction: Affective and Psychophysiological Features of Live
Collaborative Musical Improvisation.” In NIME, 2014, pp. 23-28.
[44] G. Novembre, M. Varlet, S. Muawiyath, J.C. Stevens, and P.E. Keller.
”The E-music box: an empirical method for exploring the universal
capacity for musical production and for social interaction through music.”
Royal Society Open Science, 2(11), 150286, 2015.
[45] M.C. Pichiliani, and C.M. Hirata ”A tabletop groupware system
for computer-based music composition.” In Collaborative Computing:
Networking, Applications and Worksharing, 2009. CollaborateCom
2009. 5th International Conference on, IEEE, 2009, pp. 1-10.
[46] S. Barrass, and T. Barrass ”Musical creativity in collaborative virtual
environments.” Virtual Reality vol. 10, no. 2, pp. 149-157, 2006.
[47] E. Morgan, H. Gunes, and N. Bryan-Kinns ”Using affective and
behavioural sensors to explore aspects of collaborative music making.”
International Journal of Human-Computer Studies, vol. 82, pp. 31-47,
2015.
[48] D. Rosenboom, Biofeedback and the arts. Result of early experiments.
Vancouver, BC, Canada: Aesthetic Res Center Can Publ, 1975
[49] T.M. Rutkowski, F. Vialatte, A. Cichocki, D.P. Mandic and A.K. Barros,
”Auditory feedback for brain computer interface managementan EEG
data sonification approach.” In Knowledge-Based Intelligent Information
and Engineering Systems, Springer Berlin Heidelberg, 2006, pp. 1232-
1239.
[50] E.R. Miranda, and A. Brouse ”Toward direct brain-computer musical
interfaces.” In Proceedings of the 2005 conference on New interfaces for
musical expression, National University of Singapore, 2005, pp. 216-219.
[51] S. Le Groux, J. Manzolli, P.F. Verschure, M. Sanchez, A.
Luvizotto,
A. Mura and U Bernardet, ”Disembodied and Collaborative Musical
Interaction in the Multimodal Brain Orchestra.” In NIME, 2010, pp. 309-
314.
[52] S. Makeig, G. Leslie, T. Mullen, D. Sarma, N. Bigdely-Shamlo, & C.
Kothe, C. ”First demonstration of a musical emotion BCI.” In Affective
Computing and Intelligent Interaction, Springer Berlin Heidelberg, 2011,
pp. 487-496.
[53] Y.C.D. Chew, and E. Caspary ”MusEEGk: a brain computer musical
interface.” In CHI11 Extended Abstracts on Human Factors in
Computing Systems, ACM, 2011, pp. 1417-1422.
[54] A. Pinegger, S.C. Wriessnegger, and G.R. Muller-Putz ”Sheet music by
mind: Towards a brain computer interface for composing.” in
Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual
International Conference of the IEEE IEEE. 2015, pp. 1053-1056.
[55] T. Haenselmann, H. Lemelson, and W. Effelsberg, ”A zero-vision music
recording paradigm for visually impaired people.” Multimedia Tools and
Applications, vol. 60, no. 3, pp. 589-607, 2012.
[56] T. Anderson, ”Using music performance software with flexible control
interfaces for live performance by severely disabled musicians.” In
 EUROMICRO Conference, 1999. Proceedings. 25th, Vol. 2, IEEE, 1999,
pp. 20-27.
[57] E.R. Miranda, ”Plymouth brain-computer music interfacing project:
from EEG audio mixers to composition informed by cognitive
neuroscience.” International Journal of Arts and Technology, vol. 3, no.
2-3, pp. 154-176, 2010.
[58] N. Diniz, P. Coussement, A. Deweppe, M. Demey, and M. Leman, ”An
embodied music cognition approach to multilevel interactive sonification.” [80] D. Kim, K.S. Kim, K.H. Park, J.H. Lee, and K.M. Lee, ”A music
Journal on Multimodal User Interfaces, vol. 5, no. 3-4, pp. 211-219, 2012.
[59] A. Camurri, and G. Volpe. ”Active and personalized experience of sound
and music content.” In Consumer Electronics, 2008. ISCE 2008. IEEE
International Symposium on, IEEE, 2008, pp. 1-4.
[60] D. Livingstone, and E. Miranda, ”Orb3: adaptive interface design for
real time sound synthesis and diffusion within socially mediated spaces.” [82] A. Theodoridis, C. Kotropoulos, and Y. Panagakis, ”Music
In Proceedings of the 2005 conference on New interfaces for musical
expression, National University of Singapore, 2005, pp. 65-69.
[61] J. Carreira, and P. Peixoto, ”A vision based interface for local
collaborative music synthesis.” In Automatic Face and Gesture
Recognition, 2006. FGR 2006. 7th International Conference on, IEEE,
2006, pp. 591-596.
[62] D. S. Scholz, S. Rohde, N. Nikmaram, H.P. Brckner, M. Groβbach, J.
D. Rollnik, E. O. Altenmu¨ller. ”Sonification of arm movements in stroke
rehabilitation: a novel approach in neurologic music therapy.” Frontiers
in Neurology doi: 10.3389/fneur.2016.00106, 2016, vol. 9.
[63] J. Nika, J. Echeveste, M. Chemillier, and J.L. Giavitto, Planning Human-
Computer Improvisation Ann Arbor, MI: Michigan Publishing,
University of Michigan Library, 2014.
[64] R. Schramm, C. Rosito Jung, and E. Reck Miranda, ”Dynamic time
warping for music conducting gestures evaluation.” Multimedia, IEEE
Transactions on, vol. 17, no. 2, pp. 243-255, 2015.
[65] J.W. Newbold, N. Bianchi-Berthouze, N.E. Gold, A. Tajadura-Jime´nez,
& A.C. Williams ”Musically Informed Sonification for Chronic Pain
Rehabili-tation: Facilitating Progress & Avoiding Over-doing.” In
Proceedings of the 2016 CHI Conference on Human Factors in
Computing Systems, ACM, May 2016, pp. 5698-5703.
[66] K.A. Yu¨ksel, H. Ketabdar, & M. Roshandel ”Towards digital music
performance for mobile devices based on magnetic interaction”. In
Haptic Audio-Visual Environments and Games (HAVE), 2010 IEEE
International Symposium on, IEEE, October 2010, pp. 1-6.
[67] F. Ricci, L Rokach, B Shapira, PB Kantor, Recommender Systems
Handbook, New York USA: Spriner Science + Business Media, pp. 1.
[68] M. Lesaffre, and M. Leman, ”Using fuzzy logic to handle the users’
semantic descriptions in a music retrieval system.” In Theoretical
Advances and Applications of Fuzzy Logic and Soft Computing, Springer
Berlin Heidelberg. 2007, pp. 89-98.
[69] S.W. Bang, J. Kim, and J.H. Lee, ”An approach of genetic programming
for music emotion classification.” International Journal of Control,
Automation and Systems, vol. 11, no. 6, 2013, pp. 1290-1299.
[70] W. Zhao, X. Wang, and Y. Wang, ”Automated sleep quality
measurement using EEG signal: first step towards a domain specific
music recommendation system.” In Proceedings of the 18th ACM
international conference on Multimedia ACM, 2010, pp. 1079-1082.
[71] E. Pampalk, and M. Goto, ”MusicRainbow: A New User Interface to
Discover Artists Using Audio-based Similarity and Web-based
Labeling.” In ISMIR, 2006, pp. 367-370.
[72] M.M.R. Siddiquee, M.S. Rahman, N. Haider, R.M. Rahman, S.U.I.
Chowdhury, and S. Banik, ”A personalized music discovery service
based on data mining.” In Computer and Information Science (ICIS),
2015 IEEE/ACIS 14th International Conference on, IEEE, 2015, pp.
253-258.
[73] P. Chang, (2009, September). ”Pitch oriented automatic singer [95] I.H. Shin, J. Cha, G.W. Cheon, C. Lee, S.Y. Lee, H.J. Yoon, and
identification in pop music.” In Semantic Computing, 2009. ICSC’09.
IEEE International Conference on, IEEE, 2009, pp. 161-166.
[74] C.H. Yeh, W.Y. Tseng, C.Y. Chen, Y.D. Lin, Y.R. Tsai, H.I. Bi, and
H.Y. Lin, ”Popular music representation: chorus detection and emotion
recognition.” Multimedia Tools and Applications, vol. 73, no. 3, pp.
2103-2128, 2014.
[75] H. Blume, B. Bischl, M. Botteck, C. Igel, R. Martin, G. Roetter, and C.
Weihs, ”Huge music archives on mobile devices.” IEEE Signal
Processing Magazine, vol. 28, no. 4, pp. 24-39, 2011.
[76] D. Bogdanov, M. Haro, F. Fuhrmann, A. Xambo´, E. Go´mez, and P.
Herrera, P. Semantic audio content-based music recommendation and
visualization based on user preference examples. Information Processing
and Management, vol 49, no. 1, pp. 13-33, 2013.
[77] M. Sordo, C. Laurier, and O. Celma, ”Annotating Music Collections:
How Content-Based Similarity Helps to Propagate Labels.” In ISMIR,
September 2007, pp. 531-534.
[78] M. Tkalcˆicˆ, M. Schedl, C. Liem, and M.S. Melenhorst, ”Personalized
Retrieval and Browsing of Classical Music and Supporting Multimedia
Material.” In Proceedings of the 2016 ACM on International Conference
on Multimedia Retrieval, ACM, 2016, pp. 393-396.
[79] R. Mayer, R. Neumayer, and A. Rauber, ”Rhyme and Style Features
for Musical Genre Classification by Song Lyrics.” In ISMIR, 2008, pp.
337-342.
recommendation system with a dynamic k-means clustering algorithm.”
In Machine Learning and Applications, 2007. ICMLA 2007. Sixth
International Conference on, IEEE, 2007, pp. 399-403.
[81] M.A. Casey, and M. Slaney, ”Song Intersection by Approximate Nearest
Neighbor Search.” in ISMIR, vol. 6, 2006, pp. 144-149.
recommendation using hypergraphs and group sparsity.” in 2013
IEEE International Conference on Acoustics, Speech and Signal
Processing, IEEE, 2013, pp. 56-60.
[83] M.I. Mandel, and D. Ellis, ”Song-Level Features and Support Vector
Machines for Music Classification.” In ISMIR vol. 2005, 2005, pp. 594-
599.
[84] YH Chin, SH Lin, CH Lin, E Siahaan, A Frisky, and JC
Wang ”Emotion profile-based music recommendation.” In 2014 7th
International Conference on Ubi-Media Computing and Workshops, 2014.
[85] H.Y. Chang, S.C. Huang, and J.W. Wu, ”A personalized music
recommendation system based on electroencephalography feedback.”
Multimedia Tools and Applications, vol. 1, no. 20, 2016.
[86] Y. Pan, C. Guan, J. Yu, K.K. Ang, and T.E. Chan, ”Common frequency
pattern for music preference identification using frontal eeg.” In Neural
Engineering (NER), 2013 6th International IEEE/EMBS Conference on ,
IEEE, 201, pp. 505-508.
[87] I. Bergstrom, S. Seinfeld, J. Arroyo-Palacios, M. Slater, and M.V.
Sanchez-Vives, ”Using music as a signal for biofeedback.” International
Journal of Psychophysiology, vol. 93, no. 1, pp. 140-149, 2014.
[88] H. Liu, J. Hu, and M. Rauterberg, ”Music playlist recommendation based
on user heartbeat and music preference.” in 2009 International
Conference on Computer Technology and Development . IEEE, 2009, pp.
545-549.
[89] Y. Kinoshita, Y. Masaki, T. Muto, K. Ozawa, and T. Ise, ”Scenery
based Kansei music selection for car audio systems.” In 2009 IEEE
13th International Symposium on Consumer Electronics, IEEE, 2009, pp.
94-98.
[90] C. Forlines, B. Schmidt Nielsen, B. Raj, K. Wittenburg, and P. Wolf, ”A
comparison between spoken queries and menu-based interfaces for in-car
digital music selection.” In Human-Computer Interaction-INTERACT
2005, Springer Berlin Heidelberg, 2005, pp. 536-549.
[91] N.H. Liu, C.Y. Chiang, and H.M. Hsu, ”Improving driver alertness
through music selection using a mobile EEG to detect brainwaves.”
Sensors, vol. 13, no. 7, pp. 8199-8221, 2013.
[92] H.G. Kim, and J.M. Cho, ”Car audio equalizer system using music
classification and loudness compensation.” In ICTC 2011, IEEE, 2011,
pp. 553-558.
[93] T.C. Ho, X. Chen, ”iHeartLift: A closed loop system with bio-feedback
that uses music tempo variability to improve heart rate variability.”
In Engineering in Medicine and Biology Society, EMBC, 2011 Annual
International Conference of the IEEE, IEEE, 2011, pp. 1181-1184.
[94] B. van der Vlist, C. Bartneck, and S. Ma¨ueler, ”moBeat: Using
interactive music to guide and motivate users during aerobic exercising.”
Applied psychophysiology and biofeedback, vol. 36, no. 2, pp. 135-145,
2011.
H.C. Kim, ”Automatic stressrelieving music recommendation system
based on photoplethysmography derived heart rate variability analysis.”
in Engineering in Medicine and Biology Society (EMBC), 2014 36th
Annual International Conference of the IEEE IEEE, 2014, pp. 6402-
6405.
[96] D. Soriano, E.L. Silva, G.F. Slenes, F.O. Lima, L.F. Uribe, G.P. Coelho,
... & C.A. Anjos, ”Music versus motor imagery for BCI systems a study
using fMRI and EEG: Preliminary results.” in Biosignals and Biorobotics
Conference (BRC), 2013 ISSNIP, IEEE, February 2013, pp. 1-6.
[97] J. Kim , and E. Andre´, ”Emotion recognition based on physiological
changes in music listening.” Pattern Analysis and Machine Intelligence,
IEEE Transactions on, vol. 30, no. 12, pp. 2067-2083, 2008.
[98] G. Hoffman, and G. Weinberg, ”Interactive improvisation with a robotic
marimba player.” Autonomous Robots, vol. 31, no. 2-3, pp. 133-153, 2011.
[99] J.S. Park, G.J. Jang, and Y.H. Seo, ”Music-aided affective interaction
between human and service robot.” EURASIP Journal on Audio, Speech,
and Music Processing, vol. 2012, no. 1, pp. 1-13, 2012.
[100] P. Verma, & P. Rao, ”Real-time melodic accompaniment system
for indian music using TMS320C6713,” in 2012 25th International
Conference on VLSI Design, IEEE, January 2012, pp. 119-124.
[101] F. Farbiz, K.Y. Tang, C. Manders, C.J. Herng, Y.K. Tan, K. Wang,
& W. Ahmad, ”A multimodal augmented reality DJ music system,”
In Information, Communications and Signal Processing, 2007 6th
International Conference on (pp. 1-5). IEEE, December 2007, pp. 1-5.
[102] G. Varni, G. Volpe, and B. Mazzarino, ”Towards a Social Retrieval of
Music Content.” in Social-Com/PASSAT (2011, pp. 1466-1473.
[103] G. Varni, M. Mancini, G. Volpe, & A. Camurri, ”A system for mobile
active music listening based on social interaction and embodiment,”
Mobile Networks and Applications, vol. 16, no. 3, pp. 375-384, 2011.
[104] M.F. Schober, ”Virtual environments for creative work in collaborative
music-making,” Virtual Reality, vol. 10, no. 2, pp. 85-94, 2006.
[105] M. Schoeffer, J.L. Gernert, M. Neumayer, S. Westphal, and J. Herre,
”On the validity of virtual reality based auditory experiments: a case
study about ratings of the overall listening experience.” Virtual Reality,
vol. 19, no. 3-4, pp. 181-200, 2015.
[106] J.M. Hsu, ”Design and evaluation of virtual home objects with music
interaction in smart homes,” Journal of Intelligent Manufacturing, vol.
23, no. 4, pp. 1281-1291, 2012.
[107] M. Vorla¨nder, S. Pelzer, and F. Wefers, ”Virtual Room Acoustics,” in
Sound-Perception-Performance, Springer International Publishing, 2013,
pp. 219-242.
[108] C. Kiefer, ”Input devices and mapping techniques for the intuitive
control of composition and editing for digital music,” in Proceedings of
the fourth international conference on tangible, embedded, and embodied
interaction, ACM, 2010, pp. 311-312.
[109] H.H. Ip, K.C. Law, and B. Kwong, ”Cyber composer: Hand gesture-
driven intelligent music composition and generation.” in Multimedia
Modelling Conference, 2005. MMM 2005. Proceedings of the 11th
International, IEEE, 2005, pp. 46-52.
[110] T. Yamaguchi, and S. Hashimoto, ”Grasping Interface with Photo
Sensor for a Musical Instrument,” In Human Computer Interaction.
Novel Interaction Methods and Techniques, Springer Berlin Heidelberg,
2009, pp. 542-547.
[111] A.M. Bruns, and V. Keyson, ”MusicCube: a physical experience with
digital music.” Personal and Ubiquitous Computing, vol. 10, no. 2-3, pp.
163-165, 2006.
[112] A. Swift, ”A brief Introduction to MIDI”, in SURPRISE, Imperial
College of Science Technology and Medicine, May 1997.
[113] A. Schmeder, A. Freed, and D. Wessel, ”Best practices for open sound
control,” in Linux Audio Conference, vol. 10, 2010.
[114] M. Andujar, C.S. Crawford, A. Nijholt, F. Jackson, and J.E. Gilbert,
”Artistic brain-computer interfaces: the expression and stimulation of the
users affective state.” Brain-computer interfaces, vol. 2, no. 2-3, 2015,
pp. 60-69.
[115] P. Lucas, & E. Pel, E. ”Human-machine musical composition in real-
time based on emotions through a fuzzy logic approach.” In 2015 Latin
America Congress on Computational Intelligence (LA-CCI), IEEE,
October 2015, pp. 1-6.
[116] S. Chakraborty, G. Mazzola, S. Tewari, & M. Patra, Computational
Musicology in Hindustani Music, Springer, 2014, p. 65.
[117] G.K. Koduri, S. Gulati, P. Rao, & X. Serra ”Raga recognition based
on pitch distribution methods.” Journal of New Music Research, vol. 41,
no. 4, pp. 337-350, 2012.
[118] S. Gulati, A. Bellur, J. Salamon, V. Ishwar, H.A. Murthy, & X.
Serra ”Automatic tonic identification in Indian art music: approaches and
evaluation.” Journal of New Music Research, vol. 43, no. 1, pp. 53-71,
2014.
[119] P. Rao, J.C. Ross, K.K. Ganguli, V. Pandit, V. Ishwar, A. Bellur, & H.A.
Murthy, ”Classification of melodic motifs in raga music with time-series
matching,” Journal of New Music Research, vol. 43, no. 1, pp. 115-131,
2014.
[120] K.K. Ganguli, & P. Rao, ”Discrimination of melodic patterns in indian
classical music,” in Communications (NCC), 2015 Twenty First National
Conference on, IEEE, 2015, pp. 1-6.
[121] R. Islam, M. Xu, & Y. Fan, ”Chinese Traditional Opera database
for Music Genre Recognition,” in Oriental COCOSDA held jointly with
2015 Conference on Asian Spoken Language Research and Evaluation
(O-COCOSDA/CASLRE), 2015 International Conference, IEEE, 2015,
pp. 38-41.
[122] J. Zhang, X. Huang, L. Yang, & Y. Xu, ”Some issues of mood
classification for Chinese popular music,” In Electrical and Computer
Engineering (CCECE), 2015 IEEE 28th Canadian Conference on, IEEE,
May 2015, pp. 1193-1198.
[123] P. Medentzidou, & C. Kotropoulos, ”Restoration of missing data in
Greek folk music by interpolation techniques,” Digital Media Industry &
Academic Forum (DMIAF), IEEE, 2016, pp. 107-112.
[124] D.S. Delviniotis, ”Acoustic Characteristics of Modern Greek Orthodox
Church Music,” Journal of Voice, vol. 27, no. 5, pp. 656-e1, 2013.
[125] B. Bozkurt, R. Ayangil, & A. Holzapfel ”Computational analysis of
turkish makam music: Review of state of the art and challenges,” Journal
of New Music Research, vol. 43, no. 1, pp. 3-23, 2014.
[126] M.A.K. Saun, & B. Bolat, ”Classification of classic Turkish music
makams by using deep belief networks,” In Innovations in Intelligent
SysTems and Applications (INISTA), 2016 International Symposium on,
IEEE, 2016, pp. 1-6.
[127] T. Bekirolu, N. Ovayolu, Y. Ergu¨n, & H.. Ekerbier, ”Effect of Turkish
classical music on blood pressure: a randomized controlled trial in
hypertensive elderly patients,” Complementary therapies in medicine, vol
21, no. 3, pp. 147-154, 2013.
[128] Y. Nan, T.R. Kno¨sche, & A.D. Friederici, ”The perception of musical
phrase structure: a cross-cultural ERP study,” Brain research, vol. 1094,
no. 1, pp. 179-191, 2006.
[129] M. Muthuraman, & R. Krishnakumar, ”Effect of the Raga Ananda
Bhairavi in Post Operative Pain Relief Management,” Indian Journal of
Surgery, vol. 76, no. 5, pp. 363-370, 2014.
[130] J.H. McDermott, A.F. Schultz, E.A. Undurraga & R.A. Godoy,
”Indifference to dissonance in native Amazonians reveals cultural
variation in music perception,” Nature, vol. 535 no. 7613, pp. 547-550,
2016.
[131] T. Fritz, S. Jentschke, N. Gosselin, D. Sammler, I. Peretz, R. Turner
& S. Koelsch, ”Universal recognition of three basic emotions in music,”
Current biology, vol 19, no. 7, pp. 573-576, 2009.
[132] K.J. Bonebreak, ”Technical tips: a sound way to induce relaxation
and natural sleep a safe alternative to sedation,” American Journal of
Electroneurodiagnostic Technology, vol. 36, no. 4, pp. 264-268, 1996.
[133] M. Solca´, A. Mottaz, & A.G. Guggisberg, ”Binaural beats increase
interhemispheric alpha-band coherence between auditory cortices,”
Hearing research, vol. 332, pp. 233-237, 2016.
[134] A. Crespo, M. Recuero, G. Galvez, & A. ”Begon˜a ”Effect of binaural
stimulation on attention and EEG,” Archives of Acoustics, vol. 38, no. 4,
pp. 517-528, 2013.
[135] D.L. Bowling, K. Gill, J.D. Choi, J. Prinz, & D. Purves, ”Major and
minor music compared to excited and subdued speech,” The Journal of
the Acoustical Society of America, vol. 127, no. 1, pp. 491-503, 2010.
Roberto Avila Va´zquez was born in Mexico City
on August 25th, 1987. He received the B.S. degree
on Biomedical Engineering from Tecnologico de
Monterrey on 2014. He is currently pursuing the
Ph. D. degree in the School of Engineering and
Science in the same university. His research
interests are music information retrieval, biomedical
signal processing and human music computer
interfaces with therapeutic applications.
Rogelio Bustamante Bello was born in Orizaba,
Veracruz on 1965; He received the B.S. degree
Electronic Instrumentation from Universidad
Veracruzana, Xalapa, Ver.; his master on Electronic
and Telecommunications from CICESE, Ensenada,
Baja California, Me´xico and his Ph. D. on
Telecommunications and Electronics from SEPI,
Instituto Politenico Nacional. His research interests
are wireless networks, biomedical wearables
devices, research on emotional states and human
computer interfaces with therapeutic applications.
 Arturo Molina was born in Oaxaca, Mexico on
February 1st, 1964. He received the B.S. degree and
master on Computational Systems from Tecnologico
de Monterrey, his Ph. D. on Mechanics from
Budapest University of Technology and Economy
and his Ph. D. degree on Manufacturing Systems
from Loughborough University, United Kingdom.
He is Vice-President of Research at the Tecnolo´gico
de Monterrey. He is member of the National
Systems of Researchers of Mexico on Level III. He
has published four books, 43 articles on
arbitrated
journals, 58 book chapters and 60 articles con arbitrated conferences. His
areas of investigation include concurrent engineering, information models for
design and manufacturing, technology modeling and integration in
manufacturing and technologies for collaborative engineering.

Sergio Navarro Tuch was born in Mexico City

on August 16, 1990. He received the B.S. degree
on Mechatronic Engineering from Tecnologico de
Monterrey on 2014. He is currently pursuing the
Ph. D. degree in the School of Engineering and
Science in the same university. His research
interests are domotics with implementation on
emotional analyzing systems with social and
therapeutic implementations.

Javier Izquierdo Reyes was born in Mexico State

on September 19th 1991. He received the B.S.
degree from Instituto Polite´cnico Nacional on 2013
He is currently pursuing the Ph. D. degree in the
School of Engineering and Science in Tecnolgico
de Monterrey. His research interests are intelligent
vehicles, machine learning algorithms and security
systems in vehicles.

Rafael Ferrer was born in San Luis Potos´ı, Me´xico

on September 11th, 1974. He received the B.A. in
Music from the University of las Ame´ricas-Puebla,
Me´xico, his M.A. in Music, Mind and Technology and a
Ph.D. in Musicology from the University of
Jyva¨skyla¨, Finland. He is a member of the National System
of Researchers of Mexico at candidate level. He is a lecturer
of music theory and appreciation for engineers in digital
music production, and member of Knowledge Society
research group at the School of Humanities and Education
at the Tecnolo´gico
de Monterrey, Mexico. He investigates perception and cognition of musical
timbre, musical preferences, and soundscape, from a transdiciplinary
perspective.