audio mostly

A CONFERENCE ON INTERACTION WITH SOUND

PROC EEDI NGS

Proceedings of the
Audio Mostly Conference
- A Conference on
Interaction with Sound

October 22 23, 2008

Piteå Sweden
Content
Contributors CV 1

Sound and Immersion in the First-Person Shooter: Mixed Measuring the Player's Sonic Experience. 9
Mark Grimshaw, Craig A.Lindley & Lennart Nacke

Sound and the diegesis in survival-horror games. Daniel Kromand 16

Psychologically Motivated Techniques for Emotional Sound in Computer Games. Inger Ekman 20

Interactive Sonification of Grid-based Games. Louise Valgerður Nickerson & Thomas Hermann 27

Using audio aids to augment games to be playable for blind people. David C. Moffat & David Carr 35

BeoWulf field test paper. Mats Liljedahl & Nigel Papworth 43

Control of Sound Environment using Genetic Algorithms. Scott Beveridge & Don Knox 50

Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities. Daniel Hug 56

Saturday Night or Fever? Context Aware Music Playlists. Stuart Cunningham, Stephen Caulder & Vic Grout 64

A Musical Instrument based on 3D Data and Volume Sonification Techniques. Lars Stockmann, 72
Axel Berndt & Niklas Röber

Same but different - Composing for Interactivity. Anders-Petter Andersson & Birgitta Cappelen 80

The Harmony Pad - A new creative tool for analyzing, generating and teaching tonal music. 86
Gabriel Gatzsche, Markus Mehnert, David Gatzsche & K. Brandenburg

Sonic interactions with hand clap sounds. Antti Jylhä & Cumhur Erkut 93

Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity. Ulrich Reiter 101

An Embedded Audio-Based Vehicle Classification Based on Two-level F-ratio. JiQing Han 108

dots: an Audio Entertainment Installation using Visual and Spatialbased Interaction. Andreas Floros, 112
Nikolaos Grigoriou, Nikolaos Moustakas & Nikolaos Kanellopoulos.

AllThatSounds: Associative Semantic Indexing of Audio Data. Hannes Raffaseder, Matthias Husinsky & 117
Julian Rubisch

IMPROVe: The Mobile Phone as a Medium fo Heightened Sonic Perception. Richard Widerberg & 121
Zeenath Hasan

Audio Interface as a Device for Physical Computing. Kazuhiro Jo 123

Automatic genre and artist classification by analyzing improved solo parts from musical recordings. Jakob 127
Abesser, Christian Dittmar & Holger Grossmann

The Heart as an Ocean: Exploring meaningful interaction with biofeedback. Pieter Coussement, 132
Marc Leman, Nuno Diniz & Michiel Demey

EarMarkit: An Audio-Only Game for Mobile Platforms. David Black, Kristian Gohlke & Jörn Loviscach 135
Contributors CV

Mark Grimshaw
Mark Grimshaw was educated in Kenya, South Africa, England and New Zealand where he
gained his PhD. He is currently the Reader in Creative Technologies at the University of
Bolton, England, and his most recent work 'The Acoustic Ecology of the First-Person Shooter'
was published by VDM in May 2008.

Daniel Kromand
Daniel Kromand holds a bachelors degree in Media Studies from University of Copenhagen
and is currently attending the IT University of Copenhagen for a master's degree in Media
Technology and Games. He has previously presented research in avatar theory at the DiGRA
2007 conference. Daniel Kromand also works as a project manager in Copenhagen.

Inger Ekman
Inger Ekman researches sound design for interactive media and virtual environments. She
received a MSc in Computer Science from University of Tampere in 2003 and is currently
working on her doctoral thesis on game sound design. She is particularly interested in non-
musical sound effects and ambience and the ways in which these influence player emotion.

Louise Nickerson
Louise Nickerson is a PhD candidate at Queen Mary, University of London (QMUL) in the
Department of Computer Science. She holds an MSc in Computer Science from QMUL
(2002) and a BA in French and Italian Literature from the University of Virginia (1998). She
is part of the Interaction, Media and Communication research group which focuses on human-
human interaction with a smattering of people working on audio. Her interest stems from
accessibility and the belief that soon mobile computing will be the norm. Her research focuses
on the development and definition of auditory overviews to make auditory interfaces more
approachable.

When not being a PhD student, Louise can be found nurturing her interest in foreign
languages or on the Thames with her crew-mates from the Sons of the Thames Rowing Club.

Thomas Hermann
Dr. Thomas Hermann studied physics at Bielefeld University. From 1998 to 2001 he was a
member of the interdisciplinary Graduate Program "Task-oriented Communication". He
started the research on sonification and auditory display in the Neuroinformatics Group and
received a Ph.D. in Computer Science in 2002 from Bielefeld University (thesis: Sonification
for Exploratory Data Analysis). After research stays at the Bell Labs (NJ, USA, 2000) and
GIST (Glasgow University, UK, 2004), he is currently assistant professor (German C1
position) in the Neuroinformatics Group where he coordinates and conducts research on
sonification, human-computer interaction and cognitive interaction technology.

In his research, Thomas Hermann is developing techniques for interactive multimodal data
representation and exploratory analysis of high-dimensional data with a particular focus on
sonification, novel interactive data mining techniques and human-computer interaction. His

1
research topics include furthermore Tangible Computing, Ambient Information Systems,
Gestural Interactions and Augmented Reality.

David Moffat
David Moffat is a Lecturer in Computing at Glasgow Caledonian University.
His research interests are mainly in emotion and affective computing, often as applied to
video games, but he also has wider interests in other fields of AI and cognitive science.

David Carr
David Carr is a Masters student in Advanced Computing at Glasgow Caledonian university.
He is one of the first appointed "scholars" in the School of Engineering and Computing, for
his particular interests in AI programming for games.

Mats Liljedahl
Since the mid 80’s, Mats Liljedahl has been involved with questions concerning ICT as a tool
for learning and artistic expression. Between 1988 and 1996, Mats worked as a music teacher
and took part in projects dealing with integrating ICT as an active tool for music teachers.
Between 1996 and 2000 he worked as a teachers educator at Ingesund University College of
Music in Arvika, where he also had the opportunity to work with several development
projects on ICT, music and learning. Since August 2000, Mats works at the Interactive
Institute and has taken part in several projects dealing with music, learning and ICT.

Nigel Papworth
Nigel Papworth trained at the London College of Printing in Graphic Design after some years
working in advertising in London, a career he started at
17 before completing his education. After moving to Sweden in 1985, he co-founded of one
of Sweden’s first major games companies, Daydream, where he worked as lead game
designer for 9 years. Two of his designs ‘Safecracker’ and ‘Traitors Gate’ enjoyed
considerable success, especially in the USA. He has worked and lectured on the integration of
Ai, dialogue, game states and behavioral simulation systems utilizing chaos theory in
computer games. After joining the Interactive Institute, he has concentrated on the role audio
can play in driving gameplay and conveying game content and status to the player.
Nigel is married and has three children aged 21 to 14.

Scott Beveridge
Scott Beveridge B.Sc. (Hons.), PhD, Audio Technology, B.Sc. (1st Class Honours), Audio
Technology
Publications
D. Knox, G. Cassidy, S. Beveridge, R MacDonald. Music Emotion Classification by Audio
Signal Analysis: Analysis of Self-selected Music During Gameplay. 10th ICPMC, Sapporo,
Japan, August 25th – 29th, 2008
D. Knox, S. Beveridge, R. MacDonald. Emotion Classification in Contemporary Music.
DMRN+2: Digital Music Research Network One-day Workshop 2007, Queen Mary,
University of London, December 2007.

2
Research Interests
Algorithmic composition and music generation using non-deterministic source data. In
particular the sonification of socio-spatial behaviour in large sensate environments. Current
research investigates the intersection between Human Computer Interaction and Music
Information Retrieval in a framework which uses models of emotion to generate socially
reflexive audio material.

Daniel Hug
Daniel Hug has a background in music, sound design, interaction design and project
management in applied research. Since 1999 he investigates sound and interaction design
related questions through installations, design works and theoretical publications. Since 2005,
he is teaching sound studies and sound design for interactive media and games at the
Interaction Design department of the Zurich University of the Arts, Switzerland. Hug pursues
a PhD on sound design for interactive commodities at the University of the Arts and Industrial
Design of Linz, Austria, in close exchange with the European COST-initiative "Sonic
Interaction Design".

Stuart Cunningham
Stuart Cunningham was awarded the BSc degree in Computer Networks in 2001, and in 2003
was awarded the MSc Multimedia Communications degree with Distinction, both from the
University of Paisley (UK). He is a Chartered IT Professional (CITP), member of the British
Computer Society (BCS), the Institution of Engineering & Technology (IET) and the Institute
of Electrical and Electronics Engineers (IEEE). Stuart was also a member of the MPEG Music
Notation Standards (MPEG-SMR) working group.
Stuart is currently a Senior Lecturer in Computing and a PhD student at Glyndŵr University
in the UK, studying under the supervision of Professor Vic Grout. His research interests
include: measurement of audio similarity, audio compression, image sonification, and musical
content & context analysis.

Lars Stockmann
Lars Stockmann has just received his Diplom (Master's Thesis) in Computational Visualistics
from the Otto-von-Guericke University of Magdeburg. His current research interests are
interactive acoustic environments. This includes interactive sonification techniques as well as
computer-based instruments for live performances and computer games.
Previous studies include API-design for audio applications on mobile devices, stereo vision
and rendering (raytracing and real-time cg). He is currently working as developer and
programmer at a company in the course of formation.

Anders-Petter Andersson
Anders-Petter Andersson is a musician, composer and doctoral researcher in Musicology at
Göteborg University / Malmö University / Interactive Institute. The working title of his Ph D
project is "Interactive Music Composition". He tries to answer the question of how one can
compose musically satisfying sound and music for games and interactive applications. In
interactive music, the listening role is complex, as the listener participates and alters the
composition. With knowledge from musical traditions such as improvisation he develops
composition methods for audio-tactile and physical environments such as Strainings, Do-Be-
DJ and Mufi.

3
Anders-Petter is co-ordinator of a new education and BA programme in Interactive Sound
Design at Kristianstad University, combining music and the computer within mobile services,
game and music industry. Join the Interactive Sound Design community at the website:
www.interactivesound.org

Birgitta Cappelen
Birgitta Cappelen is an industrial designer (SID), interaction designer and associate professor
at Oslo school of Architecture and Design (AHO). She is also working on her Ph D at Malmö
University (K3). The working title of her Ph D project is "Co-create and Re-create -
rethinking Industrial Design in the digital age." In her work she tries to answer the question of
what meaningful design can be in our time after postmodernism and with the computer as a
material requirement. Instead of designing beautiful and user-friendly objects, she suggests
designing fields of possibilities with a high degree of inscription and potential of circulation.
She calls this design quality "multivalence".
www.musicalfieldsforever.com

Gabriel Gatzsche
Gabriel Gatzsche studied Media Technology at the Technische Universität Ilmenau and
received his Diploma in 2003. After that he joined to the Fraunhofer IDMT Ilmenau where he
works on the MPEG-4 based storage and transmission of object oriented WFS audio scenes,
the software development of spatial audio reproduction systems for large venues and the
development and standardization of data formats for the Digital Cinema. Since 2005 he works
on a doctoral thesis which deals with the analysis of synthesis of musical audio signals.
Within that relationship he developed the HarmonyPad which is presented at the
AudioMostly2008.

Antti Jylhä
Antti Jylhä was born in Helsinki, Finland, in 1981. He received the M.Sc. degree in
telecommunications from Helsinki University of Technology (TKK), Espoo, Finland, in 2007.
He is currently working at the Department of Signal Processing and Acoustics at the same
institute as a researcher and pursuing a doctoral degree in acoustics and audio signal
processing. His current research interests include auditory and multi-modal interfaces in
human-computer interaction, and modeling and analysis of multiple interacting sound
sources. He is also instructing student projects related to automatic sports monitoring, and is
involved in the activities of the Helsinki Mobile Phone Orchestra.

Cumhur Erkut
Cumhur Erkut was born in Istanbul, Turkey, in 1969. He received B.Sc.
and the M.Sc. degrees in electronics and communication engineering from the Yildiz
Technical University, Istanbul, Turkey, in 1994 and 1997, respectively, and the Dr.Sc.(Tech.)
degree in electrical engineering from the Helsinki University of Technology (TKK), Espoo,
Finland, in 2002. Between 1998 and 2002, he worked as a researcher, and between 2002 and
2007 as a postdoctoral researcher at the Laboratory of Acoustics and Audio Signal Processing
of the TKK, and contributed to various national and international research projects. 2007
onwards, he is appointed as an Academy Research Fellow, he conducts his research project
Schema-SID [Academy of Finland, 120583] and contributes to the COST IC0601 Action
"Sonic Interaction Design" (SID). His primary research interests are sonic interaction design,
and physics-based sound synthesis and control.

4
Ulrich Reiter
Ulrich Reiter studied Electrical Engineering at RWTH Aachen, from where he received a
Diplom-Ingenieur degree (equ. M.Sc.) in 1999. He has been working at the Institute of Media
Technology (director Prof. Dr. Karlheinz Brandenburg) at Technische Universitaet Ilmenau as
a researcher from 1999-2007, where he was also giving lectures on Virtual and Applied
Acoustics and on Recording Studio Technology. He did his Ph.D. on perceived quality in
interactive audiovisual application systems of moderate complexity. Since 2008 he is working
with Prof. Peter Svensson at the Centre for Quantifiable Quality of Service in Communication
Systems (Q2S), a Norwegian Centre of Excellence, at the Norwegian University of
Technology in Trondheim. His current research interests include multimodal perception,
virtual acoustics and interactive audiovisual application systems using the MPEG-4 standard.

JiQing Han
JiQing Han, received the Master degree and Doctor degree both from Harbin Institute of
Technology, China in 1990 and 1998, respectively. He is a Professor in School of Computer
Science and Technology, Harbin Institute of Technology, China. His research interests
include Speech recognition and synthesis, Audio signal processing, Pattern recognition.

Andreas Floros
Andreas Floros was born in Drama, Greece in 1973. In 1996 he received
his engineering degree from the department of electrical and computer
engineering, University of Patras, and in 2001 his Ph.D. degree from the
same department. His research was mainly focused on digital audio
signal processing and conversion techniques for all-digital power
amplification methods. He was also involved in research in the area of
acoustics. In 2001, he joined ATMEL Multimedia and Communications,
working in projects related with digital audio delivery over PANs and WLANs, Quality-of-
Service, mesh networking, wireless VoIP technologies and lately with audio encoding and
compression implementations in embedded processors. Since 2005, he is a visiting assistant
professor at the Department of Audio Visual Arts, Ionian University. Dr. Floros is a member
of the Audio Engineering Society, the Hellenic Institute of Acoustics and the Technical
Chamber of Greece.

Nikos Moustakas
Nikos Moustakas was born in Athens, Greece. He graduated from high
school in 2004 and till now is an undergraduate student at the Ionian
university in the department of Audio-Visual Arts in Corfu. He
participated in 3 photo exhibitions that his department organized as well
as Audio-Video festivals in the last two years. Nikos has attended the
“Miden Festival” in Kalamata where he presented his own work (some
videos). In addition, he has some experience on the conservation and
restoration of old photos. Also he is an operator in the computer
laboratory of his Department. His main focus is on audio-video interactive installations.

5
 Nikolas Grigoriou
Nikolas Grigoriou was born in Heraklion, Crete, Greece. He graduated
from high school in 2004 and till now is an undergraduate student at
the Ionian university in the department of Audio-Visual Arts in Corfu.
He participated in 3 photo exhibitions that his department organized as
well as Audio-Video festivals in the last two years. Nikolas has attended
the “Miden Festival” in Kalamata where he presented his own work
(some videos). In addition, he has some working experience in terms
of practicum in a TV Studio in his hometown and this year he won the
first prize at a student oriented conference “Student Eureka 2008”. His
main focus is on audio-video interactive installations, surround effects,
and sound in space.

Matthias Husinsky
Matthias Husinsky was born in 1982. He studied Media Engineering and Media Design at the
University of Applied Sciences Hagenberg, Austria from 2000 to 2004 and received his
Masters degree with a work on a guitar tuner for mobile phones. From 2005 till now he
worked as a project assistant in several research projects at the University of Applied Sciences
St.Pölten, Austria, mainly focusing on modern audio technologies in a networked society. He
is now also a PhD candidate at the Johannes Kepler-University in Linz, Austria, working in
the field of MIR (music information retrieval).

Julian Rubisch
Julian Rubisch was born in 1981. He studied Telecommunications and Media at the
University of Applied Sciences St. Pölten, Austria from 2004 to 2007 and received his
Bachelor of Science degree for a work on psychoacoustical models within audio information
retrieval. He is currently employed as research assistant at the University of Applied Sciences,
St. Pölten, where he is also working on his Masters thesis on generative music for media
applications.

Richard Widerberg
Richard Widerberg is a sound artist and new media designer living in Göteborg, Sweden. He
has a background working as a new media designer, organizing events, making radio, doing
sonic works, and playing music.
His main focus during the last years has been to investigate the many dimensions of sound
and listening. Also related to his sonic works are works that deal with location, mobility,
interaction and social exchange. He has a strong interest in new forms of copyright and
distribution of both music and sound as well as open-source development. He is also an active
musician.
http://www.riwid.net
rwiderberg@gmail.com

Zeenath Hasan
Zeenath Hasan involves herself in the people centered practice of design research to exercise
the potential of media technologies for socially appropriate intervention. The studio from
where she conducts her doctoral research on the role of media in a democracy is located in the
School of Arts and Communication, Malmö University. She was born in Kolkata and

6
currently resides in Malmö. Her stint as new media practitioner was sparked after her training
in the MS Communications programme at Manipal. Her footprints crossed over to Finland
during her training in MA New Media Studies from the University of Art and Design
Helsinki. She has worn the labels of Information Architect, Interface Designer, Design
Researcher, Cultural Producer, and Media Artist and Researcher. She also runs a 1-person
design research firm.
http://www.zeeniac.net
zeenath.hasan@gmail.com

Kazuhiro Jo
Kazuhiro Jo <http://jo.swo.jp/> is a Research Fellow at RCAST, University of Tokyo and
Visiting Research Fellow at Digital Media group, Culture Lab, Newcastle University. He is
also a member of The SINE WAVE ORCHESTRA <http://swo.jp/>, a member of the
Monalisa project <http://monalisa-au.org/plog/>, a member of AEO, and a co-organizer of
dorkbot Tokyo <http://dorkbot.org/dorkbottokyo/>.

Jakob Abesser
Dipl.-Ing. Jakob Abesser studied computer engineering with specialization in
telecommunication and measurement engineering at the Ilmenau University of Technology
from 2002 to 2008. He investigated the characterization of instrumental solo parts in music
pieces by means of musical high-level features as well as their application for genre and artist
classification within his diploma thesis. After his graduation he joined the Fraunhofer IDMT
and is now working in the Metadata department as a Ph.D. student. His research interests are
the automatic transcription of stringed instruments like bass and guitar as well as performance
analysis and artist classification.

Christian Dittmar
Dipl.-Ing. (FH) Christian Dittmar studied electrical engineering with specialization in digital
media technology at the University for Applied Sciences in Jena from 1998 to 2002. In his
diploma thesis, he investigated Independent Subspace Analysis as a means of audio signal
analysis. Subsequent to his successful graduation he joined IDMT in 2003 to work at the
Metadata department. He has contributed to a number of scientific papers in the field of music
information retrieval and automatic transcription. In 2005 he participated in the MIREX
evaluation category automatic drum detection. Since late 2006 he is Semantic Metadata
Systems group manager at Fraunhofer IDMT.

Holger Grossman
Dipl.-Ing. Holger Grossmann studied electrical engineering at the Ilmenau University of
Technology from 1986 until 1993. Afterwards he worked as software developer in the fields
of electronic musical instruments as well as client-/server e-business systems. In 2001 he
joined the Fraunhofer-Arbeitsgruppe für Elektronische Medientechnologie AEMT, Ilmenau
where he worked as engineer and researcher in the Metadata department. During the
following 3 years his research focus was the development and realisation of the music
identification system AudioID which was standardized in MPEG-7 as the
AudioSignatureType Description Schema. The Fraunhofer-Arbeitsgruppe became an
independent institute in 2004. Since then Holger Grossmann is head of the Metadata
department at IDMT, focussed on research in the fields of automated semantic media
annotation and multimedia search. He is co-author of scientific papers and was invited as

7
speaker on international conferences. In 2007 he chaired the AudioMostly conference in
Ilmenau.

Pieter Coussement
Pieter Coussement is a new media artist, performer and composer based out of Ghent,
Belgium. His main focus is on the function and position of the human body within interactive
art with sound as the main mediator. After several years of teaching new media, he started his
PhD at the Institute for Psychoacoustics and Electronic Music –IPEM at the university of
Ghent, where he furthers his artistic research on interactive art.
Education:
PhD 2008 Institute for Psychoacoustics and Electronic Music –
IPEM, University Ghent, Belgium
drs. Audiovisual arts, (re)experiencing interactive art
MA 2003 Royal Academie of Fine Arts, Ghent, Belgium
Master of Fine Arts, Mixed Media
BA 1999 Royal Academie of Fine Arts, Ghent, Belgium
Three Dimensional Art, Multi Media
Professional Experience:
2007 PHL department of visual arts, Hasselt, Belgium
Workshop teacher Interaction with audiovisual media
2007 Howest PIH, Kortrijk, Belgium
Workshop teacher Creative Coding: Realtime user responsive 3D Visuals in
openGL
2004-2008 City Academy of Arts, Ostend, Belgium
Teacher Digital Visual Design
2003-2008 City Academy of Arts, Ostend, Belgium
Teacher Experiment Digital Design

David Black
David Black is a master student at the Digital Media program of the Hochschule Bremen. As
a graduate of the University of Southern California School of Music and the Royal
Conservatory of Den Haag's Institute of Sonology, he is involved with interactive music
pieces for electronics, dance, percussion, electroacoustic composition, guided walking audio
tours, and auditory display.

Kristian Gohlke
Kristian Gohlke is currently a master student at the Hochschule Bremen. His interests circle
around physical-computing, electronics and programming as well as human-computer
interaction. He also works as a student tutor at Bremen University and the University for the
Arts in Bremen where he teaches physical computing to children and artists, and does and
consulting on related art and design projects. He lives at: http://krx.at/

Jörn Loviscach
Joern Loviscach is a professor at Hochschule Bremen. His major interests lie in computer
graphics, human-computer interaction, and audio and music computing. He is a regular
contributor to conferences such as SIGGRAPH, Eurographics and the AES Convention. In
addition, he has published numerous chapters in book series such as Game Programming
Gems and ShaderX Programming. He lives at: http://www.l7h.cn/

8
 Sound and Immersion in the First-Person Shooter: Mixed Measurement
of the Player's Sonic Experience

Mark Grimshaw (mark@wikindx.com)

Craig A. Lindley (craig.lindley@bth.se)
Lennart Nacke (Lennart.Nacke@bth.se)

Abstract. Player immersion is the holy grail of computer game designers particularly in environments such as those found in first-
person shooters. However, little is understood about the processes of immersion and much is assumed. This is certainly the case
with sound and its immersive potential. Some theoretical work explores this sonic relationship but little experimental data exist to
either confirm or invalidate existing theories and assumptions.

This paper summarizes and reports on the results of a preliminary psychophysiological experiment to measure human arousal and
valence in the context of sound and immersion in first-person shooter computer games. It is conducted in the context of a larger set of
psychophysiological investigations assessing the nature of the player experience and is the first in a series of systematic experiments
investigating the player's relationship to sound in the genre. In addition to answering questionnaires, participants were required to
play a bespoke Half-Life 2 level whilst being measured with electroencephalography, electrocardiography, electromyography,
galvanic skin response and eye tracking equipment. We hypothesize that subjective responses correlated with objective
measurements provide a more accurate assessment of the player's physical arousal and emotional valence and that changes in these
factors may be mapped to subjective states of immersion in first-person shooter computer games.

Introduction suggests that immersion is a participatory activity [9] and

An increasing amount of research in games studies and games
technology deals with presence in virtual environments and, of
more interest for the purposes of our research, player immersion
in digital game worlds. Player immersion may be said to be the
holy grail of digital game design particularly in the types of
game environment found in the First-Person Shooter (FPS)
genre. This type of game is typically exemplified by the run ‘n’
gun sub-genre (games such as the Doom, [1] Quake [2] and
Half-Life [3] series – even though the latter has an unusually
strong focus on narrative) which is visually characterized by a
hand or pair of hands holding a weapon on screen and
conceptually characterized as ‘the hunter and the hunted’. The
intention is that the player identifies with the game character
whose hands, the player’s virtual prostheses, are seen receding
into the game environment. [4] This identification with the
character, and the use of hands only, provides a first-person
perspective with which it is proposed that, visually, player
immersion in the game world derives from the player
‘becoming’ the game character, in the sense of the player having
the experience of acting ‘within’ the game world. This sense of
immersion is strengthened further through the player’s actions
having a non-trivial effect on the environment and game play.
[5] For example, operating the game interface (the computer
mouse or keyboard, for instance) may cause the image on the
screen to change: the weapon may recoil and flash or an on-
screen animation indicates the weapon reloading. The player
perceives movement through the 3-dimensional world of the
game because visual artifacts rotate, magnify and diminish or
appear and disappear on the screen.
There are a variety of definitions of immersion in computer
games. Kearney and Pivec claim that immersion provides the
motivation or flow required for the player to be repeatedly
engaged with the game [6] while Ermi and Mäyrä, paraphrasing
Pine and Gilmore, [7] state that “immersion means becoming
physically or virtually a part of the experience itself”. They also
distinguish between different forms of immersion: sensory,
imaginative and challenge-based immersion. [8] Murray
McMahan provides three conditions for immersion: “[T]he
user’s expectations of the game or environment must match the
environment’s conventions fairly closely […] the user’s actions
must have a non-trivial impact on the environment [and] the
conventions of the world must be consistent”. [5] For
McMahan, two factors influencing immersion are the level of
social realism and the level of perceptual realism. Garcia claims
that “[i]n the most immersing environments reminders of the
structural level of the game are gone” [10] while Carr provides
two categories of immersion: perceptual (when the participant’s
senses are monopolized by the experience) and psychological
(an imaginative or mental absorption through which the
participant becomes engrossed in the experience). [11]
In the context of this paper, other work mentions or focuses
upon the role of sound in facilitating player immersion in the
game world. Laurel makes the case that the “[t]ight linkage
between visual, kinaesthetic, and auditory modalities” is key to
the sense of immersion. [12] Jørgensen believes that players are
immersed in an auditory world through the use of realistic audio
samples [13] while Murphy and Pitt make similar claims for
spatial sound. [14] Autopoiesis and acoustic ecologies have
been used to model player immersion through sound in FPS
games [4] and Grimshaw and Schott provide a range of
conceptual tools with which to analyze the immersive functions
of game sound. [15] Some authors make a distinction between
modes of immersion, particularly where immersion is enabled
through the spatial qualities of the sound: Stockburger implies
that the player is physically immersed in the game sound [16]
and this is amplified and explicitly stated by Grimshaw. [17]
This physical sonic immersion has also been observed for film
audiences and the concept transferred to the design of sound for
FPS games and simulators. [18]
Most of the work cited above is theoretical and, where authors
describe the immersive potential of sound in computer games,
there is the assumption that sounds, more sounds, realistic
sounds, spatial sounds all inexorably and incontrovertibly equate
to greater player immersion. This may well be the case but the
assumption lacks thorough evidence to support the various

-1-
9
concepts of immersion outlined above. Attempts to provide
evidence include Jørgensen, who uses player surveys, [13] and
Shilling, Zyda and Wardynski. [18] The latter paper is of
particular interest to this work because it not only explores the
use of sound in an FPS game/simulation (America’s Army [19])
but it also attempts to objectively measure the player’s
emotional arousal through the use of temperature, electrodermal
response and heart-rate measurements. However, although the
authors state that “emotional arousal has a positive impact on
[the] sense of immersion in virtual environments” and that the
precise conjunction of a sound and an action seen on the screen
is “crucial for immersing the player”, the paper is a description
of their attempts to introduce, and amplify, emotion within the
game environment through sound rather than an attempt to
effect and measure immersion. The link between emotional
arousal and immersion is assumed and so the relationship
between sound and player immersion remains undefined in
objective terms.
Emotions are a central part of the game experience, motivating
the conscious cognitive judgments and decisions made during
gameplay. Psychophysiological investigations suggest that at
least some emotional states may be quantitatively characterized
via physiological measurements. Specific types of measurement
of different physiological responses (such as GSR, EMG, ECG
and EEG, as described below) are not by themselves reliable
indicators of well-characterized feelings; [20, 21] a de rigueur
cross-correlation of all measurements is crucial to identify the
emotional meaning of different patterns in the responses.
Moreover, the often described many-to-one relation between
psychological processing and physiological response [20] allows
for psychophysiological measures to be linked to a number of
psychological structures (for example, attention, emotion,
information processing). Using a response profile for a set of
physiological variables enables researchers to go into more
detail with their analysis and allows a better correlation of
response profile and psychological event. [21] The crucial issue
here is the correlation of patterns of measurement characteristics
for a set of different measures with subjective characterizations
of experience such as emotion and feelings (for example, the
feeling of immersion in gameplay).
Facial electromyography (EMG) is a direct measure of electrical
activity involved in facial muscle contractions; EMG provides
information on emotional expression via facial muscle activation
(even though a facial expression may not be visually observable)
and can be considered as a useful external measure for hedonic
valence (that is, degree of pleasure/displeasure). [22] Positive
emotions are indexed by high activity at the zygomaticus major
(cheek muscle) and orbicularis oculi (periocular muscle)
regions. In contrast to this, negative emotions are associated
with high activity at the corrugator supercilii (brow muscle)
regions. This makes facial EMG suitable for mapping emotions
to the valence dimension in the two-dimensional space
described in Lang’s dimensional theory of emotion. [22] This
dimension reflects the degree of pleasantness of an affective
experience. The other dimension, the arousal dimension, depicts
the activation level linked to an emotionally affective experience
ranging from calmness to extreme excitement. In this kind of
dimensional theory of emotion, emotional categories found in
everyday language (for example, happiness, joy, depression,
anger) are interpreted as correlating with different ratios of
valence and arousal, hence being mappable within a space
defined by orthogonal axes representing degrees of valence and
arousal, respectively. For example, depression may be
represented by low valence and low arousal, while joy may be
represented by high valence and high arousal.
-2-
Arousal is commonly measured using galvanic skin response
(GSR), also known as skin conductance. [23] The conductance
of the skin is directly related to the production of sweat in the
eccrine sweat glands, which is entirely controlled by the human
sympathetic nervous system. Increased sweat gland activity is
thus directly related to electrical skin conductance. Hence,
measuring both GSR and EMG provides sufficient data to
provide an interpretation of the emotional state of a game player
in real time, according to a phasic emotional model.
This paper describes and analyzes the results of a preliminary
experiment that investigates the role of sound in enabling player
immersion in the FPS game Half-Life 2. [24] The investigation
is designed to provide both subjective responses (through the
use of questionnaires) and objective measurements (through the
use of electromyography (EMG), galvanic skin response (GSR),
electroencephalography (EEG), electrocardiography (ECG) and
eye tracking equipment). The overall aim of the experiment is
to find external (that is, objective) measures that may be reliably
correlated with subjective experiences assessed via
questionnaires in order to provide more detailed descriptions of
the emotional experience of game players during gameplay, both
in the degree of emotions experienced and in the timescale of
emotional changes and modulations. It is further hoped that this
method may lead to real-time measures for states of immersion
of players playing first-person shooter computer games. Finally,
correlating discriminations within psychophysiological data with
different categories of immersion can provide at least one
method for validating those categorizations. The experiment is
preliminary since the psychophysiological characterization of
states of immersion is not yet well developed.
The study further aims to provide a psychophysiologically-based
answer to the assumption that sound plays a role in enabling the
immersion of the player in the FPS game world. If the results of
the experiment provide a positive answer, that sound does
indeed play this role, it is envisaged that future experiments,
using a similar methodology, will be designed to investigate
more specific questions about the relationship between the
player and sound in FPS games.
The experiment was conducted in May 2008 in the Game and
Media Arts Laboratory at Blekinge Institute of Technology
(BTH) in Sweden. The investigation of sound formed part of a
larger psychophysiological investigation into the nature of the
player experience in computer games. This paper is also limited
to the analysis of GSR, EMG and questionnaire data. Further
analysis taking into account the other data types is ongoing.
Method
Subjects played a Half-Life 2 game mod especially designed for
a short immersive playing time of maximum 10 minutes. The
game mod was played four times with different sound
modalities and physiological responses were measured together
with questionnaires (assessing subjective responses) for each
modality.
2.1 Design
The game sessions were played under four different conditions,
corresponding to the permutations of the independent variable
(sound modality): playing with diegetic game sounds (normal
sounds), playing with speakers completely turned off (no
sounds, no music), playing with diegetic game sounds and an
additional music loop (sounds and music), and playing with
diegetic game sounds turned off and hearing only the music loop
(only music). Participants played under each condition in a
10
shifting order to eliminate repeated-measures effects (using a
Latin Squares design). Physiological responses (as indicators of
valence and arousal) were recorded for each session as well as
questionnaire answers. Questionnaire item order was
randomized for each participant using the open-source software
LimeSurvey. [25]
2.2 Participants
Data were recorded from 36 students and employees, recruited
from the three BTH University campuses and their age ranged
between 18 and 41 (M = 24, SD = 4.89). 19.4% of all
participants were female. When asked how frequently they play
digital games, 50% answered that they play games every day,
22.2% play weekly, 22.2% play occasionally and only 5.6%
play rarely or never. However, it should be noted that 62.1% of
all the males play on a daily basis and 20.7% play weekly. In
contrast to that, most of the females enjoyed playing only on an
occasional (57.1% of all females) or weekly (28.6%) basis.
Out of all participants, 47.2 % considered themselves casual
gamers, 38.9% said that they belong to the hardcore gamer
demographic and 13.9% could not identify themselves with any
of those. Nevertheless, no female participant considered herself
to be a hardcore gamer and 71.4% of all females said they were
casual gamers. Male gamers were more evenly distributed
among hardcore (48.3%) and casual (41.4%) gamers: the larger
percentage of males considering themselves hardcore players.
91.7% of the participants were right-handed and 50% were
wearing glasses or contact lenses. 94.4% believed they had full
hearing capacity (5.6% stated explicitly that they lack full
hearing capacity). 69.4% had a preference for playing with a
music track on. 44.4% preferred playing with surround sound
speakers, while 33.3% opted for playing with stereo
headphones. 11.1% liked playing with stereo speakers and the
final 11.1% preferred surround sound headphones. 33.3%
played an instrument. 13.8% played the piano or keyboard and
8.3% played the guitar. 41.7% saw themselves as hobby
musicians – some people worked with sound recording and
programming but did not play instruments.
66.7% of participants were enrolled as University students.
16.7% already had a Bachelor’s degree and 13.9% had a
Master’s degree. 61.1% of the participants had already played
the digital game Half-Life 2 before, 30.6% played it between 10
and 40 hours and 58.3% played it on a PC, leaving only one
participant who played it on an Xbox 360.
To estimate preconceptions of sound immersion, participants
were also asked how important they considered sounds, in
general, for first-person shooters (FPS). The results were rated
on a 5-point scale ranging from 1 (not important) to 5 (very
important). 55.6% claimed that sound was very important and
36.1% said it to be important. The term “immersive”, which was
also part of the questionnaire items assessing sound immersion,
was explained to participants beforehand as “the feeling of being
encapsulated inside the game world and not feeling in front of a
monitor anymore”. This was so phrased for reasons of lay
intelligibility and deemed to be a synthesis of previous
definitions of game immersion noted above, particularly those of
Ermi and Mäyrä, Garcia and Carr. This is suitable for
investigating whether immersion in a very general sense may be
distinguishable in psychophysiological measurement features; if
so, ongoing experiments may address the psychophysiological
detection of finer distinctions within the broad category of
immersion.
-3-
2.3 Apparatus
Facial EMG. We recorded the activity from left orbicularis
oculi, corrugator supercilii, and zygomaticus major muscle
regions, as recommended by Fridlund and Cacioppo, [26] using
BioSemi flat-type active electrodes (11mm width, 17mm length,
4.5mm height) electrodes with sintered Ag-AgCl (silver/silver
chloride) electrode pellets having a contact area 4 mm in
diameter. The electrodes were filled with low impedance highly
conductive Signa electrode gel (Parker Laboratories, Inc.). The
raw EMG signal was recorded with the ActiveTwo AD-box at a
sample rate of 2 kHz and using ActiView acquisition software.
Galvanic skin response (GSR). The impedance of the skin was
measured using two passive Ag-AgCl (silver/silver chloride)
Nihon Kohden electrodes (1 microamp, 512 Hz). The electrode
pellets were filled with TD-246 skin conductance electrode
paste (Med. Assoc. Inc.) and attached to the thenar and
hypothenar eminences of the participant’s left hand.
Video recording. A Sony DCR-SR72E video camera
(handycam) PAL was put on a tripod and positioned
approximately 50 cm behind and slightly over the right shoulder
of the player for observation of player movement and in-game
activity. In addition, the video recordings served as a validation
tool when psychophysiological data were visually inspected for
artifacts and recording errors.
Game experience survey. Different components of game
experience were measured using the game experience
questionnaire (GEQ). [27] As shown in a previous study by
Nacke and Lindley, [28] the GEQ components can assess
experiential constructs of immersion, tension, competence, flow,
negative affect, positive affect and challenge with apparently
good reliability.
Sound immersion. Subjective player experience of sound
immersion was measured using our own additional
questionnaire items rated on a 5-point scale ranging from 1 (for
example, not immersive) to 5 (for example, extremely
immersive) for sessions where sound was audible. Specific
sound questions included the following:
• How important are sounds in general for you in FPS
games?
• Diegetic Sounds:
How immersive were the following?
• Background sounds
• Sounds of opponents
• Sounds that you produced yourself (player-
produced sounds)
How important was the sound for you in the level you
just played?
• No Sound, No Music:
How much did it bother you to play without sound?
• Nondiegetic Music Only:
Did you miss the sound effects in this level? (Yes/No)
11
 all modalities, participants were thanked for their participation
and paid a small participation fee before they were escorted out
of the lab.

2.5 Reduction and Analysis of Data

Recorded psychophysiological data were visually inspected
using BESA (MEGIS Software GmbH, Germany) and EMG
data were also filtered using a Low Cutoff Filter (30 Hz, Type:
forward, Slope: 6 dB/oct) and a High Cutoff Filter (400 Hz,
Type: zero phase, Slope: 48dB/oct). If data remained noisy, they
were excluded from further analysis. EMG data were rectified
and exported together with GSR data at a sampling interval of
0.49 ms to SPSS for further analysis.

Mean values for physiological responses were calculated for

epochs of complete session times (varying between five and 10
Figure 1: The eye-tracking screen, EMG sensors and minutes). Psychophysiological data were corrected for errors
electroencephalography array using log and ln transformations. After histogram inspection,
data was assumed to be close to a normal distribution (without
Other apparatus used, but not included in this analysis (the data elimination of single outliers). Means were calculated for items
will form the subject of a future paper), were a Biosemi 32- of each of the seven GEQ questionnaire components
channel EEG system and a Tobii 1750 eye tracker (cf. Figure 1). (immersion, tension, competence, flow, negative affect, positive
affect, and challenge).
2.4 Procedure
We conducted all experiments on weekdays in the time from To test statistical significance of the results, one-way repeated-
10:00 a.m. to 6:00 p.m. with each experimental session lasting measures ANOVAs were conducted in SPSS using sound
approximately two hours. The experiments were advertized modality as the within-subject factor for each of the three EMG
especially to graduate and undergraduate students. All measures (orbicularis oculi, corrugator supercilii, zygomaticus
participants were invited to the newly established Game and major), the galvanic skin response (GSR) and all GEQ
Media Arts Laboratory at Blekinge Institute of Technology, components.
Sweden. After a brief description of the experimental procedure,
each participant filled in two forms. The first one was a Results
compulsory “informed consent” form (with a request not to take
part in the experiment when suffering from epileptic seizures or For each sound modality, the participants were asked a few
game addiction). The next one was an optional photographic assessment questions. This included 5-point scale ratings for the
release form. Each participant had to complete an initial immersiveness of sounds (5 = most immersive) after they played
demographic and psychographic assessment questionnaire prior the level with diegetic game sounds only. The background
to the experiment, which was immediately checked for sounds were rated just below the median value (M = 2.97, SD =
completeness. Participants were then seated in a comfortable 1.03) and opponent-produced sounds were rated higher (M =
office chair, which was adjusted according to their individual 3.81, SD = 1.06) than player-produced sounds (M = 3.14, SD =
height, electrodes were attached and the participant was asked to 1.44).1 The presence of sounds in this level was rated very
rest and focus on a black cross on a grey background on the important (M = 4.17, SD = 1.03). After playing without any
monitor. During this resting period of 3-5 minutes, physiological sounds or music, most participants also claimed that it bothered
baseline recordings were taken. them a lot (M = 4.06, SD = 1.19). For the “music only”
modality, it was noted as well that 75% of the participants
Next, participants were seated in front of a high-end gaming missed the sound effects.
computer, which used a 5.1 surround sound system for playback
(Half-Life 2 sound quality settings on ‘high’), and were GEQ Sound & Diegetic Nondiegetic No Sound
component Music Sound Music Only or Music
encouraged to get acquainted with the game controls for two Only*
minutes (using a non-stimulus game level) if they did not Immersion 1.53 (1.08) 1.51 (1.01) 1.13 (0.95) 0.85 (0.79)
indicate a priori FPS experience. The participants played the Tension 0.97 (1.12) 1.41 (0.81) 1.94 (1.13) 1.57 (1.14)
Competence 2.18 (1.21) 2.24 (0.93) 1.88 (1.10) 1.57 (1.06)
same Half-Life 2 game level four times for 10 minutes (or until Flow 2.10 (1.28) 2.37 (0.94) 1.72 (1.32) 1.50 (1.12)
completed) in a counter-balanced order to eliminate repeated- Negative 0.86 (0.86) 1.06 (0.62) 1.63 (1.02) 1.71 (1.10)
measures effects. As this was a preliminary experiment designed affect
to produce broad subjective answers and objective Positive affect 2.06 (1.11) 2.16 (0.93) 1.61 (0.93) 1.49 (1.00)
Challenge 2.18 (1.08) 2.19 (0.69) 1.96 (0.85) 1.60 (0.81)
measurements from which future, more refined experiments can
be designed, the following broad sound on/off modalities were
chosen: Table 1: Means (and standard deviations) of the GEQ
components for the four test sound modalities2
1. the level with all diegetic sounds and nondiegetic
music audible Table 1 shows a comparison of GEQ mean scores. A
2. the level with just diegetic sounds audible comparison of these values shows that regardless of GEQ
3. the level with just nondiegetic music audible components, people gave higher ratings (except for tension and
4. the level with no sound or music audible negative affect) when sound was present. The presence of
1
Potentially supporting the theory of challenge-based immersion. [8]
After each modality, participants were asked to report their 2
subjective experiences using questionnaires. After completion of N = 36, N of items = 2(*5), GEQ scale has values between 0 and 4
(median = 2).

-4-
12
diegetic sound (whether combined with music or not) also seems
to be an enabling factor of the subjective experience of
challenge and flow — flow especially seems be experienced
more easily with diegetic sounds.
The complete absence of sound seems to negatively influence
the subjective feeling of immersion to a significant degree as it
is the lowest rated item in this modality. With missing auditory
feedback, there is also a decrease in the feeling of competence
among all participants. The combined presence of sound and
music seems to also have a soothing effect on play as ratings for
tension and negative affect are very low under this modality. It
is also the modality that has the highest score for the immersion
item. However, it should be noted that music also seems to be
somewhat distracting from game flow since flow ratings are
higher when music is omitted and only diegetic sounds are
presented.
For GEQ components Immersion (χ2(5) = 3.49, p > .50),
Competence (χ2(5) = 10.28, p > .05), Negative Affect (χ2(5) =
5.36, p > .30), and Flow (χ2(5) = 10.12, p > .05), Mauchly’s test
indicated that the assumption of sphericity had been met, but for
the remaining items Tension (χ2(5) = 11.98, p < .05), Positive
Affect (χ2(5) = 11.56, p < .05) and Challenge (χ2(5) = 23, p <
.05) it was violated. Therefore, degrees of freedom were
corrected for the latter three using Greenhouse-Geisser estimates
of sphericity (ε = .80, ε = .84, and ε = .71).
Statistical significance was achieved for all components
(Immersion: F(3, 105) =8.20, p < .001), Competence: F(3, 105)
= 4.49 , p < .01), Negative Affect: F(3, 105) = 9.75 , p < .001),
Flow: F(3, 105) = 9.42 , p < .001), Tension: F(2.39, 83.73) =
7.85 , p < .001), Positive Affect: F(2.52, 88.21) = 6.18 , p < .01),
and Challenge: F(2.14, 74.78) = 5.17 , p < .01)). These results
show that the subjective game experience measured with the
GEQ was significantly affected by the different sound
modalities.
Table 2 shows a comparison of the normalized physiological
responses. Negatively valenced arousal would be indexed by
increased GSR and corrugator supercilii activity (with decreased
zygomaticus major and orbicularis oculi activity). [29] This is
not the case for any of the accumulated measurements shown.
The only notable decrease of orbicularis oculi and zygomaticus
major activity is shown under the no sound condition. However,
corrugator supercilii activity is also decreased and galvanic skin
response is somewhat consistent across conditions.
Physiological Sound & Diegetic Nondiegetic No Sound or
Response Music Sound Only Music Only Music
Orbicularis 1.85 (0.37) 1.85 (0.37) 1.86 (0.42) 1.79 (0.31)
oculi (ln[µV])
Corrugator 1.94 (0.25) 1.90 (0.27) 1.95 (0.33) 1.89 (0.26)
supercilii
(ln[µV])
Zygomaticus 1.98 (0.40) 2.00 (0.38) 2.00 (0.43) 1.94 (0.35)
major (ln[µV])
Galvanic skin 0.72 (0.18) 0.73 (0.17) 0.70 (0.18) 0.72 (0.17)
response
(log[µS])
Table 2: Means (and standard deviations) for the corrected
physiological measurements (EMG and GSR) under the
different modalities3
Accordingly, Mauchly’s test indicated that the assumption of
sphericity had been violated for orbicularis oculi EMG means
3 an additional method for recording subjective
N = 29 (after data reduction).
-5-
(χ2(5) = 25.16, p < .001), corrugator supercilii EMG means
(χ2(5) = 57.65, p < .001), zygomaticus major EMG means
(χ2(5)= 16.43, p = .006) and GSR means (χ2(5)= 52.41, p <
.001). Hence, degrees of freedom were corrected using
Greenhouse-Geisser estimates of sphericity for EMG means (ε =
.63, ε = .45, and ε = .76) and GSR means (ε = .47).
Nevertheless, neither EMG responses (orbicularis oculi: F(1.90,
53.21) = 0.86, p > .40, corrugator supercilii: F(1.36, 38.02) =
0.66, p > .40, zygomaticus major: F(2.27, 63.58) = 0.61, p >
.40), nor GSR [F(1.40, 39.05) =0.68, p > .40] could achieve
statistical significance for the repeated measures design. The
results of the ANOVA show that tonic measurements of
physiological response from an accumulated game session were
not significantly affected by different sound modalities.
Discussion and Future work
This paper has described and analyzed the results of a
preliminary experiment to measure the effect of FPS sound and
music on player valence and arousal and to detect any possible
correlations between measurable valence and arousal features
and self- reported subjective experience.
There are two important and related results. Firstly, the data
gathered from the subjective questionnaires (see Table 1) shows
a significant statistical difference between the four modalities
over the GEQ components. This is particularly the case with
Flow and Immersion, the results of which show higher scores
when diegetic sound is present than when it is not. Prima facie,
this would indicate that diegetic sound does indeed have an
immersive effect in the case of FPS games. Music also appears
to increase immersion, while reducing tension and negative
affect, at the expense of a reduction in the experience of flow
within gameplay.
Secondly, the psychophysiological data do not support the
subjective results, but are instead both inconclusive and lacking
statistical significance (see Table 2). If we maintain the
assumption that physiological evidence, in these circumstances,
can be used to confirm the subjective evidence, then there are
several potential explanations for the lack of correlation between
the two result sets. Further analysis and experimentation will be
required to explain this disparity. Some initial possible
explanations (assuming a valid experiment design and
implementation) include:
1. The GEQ incorporates distortions derived from the
retrospective “storytelling” context of the
questionnaire.
2. The physiological data, gathered over 10 minutes of
play, contains too much noise to produce a significant
result. It must be noted that the data analyzed here was
accumulated over one game session and even after
inspection of histograms and logarithmic correction
not all measurements were perfectly normally
distributed. Even though a non-parametric statistical
analysis or a range correction of physiological
responses could be conducted, it is unlikely that this
will show significant effects over the 10 minute
timescale used. Connecting physiological response
data to game events using more precise phasic
measurements, as described in Nacke et al., [30] could
yield more insight into the emotional effects of sound.
This level of detail can be achieved but it would need
13
 responses at the same event level precision to be
correlated with.
3. The subjectively reported experience is a function of
the modulation of emotions within a smaller time
scale than that used in the analysis of
psychophysiological data. This means that the
emotional net effect may be the same, but the details
of emotional dynamics produce different subjective
experiences as reported by the GEQ. As analogy: a
flat sea and a sea with big waves may have the same
mean level, but one makes for much better surfing
than the other. This might be detectable by derived
measures form the current data set.
4. The subjectively reported results are not measurable
using our equipment and methods. In particular, the
source of the GEQ components reported in Table 1
may have a different psychological explanation than
that captured by the arousal/valence model of
emotion. This consideration raises the need for more
thorough ongoing conceptual investigations of terms
such as immersion, presence, flow, challenge and fun
(as started in [31]). Based upon a richer range of
linguistic and conceptual distinctions, it may be
possible to devise experiments having more
discriminating power among the range of descriptive
models thus created. In particular, these are complex
concepts used in different ways by different authors,
and it may not be the case that they have simple
mappings to instantaneous emotions measured by
psychophysiological techniques. Explanatory theories
then need to move to higher levels in modeling the
structuring of a series of measurable emotions, related
to perceptions and player actions, to provide a more
systemic account of the foundations of the quality of
play experience, as suggested by Lindley and
Sennersten. [32]
These questions must be addressed by ongoing research. To our
surprise, our research contradicts the results presented by
Shilling et al. [18], who indicated a strong correlation between
sounds and physiologically elicited emotions. Unfortunately,
Shilling et al. did not report direct values of their measures that
would allow a direct comparison. It remains for more thorough
future analysis to find greater scientific evidence for a
relationship between sound and psychophysiological measures.
Our future aim is to investigate this within our research.
Acknowledgements
The research reported in this paper has been partially funded by
the FUGA (FUn of GAming) EU FP6 research project (NEST-
PATH-028765). We thank our FUGA colleagues, especially
Niklas Ravaja, Matias Kivikangas, and Simo Järvelä, for many
stimulating inputs to this work. We would also like to thank
laboratory assistant Dennis Sasse for helping in the execution of
the experiment.
References
[1] id Software, Doom series, Activision, (1993-2004).
[2] id Software, Quake series, Activision, (1996-2005).
[3] Valve Software, Half-Life series, Electronic Arts, (1998-
2004).
-6-
[4] Grimshaw, M., The Acoustic ecology of the first-person
shooter: The player, sound and immersion in the first-person
shooter computer game, Saarbrücken, VDM Verlag Dr. Mueller
e.K., (2008).
[5] McMahan, A., Immersion, engagement, and presence: A
new method for analyzing 3-D video games, in M. J. P. Wolf &
B. Perron (Eds.), The Video Game Theory Reader, New York,
Routledge, 67–87 (2003).
[6] Kearney, P. R., & Pivec, M., Immersed and how? That is the
question, Game in' Action, Sweden, (2007, June 13—15).
[7] Pine, B. J., & Gilmore, J. H., The experience economy: Work
is theatre & every business a stage, Boston, Harvard Business
School Press, (1999).
[8] Ermi, L., & Mäyrä, F., Fundamental components of the
gameplay experience: Analysing immersion, Changing Views –
Worlds in Play, Toronto, (June 16—20, 2005).
[9] Murray, J. H., Hamlet on the holodeck: The future of
narrative in cyberspace, Cambridge, The MIT Press, (2000).
[10] Garcia, J. M., From heartland values to killing prostitutes:
An overview of sound in the video game Grand Theft Auto
Liberty City Stories, Audio Mostly 2006, Piteå, Sweden,
(October 11—12, 2006).
[11] Carr, D., Space, navigation and affect, in Computer Games:
Text, Narrative and Play, Cambridge, Polity, 59-71 (2006).
[12] Laurel, B., Computers as theatre, New York, Addison-
Wesley, (1993).
[13] Jørgensen, K., On the functional aspects of computer game
audio, Audio Mostly 2006, Piteå, Sweden, (October 11—12,
2006).
[14] Murphy, D., & Pitt, I., Spatial sound enhancing virtual
story telling, Lecture Notes in Computer Science, 2197, 20–29
(2001).
[15] Grimshaw, M. & Schott, G., A Conceptual framework for
the design and analysis of first-person shooter audio and its
potential use for game engines, International Journal of
Computer Games Technology, 2008, (2008).
[16] Stockburger, A., The rendered arena: Modalities of space
in video and computer games, unpublished PhD thesis, London,
University of the Arts, (2006).
[17] Grimshaw, M., The Resonating spaces of first-person
shooter games, Proceedings of The 5th International Conference
on Game Design and Technology, Liverpool, (November 14—
15, 2007).
[18] Shilling, R., Zyda, M., & Wardynski, E. C., Introducing
emotion into military simulation and videogame design:
America’s Army: Operations and VIRTE, GameOn, London,
(2002).
[19] MOVES Institute, America’s army, Monterey, Naval
Postgraduate School, (2002).
[20] Cacioppo, J.T., Tassinary, L.G. and Berntson, G.G.
Psychophysiological science, Handbook of psychophysiology,
3-26 (2007).
[21] Cacioppo, J.T., Handbook of Psychophysiology, Cambridge
University Press, (2007).
[22] Lang, P. J., The emotion probe. Studies of motivation and
attention, American Psychologist, 50, 372–385 (1995).
[23] Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm,
A. O., Looking at pictures: Affective, facial, visceral, and
behavioral reactions, Psychophysiology, 30, 261–273 (1993).
[24] Valve Software, Half-Life 2, Electronic Arts, (2004).
[25] Schmitz, C., LimeSurvey (v1.70) from
http://www.limesurvey.org (2008).
[26] Fridlund, & Cacioppo., Guidelines for human
electromyographic research, Psychophysiology, 23(5), 567-589
(1986).
[27] IJsselsteijn, W.A., Poels, K. and de Kort, Y.A.W.,
Measuring player experiences in digital games. Development of
14
the Game Experience Questionnaire (GEQ), Manuscript in
preparation.
[28] Nacke, L. and Lindley, C., Boredom, Immersion, Flow - A
Pilot Study Investigating Player Experience, IADIS Gaming
2008: Design for Engaging Experience and Social Interaction,
IADIS, Amsterdam, The Netherlands, (2008).
[29] Ravaja, N., Turpeinen, M., Saari, T., Puttonen, S. and
Keltikangas-Jarvinen, L., The Psychophysiology of James Bond:
Phasic Emotional Responses to Violent Video Game Events,
Emotion, 8 (1), 114-120 (2008).
[30] Nacke, L., Lindley, C. and Stellmach, S., Log who’s
playing: psychophysiological game analysis made easy through
event logging, International conference on Fun and Games 2008,
Springer, Eindhoven, The Netherlands, (2008).
[31] Lindley, C.A., Nacke, L. and Sennersten, C., What does it
mean to understand gameplay? First Symposium on Ludic
Engagement Designs for All, Aalborg University, Esbjerg,
Denmark, (2007).
[32] Lindley C. A. and Sennersten C., 2006 A Cognitive
Framework for the Analysis of Game Play: Tasks, Schemas and
Attention Theory, Workshop on the Cognitive Science of Games
and Game Play, The 28th Annual Conference of the Cognitive
Science Society, Vancouver, Canada, (July 26—29, 2006).

-7-
15
 Sound and the diegesis in survival-horror games

Daniel Kromand
Brydes Allé 23, 431
2300 Copenhagen S
kromand@itu.dk

Abstract. The paper analyzes the affordances of the soundscape in survival-horror games by examining the barrier between diegetic
and non-diegetic sounds and how the validity of a sound cue is challenged. The three game examples all veil their aural warning cues
with a broken causality and causes the player to distrust the cues’ tie to the game world. This results in an uncertainty whether the
sound comes from an in-game object or from non-diegetic ambience. The paper argues that the horror genre draws upon this
discrepancy between soundscape and perceived reality to create an ambience of fear.

1. Introduction Grimshaw defines the acoustic ecology of multiplayer first-

The screeching tires in a racing game; the high pitch of
incoming artillery fire; the birds chirping in a haunted forest; all
of these are a natural and ingrained part of the virtual worlds we
inhabit in video games. But in contrast to the real world, these
sounds are not the causal effect of rubber moving across asphalt,
a projectile moving through air at high speed, or a bird pressing
air out of its lungs. Instead they are sound files played when
certain parameters in the virtual world have been met. This
difference might be subtle, but justifies a thorough analysis of
acoustic environments in games, as these have all been designed
through conscious effort and are not results of physics. Games
are virtual environments with content hard-coded into them. In a
linear single player game certain sounds can be set to play when
the player moves through the module, creating a specific
soundscape for each in-game area. Such a modular approach to
audio design can create a very tightly tailored soundscape,
where sound can provide a dramatic increase of tension.
This paper aims to analyze the dramatic use of audio design in
survival-horror games due to the fact that this particular genre
has a significant use of audio, which plays a central part in the
main feature of the genre: Fear and uneasiness. The genre of
survival-horror rests upon a series of stylistic and design
choices, the most notable being: A scary setting, a scarcity of
gun ammunition, awkward camera angles, and puzzle solving
(Whalen 2004). Defining the semantic elements of a genre is
often difficult as neither of them are singularly essential, but
rather make up the parts for the holistic definition (Buscombe
2003). I have chosen three broadly recognized survival-horror
games –BioShock, F.E.A.R., and Silent Hill 2– that together
form a generalizable reference to the genre as a whole. The goal
is to prove how the audio design purposefully creates
uncertainty in the division between the diegetic and the non-
diegetic. This stresses the player into carefully considering his
actions and puts him on alert even though no visual threats are
apparent. The collapse of the barrier between the diegetic and
non-diegetic soundscape is a strategy to build a horror
atmosphere.
2. Sound in an acoustic ecology
To analyze the use of audio in survival-horror games it is
imperative to structure a framework for the soundscape of video
games. Mark Grimshaw argues for such a framework by
utilizing theories of acoustic ecology, which combines a notion
of spatiality and the aspect of time (Grimshaw 2007; 6).
person shooters as “a set of relationships between players
founded upon sound” (Grimshaw 2007; 6). This statement, apart
from the expectation of several players, forms this paper’s basic
understanding of sound in an acoustic ecology, but it needs
further elaboration in order to be operational within a broader
theoretical framework. Grimshaw perceives an acoustic ecology
as divided into sounds that relate to objects within the game
(‘players’ in the aforementioned quote) and those that do not:
The former form part of an understanding of the
game’s acoustic ecology whilst the latter do not.
A conceptualization of an acoustic ecology helps
to explain objects and actions in the game world
because it comprises sounds produced by those
objects and actions. (Grimshaw 2007; 221)
The understanding, which Grimshaw refers to, requires a player
that is able to discern the sound object from the rest of the
soundscape and to select the various affordances offered by it.
The player relies on experience and contextualization to do so,
which means that a sound object might produce different
meanings for the same player depending on his familiarity with
the causality inside the game and on the situation in which it
arises (Grimshaw 2007; 136). The player thus continuously
connects experienced sound objects to causal effects within the
game world to build a personal framework for deciphering
relevant information through the acoustic ecology.
The acoustic ecologies contain information about the
relationships between objects in the game world and the actions
that they perform. The players discern this information from the
rest of the sensory filler through a continued connection of
causal effects. Sound is a carrier of both immersion and
information, the latter extracted through affordances as
described earlier. Immersion through sound can come from a
variety of aural objects, for example the beating heart that
typically accompanies low health status, seeks to give the player
a sense of embodying the avatar.
Grimshaw’s framework for an acoustic ecology and aural
immersion in the game world will be the primary tool for
analyzing the soundscape of single-player games, in the shape of
the selected survival-horror games. The player’s found
affordances will be examined closely in order to reveal how
sound is used for dramatic effect.

16
3. Trans-diegetic sound
The trans-diegetic effect of audio examined by Kristine
Jørgensen (2007) is a transgression of the traditional barrier
between diegesis and non-diegesis as explained by
Bordwell/Thompson (2004). The traditional distinction between
diegesis and non-diegesis divides the soundscape into sounds
that are heard within the fictional world and those that are not.
The film viewer’s ability to understand this divide comes from
repeated exposure to the language of films, where breakdowns
of the barrier are rare and usually act as comic relief (such as the
big band in Mel Brooks’ Blazing Saddles that is thought to be
non-diegetic, but then happens to be located out in the desert).
Jørgensen argues however that due to the interactivity of video
games, sound can pass the barrier and in effect become trans-
diegetic. Units in Warcraft III for example speak directly to the
player as a way of conveying information, i.e. they speak from
within the diegesis to the outside, while music in the game can
function as a leitmotif for certain events, allowing the player to
anticipate future events (Jørgensen 2007; 110). Trans-diegetic
sound does not dissolve the barrier though: It merely causes a
short transgression that still keeps the division between diegesis
and non-diegesis intact. The trans-diegetic effect therefore
typically transfers information from the game to the player,
according to Jørgensen in two different versions: Either as a
reactive sound affirming player input or as a proactive sound
informing the player of an altered game state (Jørgensen 2007;
116).
Jørgensen’s argument relies upon the fact that players can
interpret the soundscape of a given game as triggered by specific
events, i.e. not being completely random. The ability to interpret
can partly be learned through genre conventions and by keeping
a consistency of sound within the individual game itself
(Grimshaw 2007; 102-103).
4. Welcome to Rapture
When first checking into the grim world of BioShock (2K
Games 2007) and immersing into the city of Rapture the scene is
one of confusion and paranoia: Monsters –the so-called
splicers– are roaming the hallways, ammunition is limited, and
the ambience is filled with running water and creaking metal.
The soundscape of BioShock is densely populated with a
general ambience, and sounds tied to the inhabitants of Rapture
and the player.
The in-game objects of BioShock, such as the player, enemies,
robots and vending machines, offer a quite varied selection of
sound objects. Movement creates sound, which can help the
experienced player identify the source (e.g. the heavy thud of a
Big Daddy is easily recognized, while stealthier opponents, such
as the spider splicer, create a minimum of noise). Speech is
another form of identifying opponents, since several of them
will replay sentences: Some are preset, for example a specific
splicer early in the game –a deranged mother bending over a
baby carriage– plays the same monologue when encountered
(although it is interruptible according to the player’s actions),
while other enemies replay their sentences at random, which
both adds to the atmosphere of Rapture, but also informs the
player of possible danger.
The ambience in BioShock lingers between the diegetic and the
non-diegetic: a typical engine hum is for example audible, but
underneath there are sporadic, high-pitched sounds. The latter
does not appear to have any connection to the diegetic world of
Rapture, and likewise while in Neptune’s Bounty (an area of
Rapture) the player will hear the omni-present creaks of metal
and running water, but might also notice a monotonous note
played in crescendo. These two examples are arguably not
background music as they do not have a constant presence and
only constitute a limited tune. It might however be argued that
they are examples of Jørgensen’s point that ambience should not
be taken literally: Instead ambience frames the general
atmosphere in the player’s current area (Jørgensen 2007; 110).
A crescendo is traditionally used in horror movies to instill
anticipatory fear, an effect that BioShock mimics in a slightly
different form. In BioShock the crescendos are activated after a
time lapse and not on basis of narrative structures. I argue that
this leads to an increase in tension, as most players will be
familiar with the culturally implied meaning of such a crescendo
(a leitmotif for shocks), and thus they will be prepared for
encounters, which may or may not take place. The soundscape
furthermore often has a deep throb present in the ambience, only
clearly audible at high volume settings. This throb is not held at
the same note, and thus dispels silence and helps fill the
soundscape with unfamiliar noises. Not all ambience in
BioShock is unmelodic though: At select fights throughout
BioShock regular background music is played, for example at
the first encounter with a Houdini splicer in Arcadia or the fight
against Dr. Steinman in the Medical Pavilion, which fulfills the
dramatic effect of a standard boss fight.
The part of the ambience that is closer to the diegesis (e.g. the
creaks, announcements over the speakers, etc.) serves to create a
general atmosphere and fills the sensory system. Grimshaw
refers to ‘sensory fillers’ as music that is irrelevant for gameplay
(2007; 4), but in BioShock the sensory filler feeds the player too
much aural input to cope, which in turn causes confusion
regarding the diegetic ties to the game world. The creaking noise
of Rapture fills the soundscape and potentially masks or
muddles the sounds produced by real, in-game enemies. The
player’s understanding of affordances can help to perform better
in the game of BioShock, as certain sounds pass information
regarding nearby opponents, but at the same time these exact
affordances are mimicked in the ambience. This creates a dual
relationship to the soundscape of BioShock: On one hand the
spatial location of opponents is revealed through hearing and
helps to prepare the player for an encounter, but on the other
hand the ambiences causes a distrust in the player towards the
division of diegetic and non-diegetic sounds.
5. Paced horror in F.E.A.R.
F.E.A.R. (Monolith Productions 2005, henceforth FEAR) is set
in modern day, albeit with slightly futuristic technology, and
revolves around a special unit’s attempt to handle a paranormal
crisis. The game draws heavily on Japanese horror with frequent
submersions in water, movement through dark crawlspaces, and
a The Ring-inspired ghost called Alma. The game resembles a
generic shooter at times and player will often find himself in
firefights with enemy soldiers, but other elements of the game
creates an atmosphere of horror and uneasiness, for example
crawlspaces, which –paired with a limited flashlight battery and
a giggling ghost– builds a very tense experience.
The ambience in FEAR does not adapt to fights as seen in other
games (Whalen 2004), and is usually deep bass or a slowly
oscillating pitch. Similar to BioShock the ambience lacks
tonality and a clear melody, and therefore is likely to be
perceived as part of the diegetic world. An analytical player
might identify the ambience as non-diegetic, but the atonality
breaks the usual expectations to background music. Arguably
atonal ambience can be experienced as closer to the diegesis as
it is less cohesive, thereby provoking uncertainty about the
17
sound’s non-diegetic nature. The ambience in FEAR thus
assumes the role of sensory filler with uncertainty as an effect.
FEAR paces the player in a manner, which is not unlike
BioShock: The action sequences are designed as a wavelike
curve with enemy resistance focused in smaller areas in contrast
to being spread out over the level. The player rarely encounters
a lone soldier and does not experience a steady stream of
enemies. Therefore the player will also traverse a relatively
large amount of space with no enemy encounters. Ironically it is
in these unchallenged spaces that the player can be expected to
fully experience the game’s horror atmosphere. In areas where
the player is left alone both visual and aural cues (such as the
flickering of static, which is often, but not always, a premonitory
warning) allude that paranormal threats are nearby. The second
encounter with the ghost Alma –inside the aforementioned
crawlspace– the avatar’s pulse and breathing becomes audible
while Alma can be heard shuffling around and giggling. Earlier
Alma was capable of killing the player, and thus her presence
inside the crawlspace naturally causes distress. Before
encountering Alma in the crawlspace, the player is aware that
something is about to happen, primarily due to the set-up –a
dark, claustrophobic space– and the aural cues of heavy
breathing and pulse. Nothing paces the player to go through the
crawlspace, but the space has to be traversed if the player wants
to progress the game. At his own pace the player has no option
but to move through an area where aural cues have indicated
that danger might be lurking. This design resembles Hitchcock’s
well-known example of a bomb hidden underneath a table where
two people are having a conversation, which will cause
prolonged suspense for the audience if they are aware of its
existence (cited in Perron 2004; 2). In FEAR the player knows
that a threat is present, but the cues are slow to reveal exactly
when and where it will happen. This threat warning system
combined with a minimum of pacing is created to make the
player move very slowly and to examine every minor aural or
visual cue. The horror in FEAR thus relies on an uncertain
cueing of threats and slow pacing that allows the player to fully
immersive himself in the sinister atmosphere.
FEAR utilizes a variety of sounds that afford a threat warning,
such as the radio communication between enemy soldiers, which
can be heard far away and is the primary source for identifying
their presence before visual confirmation can be made. Other
sounds afford a similar warning, but lack the consistency of the
radio signals, such as the static caught in the player’s earpiece.
The static is usually an indicator for approaching paranormal
events, but is sometimes played without any following
consequences. This pseudo-causality is designed to put the
player on edge and make him carefully considering his moves
even though no threat is imminent. The misuse of the static
reduces the player’s faith in it as a reliable tool, but accentuates
that something might happen.
Aural affordances are, as in BioShock, limited because they
have flawed causality. The unreliable affordances delivered by
the audio design keeps the player in a limbo between trust and
apathy. Furthermore the slow pacing in the game’s horror
sequences adds another dimension, as the player will have to
approach a possible threat at his own pace. FEAR gives release
to the built up tension through intense firefights with enemy
soldiers where the game resembles a generic shooter.
6. Finding way through the fog in Silent Hill 2
Silent Hill 2 (Konami 2001) takes place in a town of the same
name, which is immersed in constant fog. The player controls
the avatar in third person and has to traverse a number of
puzzles and monsters to reach the end. The games of the series
have been examined academically in their own right, and
especially the radio carried has received a lot of academic
attention (Carr 2003, Whalen 2004, Perron 2004).
The radio carried by the avatar in Silent Hill 2 produces static
whenever enemies are approaching, but similar to the static in
FEAR it fails under certain conditions. At select locations the
radio will produce static that sounds like a mixture of chirping
birds and a squeaky wheel, but is not produced by any entities.
This sound is not ambience, as it still can be heard when the
background music is turned off in the options menu.
Furthermore the radio does not register the “crawlers” (the
zombies in their cockroach-like state) and the player is likely to
be surprised when one them runs off with a high-pitched squeal
from underneath a car (as it happens on Martin St.). Therefore
the radio, which is the primary tool for locating enemies –due to
the omnipresent fog– has both too much and too little sensitivity
to be a completely reliable warning system.
The ambience of Silent Hill 2 is in some sense similar to both
BioShock and FEAR, as it is highly metallic and often a deep
bass with an oscillating pitch, but with a significant difference in
the fact that ambience is situated, which means that topological
locations have their own sound (e.g. one apartment has another
ambience than others). The dynamic and unsettling soundscape
makes it harder for the player to get accustomed with the
ambience. This distorts an efficient reading of the radio’s
affordances, as some rooms upon entry trigger static from
nearby enemies, while other rooms trigger a new ambience (for
example room 208 in Wood Side Apts.). The atonal quality of
the ambience makes it harder to differentiate between static
from the radio and from the ambience. Due to the fact that the
radio often displays different sounding warnings, new
background ambience can create uncertainty whether monsters
are nearby or not.
As in both BioShock and FEAR the soundscape in Silent Hill 2
purposefully works against the player’s effort to read the
affordances of aural warning cues. The game does supply them,
but with an irregular set of consequences and therefore a broken
causality. The soundscape of Silent Hill 2 operates within a
frame of uncertainty that constantly holds the player between
knowledge and ignorance. Along with limited ammunition and
field of vision, the soundscape efficiently builds a setting of
horror.
7. Conclusion
Analysis of the three games shows they mainly have unmelodic
ambience without a tonic (although music was included at select
moments). The soundscape affords less (or in a sense too much)
to the player, as sounds can be hard to tell apart. This was
especially the case in FEAR and Silent Hill 2 where the player’s
in-game warning system proved to be unreliable. The player’s
difficulties of dividing sounds into non-diegetic and diegetic can
be described as an actual collapse of the diegetic barrier. This
effect goes beyond Jørgensen’s theory of trans-diegesis, since
the players do not prepare for future action based on non-
diegetic information. Instead they act because they do not know
if the sound is diegetic or not. The atonal ambience reduces the
perceived field of non-diegetic sound, with the exception of the
boss music, and all sounds can be suspected to belong to the
diegesis. The diegetic collapse is the effect when player are no
longer fully capable to discern between diegetic and non-
diegetic elements. It is safe to say that the soundscape of the
three survival-horror games produce an ‘un-knowledge’ and
while it does offer affordances, it also creates an uncertainty
18
Sound and the diegesis in survival-horror games

about the validity of the said affordance. It is simply impossible

to extract certain causality from some affordances because the
ambience mimics the sounds flawlessly, and only by replaying a
level can the player discern between the real warning sounds
and those that are not. Even though the analysis accepts
Grimshaw’s thesis on the players’ usage of soundscape
affordances, I also claim that games like the ones analyzed
purposefully hinders an efficient transfer of affordance and that
exact point might well be one of reasons why some players
enjoy these games.

The paper did not aim to dismantle either Jørgensen’s or

Grimshaw’s theories, but rather to prove that certain genres have
a varied use of soundscape and affordances. The idea of
uncertainty as produced by the relation between ambience and
in-game objects helps for an understanding of both the formal
design of survival-horror games and also the players’
relationship to aural affordance. The constant guessing as to
whether the sounds have a causal connection put the players in
unusual insecure spot that might well build a more intense
experience.

References
[1] Aarseth, Espen, Quest games as Post Narrative Discourse,
Narrative Media, pp. 361-376. University of Nebraska Press
(2004)

[2] Bordwell, David & Kristin Thompson: Film Art: an

introduction, 7th Ed., McGraw-Hill 2004

[3] Buscombe, Edward: “The Idea of Genre in the American

Cinema” in Film Genre Reader III (Barry Keith Grant ed.),
University Of Texas Press 2003

[4] Carr, Diana: Play Dead – Genre and Affect in Silent Hill
and Planescape Torment, Gamestudies vol. 3 issue 1, 2003

[5] Grimshaw, Mark Nicholas: The Acoustic Ecology of the

First-Person Shooter, University of Waikato 2007

[6] Jørgensen, Kristine: “On Transdiegetic Sounds in Computer

Games” in Digital Aesthetics and Communication, Vol 5. (Arild
Fetveit & Gitte Stald ed.): pp. 105-117, Intellect Ltd. 2007

[7] McMahan, Alison: “Immersion, Engagement, and Presence:

A Method for Analyzing 3-D Video Games” in The Video Game
Theory Reader (Mark Wolf and Bernard Perron ed.), Routledge
2003

[8] Perron, Bernard: Sign of a Threat: The Effects of Warning

Systems in Survival Horror Games, University of Montreal 2004

[9] Whalen, Zack: Play Along – An Approach to Videogame

Music, Gamestudies vol. 4 issue 1, 2004

19
 Psychologically Motivated Techniques for Emotional Sound in
Computer Games
Inger Ekman, inger.ekman@tml.hut.fi
Department of Media Technology,
Helsinki University of Technology
P.O.Box 5400
FIN-02015 HUT

Abstract. One main function of sound in games is to create and enhance emotional impact. The expressive model for game sound
has its tradition in sound design for linear audiovisual media: animation and cinema. Current theories on emotional responses to
fiction are mainly concerned with linear medial, and only partly applicable to interactive systems like games. The interactivity
inherent to games introduces new requirements for sound design, and suggests a break in perception compared with linear media.
This work reviews work on emotional responses to fiction and applies them to the area of game sound. The synthesis is
interdisciplinary, combining information and insights from a number of fields, including psychology of emotion, film sound theory,
experimental research on music perception and philosophy. The paper identifies two competing frameworks for explaining fictional
emotions, with specific requirements, and signature techniques for sound design. The role of sound is examined in both cases. The
result is a psychologically motivated theory of sound perception capable of explaining the emotional impact of sound in film, as well
as identifying the similarities and difference in emotional sound design for these two media.

question we are concerned with here is how to tackle all these

1 Introduction issues while maintaining the emotional functions of sound.
The importance of sound for establishing mood in
Surprisingly little academic investigations into sound have
audiovisual media is well recognized. It should come as no
addressed this issue. Also, while there are many books on
surprise that viewing a movie without sound can strip it of its
game sound design, the majority of literature deals either
emotional impact, making the events pictured on screen
with the technological tools for creating the sounds or
seemingly distant and of little relevance. The potency of
managing the overall process of sound design e.g. [2][25].
sound is also rather well exploited at least in contemporary
The best foundations for game sound design decisions can be
gaming. The model for sound design in these games is often
found in the fragmental accounts written by designers
that of film sound; game sounds should be “Cinematic” [3] or
[22][3][13], sharing glimpses of their intuition and
“Bigger than life” [22].
experience in the craft through audio post mortems and
project descriptions.
However, it is a well known fact that games differ from film.
Also sound design in games is different from film sound on
Underlying many successful design decisions is an implicit
several points. An obvious disparity resides in the technical
understanding of what kind of sounds and actions create
realization of game sound compared to film sound. For one,
emotional impact. The aim of this study is to bring together
the expected technical setups for game and film consumption
various sources of information, to contribute to an explicated
are of a vastly different nature. Technical aspects are,
version of this knowledge. Such knowledge may ultimately
however, outside the scope of this paper. Instead, the focus is
aid the design process, and it should be of benefit to
on the special representational requirements that games, as
researchers working to understand the phenomenon of game
interactive systems, put on the sound design as compared to
sound. To this purpose the paper will jointly address two
those within static linear media. Interactivity creates a
close questions, namely why and how sound works to create
fundamental differentiation in what constitutes as the sound
emotion. Asking why looks for the reason behind having
design for film, and games, respectively. This is to say that
emotional reactions to sound emanating from fictional worlds
the final mix for a film differs by nature from the end result
in the first place. Answering this question requires looking at
of game sound design, usually a set of programmatically
the relationship between player and game, and the role of
expressed rules for combining/manipulating sound samples
sound in shaping this relation. The second question, how,
as a player interacts with the system. While technology
deals with the mechanisms underlying emotional reactions to
dictates the alternatives on both sides, the distinction between
sound, seeking cognitive and biological explanations to the
the two is present by nature of film vs. game, and exists
emotional experiences of a certain sound used in a certain
independently of technological issues. This aspect is the
context. Finally, a preliminary synthesis of these findings
focus of this investigation.
sketches out some practical sound design implications for
emotional impact in film and games, respectively.
These are the challenges game sound has to conform to: 1)
Whereas film sound is written to a fixed set of actions, seen
This work serves several purposes. First my hope is that an
through predefined points of view, game sounds must support
understanding of the underlying processes of sound will
alternative paths and viewpoints through the story. 2) Film
provide useful concepts for practical work, by bringing to
sound is written to sequences of actions with known
light the alternative possibilities and approaches for
durations. Game sounds must span action sequences of
impacting the player experience. This view is in no way
flexible temporal duration. 3) Games ask players to become
attempting to challenge the value of intuition to design, nor to
active and play. Hence, sounds must support action, respond
explain away or trivialize the talent that goes into quality
to player control and often survive high repetitiveness. The
sound design. On the contrary, explanations of why and how

20
masterful sound designs influence emotion are as much a
celebration of the skills that went into designing them in the
first place as tools for shaping new experiences in the future.
I am also positive that a bit of clearly enunciated arguments
for why sound holds so much power can be of use, especially
in defending sound when it has to compete for attention (or
budget) with visual effects. Finally, this paper links into the
academic discussion on sound design, and provides a
synthesis of relevant literature from several vastly different
fields. The topic of sound design is still advanced in many
separate camps, partly because it is interdisciplinary in nature
and that requires tapping into so many fields all at once. The
goal in this paper is not to provide an exhaustive review.
Rather it offers a selection of findings from different sources,
suggesting both where future connections may be found and
directions how some of them may be explored further.
The structure of the paper is as follows: Chapter 2 discusses
the nature of emotion in general, and looks at emotion theory
applied to film and games. Chapter 3 considers the processes
wherein film sounds influence emotional response and
consider several sources of emotion in sound. Chapter 4
offers a summary on emotional effect of sound and compares
emotional sound in film with sound in the interactive context
of games.
2 Emotions and Emotional Responses in
Film and Games
In psychology literature, whole chapters have been written
about defining emotions. Perhaps it is due to the subjective
nature of emotional experiences, as well as culturally
negotiated nature of what emotions are suitable and for
whom, that it is hard to pin down precisely what constitutes
an emotion. Intuitively, however, the struggle to define
emotion seems puzzling. Most of us would seem to know, by
intuition, what sort of experience the word emotion refers to.
For the purpose of this text, let us though consider briefly
what it implies. Oatley and Jenkins [20] 96] mention the
following three defining features:
- Emotion is usually caused by a person (consciously or
unconsciously) evaluating an event of some
significance in relation to a goal or situation.
- The core of an emotion is a readiness to act, and
emotion influences actions by preparing for certain
types of actions and causing a sense of urgency.
- An emotion is usually experienced as a distinctive type
of mental state, sometimes accompanied or followed by
bodily changes, expressions, actions.
Emotions are thus evaluations of a specific (visceral and
urgent) type, which signal events of critical importance and
relevance to the perceiver. They provide the perceiver with
an evaluation of the event, producing (typically) either a
positive or negative sensation. This sensation is referred to as
valence. They also produce a sense of urgency. Generally,
the activation part of emotions is called arousal.
2.1 Fictional Emotions?
From a biological viewpoint the function of emotions seems
clear. They provide the foundations for successful behaviour
in an environment with promises and perils. Emotional
relations to film and games, however, is an example of
emotional relation to fictional events. This has been a very
controversial subject, and arguments go back and forth about
whether all these emotional reactions, which have their cause
in completely fictive events, are even real emotions at all.
And if they are real, what is their cause? To shed some light
on these questions, and as an inventory of the tools available
for looking at the effects of sound, let us turn for a moment to
psychology. What psychological support exists for emotional
reaction to fiction?
The simplest objection to calling emotional responses to
fiction real is related to the reality of the events underlying
these responses. Most emotion theories require that evaluated
events should be of real significance to the evaluating
subject. This is precisely the concern underlying Tan’s
argument [27]. Investigating the emotional responses in film
viewing, he bases the validity of his model in Frijda's [9]
theory of emotions and discusses in length the compatibility
of his theory with Frijda's basic tenets. In short, he argues
that emotional responses can arise to fictive events as long as
they are perceived as apparent reality. This is achieved
mainly by the diegetic effect, a cognitive and perceptual
illusion in the viewer' head. The illusion is maintained with
several cinematic techniques, usually by dissolving and
eradicating the medium. The witness position, Tan argues, is
one of the very means in which the cinema excuses the
viewer's passivity and explains, in keeping with the diegetic
effect, the viewer's ability to see it all.
Another way to make room for fictional emotions is to allow
emotional responses not only to real events, but to the
fictions of our imagination. Damasio does just so: he calls the
latter type as-if emotions, and describes in detail the
processes in which the mind simulates affective responses
[7]. According to Damasio, fundamentally affective
processes are part of all rational decision-making, and
function as a means of making decisions – we reason by
simulations on how the outcome might feel. From this point
of view, fiction would be just another way of running
simulations.
2.2 Emotional Responses to Film
There are two dominant approaches to understanding how
film elicits emotional responses. One approach is based on
Freudian psychoanalysis, invoking as a key concept the
Lacanian mirror stage, which “typifies an essential libidinal
relationship with the body-image” [14]. This stage is called
forth by identifying in film, a process that links the self with
the film experience through subconscious mental responses
(the basis of which is rooted in early childhood and
formations of the Ego). An alternative approach explains the
emotional reaction to film through cognitive appraisal of
portrayed fictional events. Instead of identifying
subconsciously, the viewer responds emotionally because of
their cognitive investment in the fictional events. The
viewers' emotional responses are related to motivational
processes [27][9][16].
According to the appraisal theory, the key to understanding
emotions is to understand the role fictive events play to the
viewer. Tan identifies two types of emotion at play in the
viewing situation: fiction emotions (F emotions, emphatic
emotions) and artefact emotions (A emotions, non-empathic
emotions). Empathetic (F) emotions are emotional reactions
to the story; they require empathy with the main character.
Non-empathetic artefact emotions are linked to sensory
pleasures – the viewer enjoys the good looks of the
protagonist or the beautiful scenery. [27]
Non-empathetic emotions require no appreciation of events.
The mechanism by which emphatic emotions are created is
21
more complicated. Tan suggest the film viewer connects to
the fiction is through a witness position. This position of
agreed-upon inaction (even in the most anxious of moments,
we are content to just watch the film) dictates the relations
between viewer and film events. Our commitment to the
protagonist's cause gives rise to emphatic relations. However,
the inactivity dictated by the viewing position also creates
tensions, such as worry or frustration when we know more
than the protagonist, but cannot change the situation. On the
other hand, inactivity may allow us to more deeply
empathize, since no action is asked of us.
2.3 Emotions in Games
The interaction in games usually has the player controlling a
main character (or other objects in the game). Whereas
cognitive appraisal of game events lends itself to an
emotional impact, a key question in games is how the
narrative reality is maintained. Especially, games remove the
passive viewing position, previously suggested a main
contributor of empathic effect in film. Also, the typical
structure of games – featuring repetitive action, often only
loosely bound together by a story – is challenging to
empathetic emotion.
On the surface, it would seem that the active nature of
players in games break the passivity that Tan proposes so
significant in relation to justification of emotions. However,
it turns out the activity of games lends itself rather well to the
purposes of explaining emotion and the cognitive appraisal
model has also been invoked in relation to computer games
and emotions [21][15]. Activity, while challenging the
passive ride of empathetic emotion, provides an alternative
frame in which to evaluate actions.
Perron [21] explicitly works from Tan’s theory to add
gameplay emotions (he calls them G emotions), that arise
from the cognitive appraisal of game situations. Perron’s G
emotions arise just because, in games, the player is invested
in acting. Emotional evaluation is fuelled by care for the
progress of the game. Lankoski [15] offers a detailed
breakdown of emotions within the gaming context,
demonstrating how different goal evaluations give rise to
basic emotions and their combinations.
In games with a protagonist, gameplay emotions may equal
care for the protagonist, but this care is essentially different
from empathetic emotion: From the perspective of gameplay,
the protagonist is a means, a tool, for playing the game (and
achieving the personal goal of completing the task).
Whereas the protagonist can also provide a vessel for
empathetic emotions, to some extent, the two frameworks for
emotional evaluation - empathetic and gameplay - are
competing. As Perron [21] notes, even in games, both the
main story line as well as individual plot elements and
narrative turning points are indeed furthered from time to
time through filmic means, using stills, predetermined
animation sequences/ dialogue or cut scenes. During these
moments, the player is stripped of control and, effectively,
reduced into a witness position. Empathetic emotion comes
with loss of activity. A similar point is made by Lankoski
[15] when he suggests the empathetic capability of the player
is inversely related to the cognitive challenge of action - in
the heat of a battle, there is little time to ponder the
protagonist's feelings.
3 Roles of Sound in Emotional Appraisal
The appraisal theory would seem to give little explicit
explanation of how sound contributes to emotion. The
question is to what extent sound influences the process of
appraisal. It seems clear that for both film and games, sound
is part of the process enforcing the inherent emotionality of
events as appraised within each framework. However, the
practical and functional approach to sound design is different.
In film, sound makes actions seem real and consequential for
the viewer, both factors mentioned as prerequisite to
empathetic or F emotions [27]. Within this category of
emotions, two specific cases can be identified for sound.
One goal of film (and game) production is to make things on
screen seem real. A silent two-dimensional representation
has limited apparent reality on its own, but adding sound
helps perceiving the pictures on screen as physical bodies.
Especially important in this aspect is the use of Foley, or
synchronized sound effects.
As another case of supporting realism, an integral role for
sound at least in classical (Hollywood) film production has
been to hide the medium. An example of such is the common
practise of continuing sounds over cuts, making them less
apparently noticeable. Many other conventions of film sound
also contribute to the invisible medium effect, either directly
or indirectly. Thus, a role of sound is to create a sense of
immediacy.
In the context of games, the significant emotional investment
goes into advancing goal-related progress. Most current
games rely on both auditory and visual content to represent
the game world. Crucially, in the context of interaction,
sound comes to take on a new task: facilitator and
confirmatory of action. Historically, technical considerations
have long dictated tools for sound expression in interactive
context, which has forced (and allowed) game sound to
deviate from some film sound conventions. The role for
sound in games is at least partly dictated by a functional
approach and a sound's impact defined by its capability of
supporting and facilitating gameplay.
Finally, in discussing empathic emotions we mentioned that
some appraisals require no cognitive investment in the story,
but are linked purely to sensory pleasures. Interestingly, there
is no explicit mention of negative affect in the context of
artefact emotions. The same artefact emotions are present in
games as well, where part of the game can be enjoyed (or
not) apart from aspects of gameplay-related progress.
Soundtrack CD sales gives strong evidence that artefact
emotions functions with regards to both film and game
music. Beautiful pieces of music obviously give themselves
to being appreciated as such, disregarding of whether the
viewer is attending to the story. Further, it is motivated to
broaden the spectrum of artefact emotions to include also
negative effects. With thought to how visual material is used
for shock effects (e.g. displaying blood and entrails), it is
easy to imagine a similar process in which unpleasant
sounds, regardless of story, could produce negative affect.
Both in film and games, sound provokes sensory pleasure
and displeasure.
3.1 The Realism Fallacy
The above categories identify the main roles of sound in
creating and steering the emotional experience. However, as
22
we shall soon see, there are unresolved contradictions hiding Inferences” [31]. This view is further fortified by subsequent
within these categories. Namely, the effects within the two results from experimental psychology, leading researchers to
latter categories of sound (gameplay and non-empathetic) suggest that there exists such a thing as unconscious emotion
seem to contradict the traditional sound design goal of [30]. For example, Öhman [32] has demonstrated fear
narrative realism. reactions in people who were presented with spider and snake
pictures subconsciously; that is, people became frightened of
Appraisal theory of emotion holds that empathetic emotional pictures they never even realized they had seen.
processing and value judgement is guided by conscious
attention to a story, and this cognitive investment is In light of unconscious value judgements, it is easy to read
heightened in the perceived realism of portrayal. On the other Tan's non-empathic emotions are but one example of these:
hand, it allows emotional experiences that arise from the sounds of the film are invoking emotion by nature of their
appreciation of the artefact: the pictures and sounds of the perceptual properties, unrelated to story at hand. This also
film/game as such. Similar appreciation is present in games, a opens up a way to understand and predict what would result
medium where technological artistry is elaborately in a positive (or negative, for that matter) value judgement.
showcased, often even used in promotion. One especially interesting factor is the importance of
familiarity and perceptual fluency for eliciting positive value
Tan’s view is that artefact emotions detach the viewer from judgements [23]. By this account, beauty is defined by the
the story, drawing attention away from the narrative towards ease in which a stimulus can be processed.
the film as artefact, thus making the actions within the
narrative less consequential for the viewer. This position 3.3 Misattribution and Making Sense of
raises a complicated question, namely how to interpret such Emotions
sounds that appear to transgress the borders of realism,
Affective responses include paraphernalia of bodily
despite experientially supporting narrative.
responses (pounding heart, sweaty palms) and may also bring
with them a certain action tendency (e.g. fight-or-flight). In
For example, while discussing the effect of sentiment, Tan
fact, it has been suggested that one possible function in which
and Frijda [28] 62] mention sound, especially orchestral
we do consciously attend to our affective processes is by
music, as one possible source for the awe-inspiring. Awe
appreciation of the abrupt changes in the felt background
requires a sense of overwhelming power, and this role is
state, or what Russell [24] has called core affect. This change
partly to be played by sound. The effect, according to Tan
would lead us seeking for a cause of our altered state, leading
and Frijda, is the emotional function of total submission, a
us via cognitive process to attribute the jolt to the most
feeling underlying e.g. crying. Sound is thus a tool in
plausible event in our environment.
portraying power, and heightening sentiment. The question
is, by which channel this non-empathetic heightening of
Whether or not we are willing to accept unconscious affect as
sentiment is capable of influencing the (empathetic)
the source of emotions, we can agree that when consciously
evaluation of narrative events.
attended to, emotions tend to have an object. Emotions are
evaluations of something. To be able to function properly, we
The problem of border transgression is most apparent in two
must be able to determine an object for our emotions,
sound conventions that would seem to deviate from the
something to be afraid of, or pleased by. This distinguishes
purpose of reality. In what seems like a blatant contradiction,
emotion from moods, which are long-term affective states
they invoke a sense of realism in highly unrealistic sounds.
without an object. However, and here comes the catch, the
One is the use of musical scoring. The other is the use of
events we cognitively allocate, as objects do not necessarily
sound effects that mismatch what is seen on screen. Similar
have to be the true cause of our initial affective response. In
breaches are omnipresent in games, as well, where sound
fact, when it comes to reasoning why we feel like we do, we
elements effortlessly transgress borders, allowing objects
are prone to make mis-attributions and erroneously
within the story world (diegesis) to refer to non-diegetic
appreciate our affects quite differently from their real causes
space, and vice versa [8][12].
even in everyday life.
The above concerns are about where to draw the line of
A classical example of misattribution is a study by Schachter
realism and about how emotional effects communicate across
and Singer [26]. They injected subjects with doses of
different categories of judgement. Upon closer scrutiny, both
adrenaline, a hormone associated with an excited body state.
questions generalize to the way non-empathetic affect
Depending on the situation that followed the injection,
influences other sources of emotion (empathetic or
subjects judged their aroused state as either anger or elation.
gameplay- related appraisal). To proceed further, we need an
It can be argued that in most cases, our appreciation of our
explanation on how non-empathetic emotions arise, and a
emotional state is at least partly determined by context.
way of predicting when and how artefact emotions lend
emotional meaning to other evaluative processes.
3.4 Misattribution and Emotion in Film and
Games
3.2 Unconscious Affective Processes
Misattribution is a process wherein the contextual appraisal
The theoretical frameworks dealt with above have considered
of perceived emotional ‘raw material’ lending emotional
affect by means of perceived experience. Nevertheless, it
meaning to an outside cause, irrelevant to that particular
seems many of the associations and effects of sound in both
emotional stir. Now consider a similar process at work during
film and games are working on an unconscious level.
film viewing or when playing a game, with music, sounds,
pictures, actions, all mingling to create emotional impacts. Is
Several findings state to the fact that at least some
it not probable, that at some point, the true causes of our
evaluations of stimuli are made precognitively. Zajonc was
feelings might remain oblivious to us? Is it not possible, that
among the first to point this out in his essay, famously
we, stirred by our passions, unwittingly, in deciphering the
entitled “Feeling and Thinking: Preferences need no

23
cause of our emotions take them to be caused by whatever
the film serves to us on a silver(screen) plate? Could it be
that we just happen to attend to a game event, and assume our
emotions are caused by that event, when in fact they are not?
This is, essentially, what Annabel Cohen proposes. Cohen
[5][6] has dealt extensively with the difficult question why,
and how, something as obviously constructed as the film
score, does not completely destroy the sense of realism in a
film. On the contrary, as many composers will confirm, a
carefully chosen (or composed) piece of music will actually
heighten the sense of reality in a film. Music also seems to
lend a great deal of emotion to events, in ways other than
proposed by the cognitive appraisal theory. Cohen's answer
lies in a congruence-associationist model of film viewing
[18], whereby music focuses attention on those objects in the
film that are congruent with the sound. At the same time, the
conscious attention is directed away from non-associated
sounds, and attending suppressed for stimuli irrelevant for
ongoing cognitive processes.
The emotional impact comes from the fact that sounds, even
when unattended to, will nevertheless affect perception of
objects in the film. Cohen [6] highlights the importance of
temporal unity as a binding factor and predictor of which
parts of the sound will draw attention. She calls our attention
to animation and the technique called mickey-mousing,
whereby sound effects are replaced by short musical motifs.
Their temporal matching allows these music snippets to
replace the original sounds of the events, at the same imbuing
both the events, and the objects part of the action, with
specific characteristics.
The account on musical meaning in film provides an equally
useful tool for approaching the question about other film
sounds as well. Applied to object sound, the theory suddenly
appears much less mysterious: Consider how we find out
properties of objects in real life. What we do is handle the
object – tap it, stroke it, bang it against something. By
perceiving synchronic sounds, we find out the normal sound
of a chair, a balloon or a mandolin. Now, turn that process
around and we have precisely what sounds do in a film (and,
to a great extent in games as well): now the temporal unity of
event and sound defines the object through what sound it
makes. Longer chains of events, if temporally matched, cause
similar perceptions. When approached from this angle, it is
not so odd that sounds in fiction may deviate somewhat from
their real life counterparts without seeming false or
unrealistic. What may seem surprising is that a whole sum of
temporally congruent sounds may become involved in the
same process, from simple Foley through more elaborate
layers of sound effects all the way to music.
3.5 Where do The Emotions Come From?
So far we have established that unconscious emotional
processes may ‘contaminate’ temporally congruent events
through misattribution and shown how this may influence
perception of events in film. The big question remains, where
do the emotions come from?
The most frequently researched category of emotional sound
is music. Huron [10] describes musical emotions in terms of
fulfilling expectations: the interplay between anticipated and
sounded music progression creates patterns of dynamic
tension and relaxation. Musical expectations arise from
several sources, most of them cultural, but according to some
studies at least part of the functions of musical meaning
appear to be universal [19].
The existence of universal function of musical emotion
suggests there may be other sources of emotion humans draw
from in their interpretations of music. The obvious case is
that music invokes memories and connotations awaken by
that music. However, those would not be universal, even less
so than culturally learned expectations. Van Leuwen [17]
turns to the human body, suggesting that the most primitive,
and also a common link between sound and emotion for all
humans, is the perception of our own bodies. Especially the
vocal system sets a reference point through simultaneous
experience of how it feels and what it requires to produce a
certain sound.
Another suggestion is that evaluation of some sounds has
biological motivation. This appears to be the fact with the
startle response (the phenomenon in which we jump to
someone shouting ‘boo’ at us), which aside from providing
for pranks also makes us more alert for dangers and
automatically directs our attention toward potentially harmful
events. However, there may well be other ways in which our
perception of sound is evolutionary determined. For example
Huron [11] suggests that the perceived cuteness of sounds
may be an evolutionary adaptation that promotes parenting.
Value judgements (which are the raw stuff of emotion) seem
to be going on even on the lowest level of perception. A
classical example within music (and other perceptual)
research is the mere exposure effect, wherein a stimulus is
judged as likeable merely as the function of familiarity.
Investigations into a phenomenon called perceptual fluency
suggest that emotional processes are influenced by the very
ease of processing [23]. These findings would imply that
such things as differences in perceptual clarity (think signal-
to-noise ratio) of audio influence the emotional impact of
sound, such as perceived beauty or likeability.
The critical requirement for musical expectations to arise is
that it is attended to as music. Further, there appears to be
boundaries in our listening schemes that separate different
styles of listening. Notably, musical listening, in which
sounds are perceived as sounds, is not the only form of
attending to events. An illustrative deviation from this frame
are listening styles provoked by compositional techniques
invoking other listening styles, like musique concrète, where
the use of real world sounds provokes listening, not at sounds
and patterns, but for causes – Chion aptly refers to this as
causal listening [4]. This is a special case of music, perhaps
seldom used in film, but appearing more and more in games.
In these cases, the framework of listening is perhaps more
determined by evolutionary and low-level perceptual
processes of meaning-making than musical listening modes.
3.6 Realism Revisited
We should now attempt a new understanding of realism.
Within fiction, realism is not an absolute, but a nominator for
a certain level of fit, an apparent realism or credibility. On
the narrative level, realism allows taking the story seriously
enough to allow emotions of empathetic quality. Good fit is
determined by whether the sound is credible (or illustrative)
of a certain sound source [1], pp 190]. When the Foley artist
(the person responsible for creating sounds to on-screen
events) smashes pumpkins in his studio, he does so in order
to produce such sounds with good fit with on screen events.
Many times, the sounds produced have little to do with the
24
actual event seen on the film screen – indeed, often non-
realistic sounds are purposefully used to make the action
sound better. It is, for example, recognized that walking on
cornstarch sounds much 'more real' on film than the actual
sounds of walking on snow.
A possible explanation underlying the perceived realism of
some Foley sounds is the notion of prototypicality. A
prototype is an object that inhabits central perceptual
characteristics of a given category. Prototypes do not
necessarily exist in reality, they are mental constructs of our
perceptual system. The prototypical chair is the average of all
chair perceptions of your brain, and by definition, it will be
the 'chairest' chair of them all. Experimental psychology has
established that people perceive prototypes as more easily
recognized [27], also more beautiful, and trustworthy [23]
than other category members.
Similarly, narrative reality determines how sound behaves
within the diegesis, and how the source sounds should sound
when listened to from different Points of Audition1, such as
listening behind a wall or under water. At the core, then, also
immediacy is but one way of creating a sense of realism. In
film, it serves to reinforce the witness position, being present
but out of control. Tan [27], pp 25] considers this in his
analysis of the camera point of view, mentioning how even in
first person view the camera is often a bit off, making space
for someone to ‘look over the shoulder’. In the case of sound,
the heightened, focussed sound including over-clear dialogue
can be considered realistic if we view it as a portrayal not of
the scene, but of the experience of listening to the scene.
Consider this: while our environment usually contains a
multitude of sounds, we only attend to a select few at a time.
We are also exemplary at picking out and following these
sounds. A person with normal hearing has no difficulty in
following a single conversation in a room filled with people,
the phenomenon so aptly named the ‘cocktail-party’ effect.
Thus, what would seem presented as the films sounds is not
the scene itself from a given point in space, but the scene as
heard if listened to by attentive ears.
The narrative realism of a sound is thus not in faithful
reproduction of sound sources, nor of their environments.
The apparent realism of a sound in the context of narrative is
defined by how representative a sound is of a certain event.
Sounds that are highly representative have good narrative
fit. High narrative fit supports empathetic emotion.
Importantly, however, the evaluation of sounds spans several
layers. Below the level of narrative meaning are (partly)
unconscious processes whereby sounds are judged
emotionally. For example, a sound can have good fit
narratively, but poor legibility, because the signal-to-noise
ratio is so high. Importantly, breaches in this level are
disruptive to the perception of sound. We have seen that
perceptual fluency is also capable of influencing affective
evaluations [23]. Thus, as with the narrative fit, unconscious
processing of sound influences emotional judgements. These
affects are unrelated to the narrative content of the sound, but
tap into the notion of artefact emotions. Depending on their
nature, they can cause pleasure or displeasure, which can
then be attributed to other temporally congruent events.
1 http://www.gamasutra.com/features/20070322/bridgett_
Similar to Point of View in camera techniques, but
using sound.
Finally, in interactive systems interpretations of sound take
on a new role, conveying functional information [29]. By this
task, sounds are evaluated on a third level, in how well they
serve a functional value, Jørgensen [12], pp 49] refers to this
as a sounds functional fidelity. On this stage, emotional
evaluations are no longer determined only by the sound itself,
but by the utility of a sound for the higher goal of performing
goal-related actions. The value of functional sound depends
on how the functional aspect supports game progress, the
utility of sound. The utility of sound is connected to goal-
related cognitive evaluations. High utility enforces gameplay
emotion.
4 Comparison of Emotional Sound in Film
and Games
The cognitive appraisal framework provides two alternatives
for emotional relation to fictive events: the passive witness
position allows empathetic emotion, while the active player
draws emotional meaning from goal-related evaluation.
These two frameworks ask the viewer/gamer to take on a
different attitudes towards the fiction and appear to be
competing. They also rely on different strategies for sound
design.
For film sound, the effort is usually on heightening narrative
reality. This is achieved through detailed attention to
narrative fit, often striving for high apparent reality. Focus is
on advancing the narrative, heightening and clarifying those
specific actions that are necessary for following the story’s
progress (usually the top priority is on dialogue).
For games, the focus point is different, as games have to
support player action. In games, the task of many sounds is
primarily to provide feedback about actions. Hence, narrative
fit is often sacrificed for utility. To the extent that auditory
cues are used to guide actions, they are treated with utmost
respect for legibility. For example, even in the case of
instructions with diegetic source (a non-player character,
voice mail, etc.) it is common that auditory instructions
remain heard even if the character runs away from their
diegetic source.
An interesting avenue for sound design in games is to shift
focus from music to the emotional impacts of Foley and
sound effects. A possible alternative for emotionality in
games is in environmental sounds, which is already used in
many games, where ambient sounds are beautifully merged
with musically suggestive elements and event sounds into a
sonic landscape in the spirit of musique concrète. However,
for this approach to be systematically explored, there is need
for better understanding of how everyday sounds influence
emotions. In these investigations, theories of unconscious
emotion may prove especially informative.
References
[1] Bordwell, D. and Thompson, K. 1985. Fundamental
Aesthetics of Sound in the Cinema. In Weis, e. and
Belton, J. (eds.) Film Sound, Theory and Practice.
Columbia University Press. 181-199.
[2] Brandon, A. 2005. Audio For Games. Planning, Process
and Production. New Riders.
[3] Bridgett, B. 2007. Audio Postmortem: Scarface: The
World is Yours. Available at
pfv.htm [Accessed 29.5.2008]
25
[4] Chion, M. 1994. Audio-Vision. Sound on Screen.
(Translation by Claudia Gorbman). Columbia University
Press.
[5] Cohen, A. 1990. Understanding musical soundtracks.
Empirical Studies of the Arts 8, 111-124.
[6] Cohen, A. 2001. Music as the source of Emotion in
Film. In Juslin, P. and Sloboda, J (eds.) Music and
Emotion. Oxford University Press. 249-272.
[7] Damasio, A. 2005. Descartes' Error: Emotion, Reason,
and the Human Brain. Penguin, paperback reprint
(1994).
[8] Ekman, I. 2005. Understanding Sound Effects in
Computer Games In Proc. Digital Arts and Cultures
2005, Kopenhagen, Denmark.
[9] Frijda, N. H. 1986. The emotions. Cambridge University.
[10] Huron, D. 2007. Sweet Anticipation: Music and the
Psychology of Expectation. MIT press. Paperback
reprint (2006).
[11] Huron, D. 2005. The Plural Pleasures of Music. Proc.
2004 Music and Music Science Conference. Kungliga
Musikhögskolan & KTH (Royal Institute of
Technology), 1-13.
[12] Jørgensen, K. 2007. ‘What are Those Grunts and Growls
Over There?’ Computer Game Audio and Player Action.
Ph.D. dissertation, Copenhagen University.
[13] Kutay, S. 2006. Bigger Than Big: The Game Audio
Explosion. A Guide to Great Game Sound. Available at
http://www.gamedev.net/reference/articles/article2317.a
sp [Accessed 29.5.2008]
[14] Lacan,J. 1951. Some reflections on the ego. (fut lu par
Lacan à la British Psycho-Analytical Society le 2 mai
1951) Available at: http://aejcpp.free.fr/lacan/1951-05-
02.htm [Accessed 29.5.2008]
[15] Lankoski, P. 2007. Goals, affects, and empathy in
games. Paper presented at Philosophy of Computer
Games, Reggio Emilia, Italy. Available at:
http://www.mlab.uiah.fi/~plankosk/blog/?p=53.
[Accessed 29.5.2008]
[16] Lazarus, R. 1991. Emotion and adaptation. Oxford
University Press.
[17] Leewen, T. van. 1999. Speech, Music, Sound.
Macmillan.
[18] Marshall, S. and Cohen, A. 1988. Effects of Musical
Soundtracks on Attitudes toward Animated Geometric
Figures. Music Perception 6, 95-112.
[19] Narmour, E. 1990. The analysis and cognition of basic
melodic structures: The implication-realization model.
University of Chicago Press.
[20] Oatley, K. and Jenkins, J. 1996. Understanding Emotion.
Blackwell Publishing.
[21] Perron, B. 2005. A Cognitive Psychological Approach to
Gameplay Emotions. Proc. DiGRA 2005 Conference:
Changig Views – Worlds in Play.
[22] Prince, B. Tricks and Techniques for Sound Effect
Design. Computer Game Developers Conference 1996.
Available at
http://www.gamasutra.com/features/sound_and_music/0
81997/sound_effect.htm [Accessed August 20, 2008.]
[23] Reber, R.; Schwarz, N. and Winielman, P. 2004.
Processing Fluency and Aesthetic Pleasure: Is Beauty in
the Perceiver's Processing Experience? Personality and
Social Psychology Review, 8 (4). 364-382.
[24] Russell, J. 2003. Core Affect and the Psychological
Construction of Emotion. Psychological Review 110 (1),
145-172.
[25] Sanger, G. 2003. The Fat Man on Game Audio: Tasty
Morsels of Sonic Goodness. New Riders.
[26] Schachter, S., & Singer, J. 1962. Cognitive, Social, and
Physiological Determinants of Emotional State.
Psychological Review, 69, 379-399.
[27] Tan, E. 1994. Film-induced affect as a witness emotion.
Poetics 23, 7-32.
[28] Tan, E. and Frijda, N. 1999. Sentiment in Film Viewing.
In Plantinga, C. and Smith, G. (eds.) Passionate Views.
Film, Cognition, and Emotion. Johns Hopkins. 48-64.
[29] Tuuri, K.; Mustonen, M.-S.; Pirhonen, A. 2007. "Same
sound - Different meanings: A novel scheme for modes
of listening." Proc. AudioMostly 2007, Ilmenau,
Germany.
[30] Winkielman, P. and Berridge, K. 2004. Unconscious
Emotion. Current Directions in Psychological Science,
13 (3). 120-123.
[31] Zajonc, R. B. 1980. Feeling and Thinking: Preferences
Need No Inferences. American Psychologist, 35, 151-
175.
[32] Öhman, A. 2005. The role of the amygdala in human
fear: Automatic detection of threat.
Psychoneuroendocrinology 30, 953-958.
26
 Interactive Sonification of Grid-based Games
Louise Valgerður Nickerson and Thomas Hermann
1 2

1
Interaction, Media and Communication, Department of Computer Science,
Queen Mary, University of London, London, U.K.
lou@dcs.qmul.ac.uk
2
Ambient Intelligence Group, Cognitive Interaction Technology · Excellence Center (CITEC)
Bielefeld University, Bielefeld, Germany
thermann@techfak.uni-bielefeld.de

Abstract. This paper presents novel designs for the sonication (auditory representation) of data from
grid-based games such as Connect Four, Sudoku and others, motivated by the search for eective auditory
representations that are useful for visually-impaired users as well as to support overviews in case that the visual
sense is already otherwise allocated. Grid-based games are ideal to develop sonication strategies since they
oer the advantage of providing an excellent test environment to evaluate the designs by measuring details of
the interaction, learning, performance of the users, etc. We present in detail two new playable sonication-
based audio games, and nally discuss how the approaches might generalise to general grid-based interactive
exploration, e.g. for spreadsheet data.

1 Introduction only a limited number possible elements of a nite set

Sonication, the auditory representation of data, has
become an important sensory channel for rapid data
scanning, real-time monitoring and exploratory data
analysis [6]. Particularly if the data is structured in
time (e.g. time series, process data), sonication is a
good choice in order to communicate the patterns by
using the auditory modality. However, a very frequent
data type consists of two-dimensional grids or matrices
of data. In fact, most data sets which are subject to the
analysis in data mining can be re-organised to take this
form, using columns for features and rows for measure-
ment vectors. Images are also naturally represented by
a 2D-grid of measured intensity values. Spreadsheets
are another frequent example of grid-based data. Thus
it makes sense to investigate how to make such data
more accessible by using sonication, or how sound
can be used eectively to deliver a concise overview
or summary of the data.
However, the sort of overview needed depends highly
on the task, and often dierent task-specic overviews
are needed, ranging from overviews that give a rough
idea how a grid is lled to very-specic overviews such
as `what cells form groups with a particular pattern',
row-wise scans, diagonally aligned patterns, symme-
tries within the grid, etc.
Grid-based games are a special case of grid data in
the sense that usually the grid dimension is xed and
ll a grid cell. Examples of grid-based games are chess,
Chinese checkers, Connect Four, noughts and crosses
(or tic-tac-toe), Sudoku, to name a few.
We develop our sonication approaches at hand of
grid-based games for the following reasons: (a) there
is a very clear task for the players, yet (b) there is
a sucient variety of required overviews so that the
task is not trivial, (c) the limited complexity facilitates
the designs, and (d) the game itself provides a very
useful test environment to evaluate all aspects of the
design, from performance and learning to the æsthetics
(acceptance, qualitative evaluation).
1.1 Sonifying grid data
When designing grid-based game sonications, a deci-
sion has to be made whether the sonication shall be
generic in the sense that it is applicable to a wide class
of games, or specic for a particular game. Generic ap-
proaches generalise better towards a more wide-spread
use, maybe even beyond the scope of grid-based games
into tasks such as video data sonication, however,
they may not deliver exactly the information that the
players need to play the game, or allow to extract these
patterns only after longer training.
A mix of sonication techniques that oer both spe-
cic and inspection general seems suitable, and puts
into the fore that the users will need control over what

27
Interactive Sonication of Grid-based Games
sonication is to be selected and what parts of the grid
to be explored. Indeed, interaction plays an important
role in inspecting grids, as can be seen also in visual
exploration of grid games, where eye-movements, xa-
tions and saccades are naturally used to serialise and
access the information. In a similar fashion we believe
that manual interaction is an important (if not the key)
ingredient to create successful designs. We present for
instance a graphics-tablet based sonication approach
where proprioceptive information serves the intuitive
understanding of the position in the grid whereas sound
conveys the information about the grid content at hand
of the 4×4 Sudoku in section 3. Dierent exploration
strategies emerge from such an approach.
To couple interaction to plausible acoustic responses,
we use ideas from Model-Based Sonication [2, 4]
(MBS). MBS describes how to use excitatory systems
in order to create informative sound as result of pro-
cesses where the user's interaction puts energy into
an data-driven sound-capable system. Even without
creating a coherent sonication model, MBS might be
helpful to create designs that are more intuitively un-
derstood.
A key problem in grid-based game sonication is the
missing persistence of the grid, as opposed to the per-
sistent visual game board. To create a close analogy to
the visual task of adding visual elements on a board,
an auditory version can use a stationary sound pat-
tern which is permanently played, allowing players to
add sound elements accordingly. Conditions to win a
game translate to corresponding auditory conditions
within the sound. This analogy might open a window
to the design of very interesting new audio games, how-
ever, we here keep the focus on grid-based games, and
thereby translate the analogy into a rhythmical soni-
cation strategy where, instead of a stationary sound,
a repetitive sound pattern is created, which can be re-
garded as one bar in a repeating sonic loop. We de-
velop this idea into a playable version of Connect Four
in section 4.
We discuss our ideas via qualitative experiments
with a limited set of users, since we are still within
the design phase towards stable sonications, and we
close the paper with outlooks on our ongoing work.
2 Background
In the visual realm, space is used to make salient in-
formation of interest. In the case of grid-based games,
it organises the items on the grid so that the players
can easily make sense of the state of the game. This
is also true of data that is visually displayed in tab-
ular format: it makes correlations between two axes
clear. We can use grid games to represent tasks that
one might perform with grid-organised data, such as
in a spreadsheet. Connect Four can represent looking
for linear patterns in data while Sudoku can represent
cross-correlating subsets of data.
2.1 Traditional methods
A grid-based representation that often gets tackled is
the auditory representation of images. This is tradi-
tionally done via scanlines. Examples of this can be
seen in representations of images where each pixel value
is mapped to sound and played in order. More ad-
vanced techniques involve nding textures in the image
to represent in sound. The diculty with the scanline
approach is the challenge of lining up the rows so that
one can understand patterns that are orthogonal to
the direction of the scanline. The pattern and audi-
tory texture approach is much closer to what we try to
accomplish here with our implementation of Connect
Four.
Other pertinent work is research into the sonica-
tion of spreadsheet or tabular data. Stockman, Hind
and Frauenberger [7] describes a system where visually-
impaired users can navigate spreadsheet data by map-
ping numerical values to a range of pitches. The data
is then played serially by row or column. This is meant
for generic use; our approach is to look to the specic
to inform the generic. Kildal and Brewster [5] describe
a method of providing overviews of numerical data in
tables by again mapping values to pitches. Here, rows
and columns are presented concurrently giving the user
quick access to where the highest and lowest values are
to be found. The idea of concurrency is one we apply
to our implementation of Connect Four.
2.2 Connect Four versus Sudoku
One can generalise grids as M × N grids with a set of
k potential token values. Connect Four is a 7 × 6 grid
with three token types (one for each player as well as
the `empty cell' item). Sudoku is a n2 × n2 grid with
n2 + 1 tokens (n2 tokens and the `empty cell' item).
The most common variant of Sudoku is where n = 3
or the 9 × 9 grid.
There are several dierences between the games and
their grids. Connect Four is a two player game while
Sudoku is a single player game. Another dierence is
that in order to win Connect Four, a pattern of four
tokens in a line must be achieved while in Sudoku the
tokens must be uniformly distributed. In both games
the grids get lled one move at a time. In Connect
Four at the end of each pair of turns there are an equal
number of each token in the grid while in Sudoku this
condition is only properly achieved when the puzzle is
completely solved.
-2-
28
Interactive Sonication of Grid-based Games
Another key dierence is that in Connect Four, when
tokens are added to the grid, they are placed in the low-
est unlled cell in the selected column while in Sudoku,
cells can be lled in any order. The playing of Connect
Four is what drives the dominant features of the soni-
cation described in section 4. The columns are primary
as their state is what informs the players where tokens
may be placed and the rows are secondary as they de-
scribe the end position of the token placed in a par-
ticular column. Sudoku is less straight-forward: it is
the structure of the grid and the rules of the game that
are important. Neither the rows, columns or cages (see
gure 1) are dominant but rather they must be inter-
compared so that a player can deduce the value that
belongs in each cell.
3 Model-Based Sonification for Sudoku
Sudoku is a single player game where a player must ll
all the cells on the grid so that the values in each row,
column and cage are unique. The most common form
of Sudoku is a 9×9 grid, shown in gure 1. The grid
is further subdivided into nine 3×3 sub-grids, called
cages. Here, we implement an easier version: 4×4 Su-
doku.
Figure 1: The Sudoku grid. The 9 × 9 Sudoku grid is
made up of nine 3 × 3 sub-grids, called cages. Cage
rows/columns are horizontal/vertical sets of cages.
3.1 Design and implementation
In the 4×4 grid, there are ve possible values for each
cell: the four tokens and `empty'. Each grid has a cer-
tain number of cells that have pre-lled-in values. In
order to play the game, players need to cross-reference
rows, columns and cages in order to deduce what val-
ues go into the empty cells. Key information for solv-
ing includes where the grid is dense/sparse and where
all the items of the same values are present/missing
in a row/column/cage/cage row/cage column. To en-
able exible self-directed exploration  much as a you
would get from glancing  we use a graphics tablet for
interaction (see gure 2). We also employ MBS to pro-
vide contextual information to the player. Sonication
examples are provided at [1].
(1,4)
(1,1)
Sudoku play area
(4,4)
(4,1)
Figure 2: Playing Sudoku on the Wacom graphics
tablet. Post-it notes dened a square play area.
3.1.1 Representation of the grid
The use of the graphics tablet means that we do not
need to provide strong location information. As a re-
sult, the grid is not explicitly represented in sound ex-
cept in the MBS that we use when a player probes a
cell of the grid. We use a standard energy ow model
as introduced in [2] to describe the eect of each cell
upon its neighbours:
dEij X
= −λEij + q · (Ekl − Eij ) (1)
dt
(k,l)∈N (i,j)
where q represents the energy ow rate between neigh-
bour cells and λ is the energy loss or the decay of
the energy. ij denotes the co-ordinates of the cell and
N (i, j) is the set of all cells that neighbour ij .
Figure 3: The Sudoku ow model When a cell is ex-
cited, the energy ows into the neighbouring cells as
described in equation 1.
3.1.2 Representation of the cell values
The four values in the grid are represented by pitch.
The pitches range evenly from MIDI note 64 to MIDI
note 80. When a cell is empty there is a white noise
sound that is modulated to sound like the wind. Our
initial design had empty cells represented by the lowest
pitch, however only players with musical training found
this comprehensible. We used SuperCollider3 for all
sonications here, using PlayBuf as sample player and
-3-
29
Interactive Sonication of Grid-based Games
standard techniques for panning and ltering of sound.
More information will be provided on our website at [1].
3.2 Playing the game
Players a stylus to explore the grid and enter values
on the graphics tablet. There is also a graphical repre-
sentation of the grid (see gure 4) which provides the
limits of the cages. To probe the grid, players either
tap or drag the stylus across the grid. When the sty-
lus enters a cell, the cell is injected with energy (as
described in equation 1) and the energy ows through
the grid. To enter a value, players click a button on the
stylus. Each click cycles the current value of the cell
to the next value. If a cell is a starting value, nothing
happens.
Figure 4: The graphical interface for Sudoku. Players
interact with the grid using the graphics tablet stylus.
3.3 Discussion
5 men and 2 women played the auditory version of 4×4
Sudoku, two of whom were musicians and one of whom
was visually impaired. Their level of experience with
playing Sudoku ranged from beginner to advanced.
3.3.1 Player feedback
Feedback for Model-Based Sudoku was varied. We pre-
sume that is this partially due to it being a single player
puzzle game (a two-player game, on the other hand, en-
gages the players competitiveness and allows them to
learn from one another). The general consensus was
that while 4 × 4 Sudoku is quite simple visually, the
auditory version was quite challenging and the smaller
versions was approximately the right level of diculty.
Here are some of the more specic ndings:
First try First attempts were often frustrating, some-
times resulting in starting over. This indi-
cates that dierent initial solving techniques are
needed: mapping out grid density rather than lo-
cation of similar items. Second games were much
smoother.
Tapping vs dragging We assumed that the major-
ity of interaction would be by dragging. However,
the majority of players (Players 2, 4, 5 and 7) pre-
ferred to tap the cells to excite the grid. Player 2
explained this by saying that the sounds made by
the model made this tapping interaction more in-
tuitive. Another explanation is that players were
tapping in order to compare only two values at a
time.
Draggers For the players who dragged more than
they tapped, the stylus and tablet interaction al-
lowed them to quickly scan a row, column or cage
by drawing lines or circles in the grid. These play-
ers appeared to be the fastest at completing puz-
zles. We anticipate that this is because a quick
scan allowed players to quickly determine which
tone was missing or if there were two tones of the
same value in the row/column/cage.
Panning One surprise was that both Players 2 and
5 (both tappers instead of draggers) found that
the panning was not helpful and in fact was dis-
tracting and made it harder to compare values.
This possibly indicates that the use of the graph-
ics tablet suciently localises a player and the
additional cues are unnecessary.
Based the two dierent ways of interacting with the
grid (dragging vs tapping), we expect that a better
tting model will need to be devised to make it more
natural for the dragging technique to be used. The
faster interaction allows for the patterns that occur
to be more quickly absorbed. With a more intuitive
model, players can more naturally take advantage of
the way we process audio.
3.3.2 Informing grid sonifications
Much like with how direct manipulation and the in-
troduction of the mouse revolutionised graphical user
interfaces, the use of the tablet enables the user to a
greater degree than keyboard interaction. The tablet
interaction contributed more to the success of the
4 × 4 Sudoku than the use of Model-Based Sonica-
tion. With a suciently small grid so that the number
of values is not overwhelming, stylus interaction gives
the user the exibility to explore the grid as they de-
sire and provides speed that is dicult to mimic with
traditional keyboard or 5-way navigation (such as on a
mobile phone or a game controller). It also neutralises
the problem of strongly localising a user in the grid
through sound.
3.4 Scaling up to 9×9 Sudoku
The 4 × 4 implementation of Sudoku does not scale up
well to 9 × 9 Sudoku. The main problem is that there
-4-
30
Interactive Sonication of Grid-based Games
are simply too many values to remember. We require
a new model that lends itself better to the larger grid.
The main concepts here are generating models that can
support the cross-hatching technique  where players
cross-correlate values in rows, columns and cages to
deduce values  and also categories and order the values
used in the grid. The problems that occur in the 9 × 9
Sudoku grid inform us how to alter the model that was
used for sonifying the 4 × 4 grid.
It is clear that we need to develop specialised
overviews and lters to allow users to focus on dier-
ent parts of the grid. Key information is about what
is present or missing and picking out items of similar
values. What this implies is that players must be able
to apply certain lters in combination as they interact
with the grid or prompt overviews to be played. How-
ever, it is also important not to lose the advantages
from the direct interaction provided by the graphics
tablet. For example, were the player interested in
an overview of a row, tapping to the left or right of
that row could play an overview of the row where to-
kens are played in a predened order using the graphi-
cal interface and panning to re-enforce their positions.
To query where a particular token is present, players
could select the token from a list and use gestures in
each cage to determine if it is present. Another lter
could be used in combination with a the token lter to
show where the token is missing. Finally a lter that
only displays where the empty cells are could highlight
where the grid is dense or sparse. To solve the prob-
lem of the large number of values to be entered, sounds
that can be vocalised can be used. This enables the
player to self-organise the tokens and these can then
also be used as input removing the necessity to make
mappings between tokens and their graphical represen-
tation. Vocal sonications have been successfully used
in the sonication of EEG data [3].
Given the complexity of the sonication and inter-
action needed, we have tabled our work on Sudoku for
the time being and are focusing on the second game we
implemented: Connect Four.
4 Rhythmic Connect Four
Connect Four, a Milton Bradley game, is a two-player
game on a 6×7 grid where each player tries to line up
four game tokens while blocking the other player from
doing the same. The traditional Connect Four grid is
shown in gure 5. The auditory version of this game is
based on adding sound events to a rhythmic pattern.
Sound examples are provided on-line at [1].
4.1 Design and implementation
The important features of Connect Four are the
columns and the locations of tokens, especially where
there are several of the same value in a line. Knowing
what is around a token is therefore very important as
well as being able to focus on each token individually.
4.1.1 Representation of the grid
We represent the grid in a short looping sound so that
players can think about the entire grid and understand
where tokens are in relation to one another. The aim is
to provide all the information quickly enough so that
the players can reason about it as a whole with the
distinct parts making up a pattern that they can work
with. The end result is that the grid is like a short bar
of music. We then punctuate this bar of music with two
drum sounds to help players localise themselves within
each loop. A stronger (or louder) drum plays at the
start of the grid and the softer (or quieter) one occurs
at the fth column of the grid. Our initial design did
not include the second drum however, it was quickly
apparent that when the grid is sparse, it did not have
the energy or liveliness for which we were aiming nor
was the localisation strong enough. This aim also drove
the rate of our auditory display. We experimented with
grid lengths of 0.7 to 3.5 seconds. Less than a second
was found to be quite manic and over two seconds a
bit too slow. Our preferred length was 1.4 seconds with
0.2 second pause between loops, coming to a total of
1.6 seconds.
Figure 5: The sonication of the Connect Four grid.
The row determines the pitch of a token and the col-
umn drives when the token plays. Two drum sounds
punctuate the auditory grid at columns 1 (louder
drum) and 5 (softer drum). There is a short pause
 the length of a column  at the end of the grid.
-5-
31
Interactive Sonication of Grid-based Games
4.1.2 Representation of the columns
The columns in the grid are evenly spaced in the audi-
tory grid loop. The values in each column are presented
concurrently. The pitch of each value is determined by
the row, where the bottom row is mapped to a low
pitch and the top row is mapped to a high pitch. The
pitch intervals used can greatly aect the æsthetics of
the auditory display. We experimented with several in-
tervals, such as `neutral' or `jazzy'. While the neutral
pitch interval allows for the greatest separation of the
notes (MIDI pitch 52, 56, 59, 61, 64, 68) 1 the jazzy in-
terval (MIDI pitch 52, 54, 57, 59, 62, 67) was the most
engaging and least irritating after many repetitions.
4.1.3 Representation of the tokens
Each token was represented by an instrument. We used
a vibraphone and an electric bass in our implementa-
tion. The pitch interval for each instrument is modu-
lated to t the instrument better. These two instru-
ments are very dierent sounding and their envelopes
are diverse making them easier to tell apart. Addi-
tionally, each token plays in a dierent speaker. These
dierences allow the players to pay attention to each
token alone.
Additionally, we use brilliance to indicate when there
are several tokens in a row. Minimum brilliance corre-
sponds to a player's token all on its own and maximum
brilliance is applied to four of a player's tokens in a line
(game is over). If the game is won, the winning com-
binations has maximum brilliance while the brilliance
of all other tokens is set to the minimum. This use
of brilliance highlights tokens that have the potential
to win the game and gives a clear indication when the
game is over.
4.2 Playing the game
The interface for playing Connect Four was graphical
(see gure 6). Each player has a slider allowing play-
ers to drop their tokens into the grid. The value of
the slider represents the columns in the grid. Each
player also has a button; until this button is pressed,
the player's move is not committed. This allows each
player to move their tokens and hear the eect of their
move before making a nal decision for the turn.
4.3 Discussion
We performed an informal evaluation with ve dier-
ent players: four men and one woman, two of whom
were musicians and one of whom was visually-impaired.
Their ages ranged from the late twenties to the mid
fties.
1 using the SuperCollider3 .midicps method
Figure 6: The graphical interface for the Connect Four
game. Each player uses their slider to select columns
and the button to enter values in the auditory grid.
4.3.1 Player feedback
The general feedback from the informal evaluation of
the game, which used the graphical interface shown in
gure 6, was quite positive. Here are some of the most
frequently mentioned topics:
Playability Most players felt that with a little prac-
tice, the game would be quite playable.
Engaging rhythms Players found the sounds æsthti-
cally pleasing. Players as well as bystanders would
nd themselves moving along with the beat.
Losing the beginning of the grid Players would
often get confused about where the grid began
in the sonication. The rst token placed in the
grid seemed to take the players' focus away from
the drum beat.
Graphical interface Several players used the posi-
tion of their opponents slider after their move to
gure out where the last token was placed. Player
2 commented that they felt this was a cheat.
Masking The higher pitched tokens overpower the
lower ones, especially in the case of the vibra-
phone. We hypothesised that this may be in part
because the players are not taking advantage of
listening to a single players tokens by listening to
one speaker at a time.
Playing patterns Players either tended to experi-
ment with several moves before committing while
others selected a column right away. It is unclear
what drives this behaviour and whether it has a
correlation to successful game play.
Interaction The slider did not always make it clear
when the player moved from one column into the
next. This was addressed by adding column de-
limiters to the interface as shown in gure 8.
4.4 Informing grid sonifications
Our rhythmic Connect Four contains several pieces of
design knowledge that can be applied to other soni-
cations of grid data.
-6-
32
Interactive Sonication of Grid-based Games

Looping through the grid The technique of making the is shown in gure 8. This allowed players more free-
grid into an auditory loop  be it column-wise as we dom in their interaction and also pushed them to rely
have done or not  shows promise for providing a grid more on the auditory feedback rather than looking at
overview. This is similar to other work in auditory where their opponent placed a token throughout their
overviews [5], but instead of the column being sonied move.
at the request of the user, it is repeated to continuously
remind the user of the state of the grid. We believe
this to be a technique that can help overcome the lack
of persistence in the auditory channel. Here, we have
implemented this technique and players of the Connect
Four game found it useful and engaging.
What remains to be tested is the limitations of
this technique. Our sonication was limited to seven
columns with a maximum of six values to represent
while most data sets encompass many more than that.
It remains to be seen whether the technique is depen-
dant on the number of columns presented or on the du-
ration of the sonication. We envisage this technique
being extended to comparing data sets as well, pro-
Figure 7: Two people playing Connect Four. The play-
vided an overview of each data set could be presented ers trade o the stylus and use areas on the tablet to
as we have presented columns here. play in a column, as show in gure 8.

Identification of a pattern Connect Four has a clear

pattern that determines if a player has won: four to-
kens of the same kind in a line, be it in rows, columns
or diagonally. We use brilliance here to highlight where
this pattern occurs in the data and where partial pat-
terns occur. This technique allows us to essentially
apply a lter to the data. In our case here, we had a
very simple pattern to match. We envisage that this
can be extended to many dierent patterns with the
potential of several patterns being applied in turn to
show dierent aspects of the data.

4.5 Formally evaluating Rhythmic Connect Four

Due to the positive user feedback from Rhythmic Con-
nect Four, we are currently taking this work forward
and have just completed a formal evaluation of the in-
terface with some minor changes. In this study 7 pairs
of players each played 3 games and were interviewed
about their experience. We focused on how they used
the audio to inform their playing strategies. The re-
sults of this evaluation will be reported in further pub-
lications however we describe preliminary results here.
To address the naturalness of the interaction and to
focus more on the auditory aspects of the game, the in-
terface was moved to the graphics tablet. This allows
two players to sit opposite each other (see gure 7) and
to divorce the sonication from any visual representa-
tion. The two main dierences that resulted from this
changes was that (1) players were not aware that the
column selection area was a slider and (2) that after se-
lecting a column, the other player could not see where
their opponent had played. The graphics tablet layout
Figure 8: The interface for the Connect Four game on
the graphics tablet. The interface is inverted, allowing
players to sit opposite each other as shown in gure 7.
The training we performed as part of the formal eval-
uation addressed several of the problem we noted ear-
lier. One such problem was losing the beginning of the
grid or sound loop. We trained players to listen for the
louder drum beat and found no indication that this was
a problem during the evaluation.
Another problem was the vibraphone instrument
overpowering the electric bass instrument. We ad-
dressed this by boosting the volume of the electric bass
so it was not so easily overpowered and by training
players to listen to a single players token at a time
(as each player has their own speaker). A problem re-
ported in this second evaluation was that the higher
pitched tones overpower the lower ones. We will look

-7-

33
Interactive Sonication of Grid-based Games
at this issue further as we complete our full analysis.
5 Conclusion
In this paper, we have presented some new approaches
for the sonication of grid-based games. These grid
games represent use cases of data displayed on a
grid allowing us to develop techniques that can be
transferred to related applications, such as real-time
video stream sonication, spreadsheet sonication for
visually-impaired users, or the generalisation to 3D
grids. These are attractive follow-up steps on our re-
search agenda towards a better exploitation of sonic
interactions for grid-structured data types.
We introduce an interactive sonication of 4×4 Su-
doku grids using direct interaction with a graphics
tablet. The Sudoku grid can inform on how we might
sonify sets of data and how they cross-correlate. The
sonication design was straightforward, following the
Model-Based Sonication idea that data parametrises
acoustic systems, and that movement on the grid ex-
cites these systems. Thereby the sounds indicate quite
directly what state a grid cell is in. Interestingly, users
start quickly to develop strategies to explore the 4×4
grids which we haven't anticipated beforehand, such as
drawing circles in cages, or doing quick line-scans, or
tapping on cells. Due to the limited complexity of the
grid, this direct interaction is suited to allow users to
solve the Sudoku. However, scaling the problem to the
9×9 Sudoku fails for two reasons: the user's memory is
exceeded with the many items, and the proprioception
is not enough accurate to understand exactly what cell
is being inspected. To better solve the 9×9 Sudoku,
possibly more specic sonication designs need to be
developed.
The Connect Four game represents grid data where
linear patterns occur. A rhythmic sonication ap-
proach was developed for the game, which can now
successfully be played with the visual display playing
a very minor role. It exemplies an auditory display in
good analogy to visual games where the board is per-
sistent for both players  here the persistence is cre-
ated by a looped sonic pattern which serialises the grid
column-wise. First comments from players are promis-
ing, however, we need to conduct user studies in order
to investigate the potential of learning to better under-
stand the grid set-up.
We are condent that by focusing on grid-based
games we will be in a very good position to evaluate
sonication designs and to compare the eectiveness
of dierent designs. These games thus represent an
ideal platform to examine sonic interactions. We hope
to make the games attractive so that players enjoy to
play and generate valuable data for us voluntarily.
Our next steps in this work is to complete our anal-
ysis of the formal evaluation of Rhythmic Connect
Four and to integrate some of ndings from Sudoku
to strengthen its implementation. This will include
a whole spectrum of grid inspection: direct cell-based
interaction, localised region overviews and overall sum-
maries into a coherent interactive sonication system.
This demands that we structure the sonications so
that the information obtained via the dierent ap-
proaches can easily be fused into an increasingly ac-
curate mental image of the grid.
Acknowledgements
We would like to thank the COST IC0601 Action on
Sonic Interaction Design (SID) for sponsoring this work
and allowing collaboration between the Ambient In-
telligence Group at Bielefeld University and the In-
teraction, Media and Communication Group at Queen
Mary, University of London. Thanks is also extended
to all the members of those groups as well as those of
the Centre 4 Digital Music (QMUL) who helped eval-
uate this work.
References
[1] Thomas Hermann. Online sonication examples.
http://sonication.de/publications.
[2] Thomas Hermann. Sonication for Exploratory
Data Analysis. PhD thesis, Bielefeld University,
Bielefeld, Germany, 2002.
[3] Thomas Hermann, Gerold Baier, Ulrich Stephani,
and Helge Ritter. Vocal sonication of pathologic
EEG features. In Proceedings of the International-
Conference on Auditory Display (ICAD), 2006.
[4] Thomas Hermann and Helge Ritter. Listen to your
data: Model-based sonication for data analysis. In
Advances in intelligent computing and multimedia
systems, pages 189194, August 1999.
[5] Johan Kildal and Stephen A. Brewster. Providing
a size-independent overview of non-visual tables.
In Proceedings of the 12th International Conference
on Auditory Display (ICAD), June 2006.
[6] Gregory Kramer. An introduction to auditory dis-
play. In Auditory Display. Addison-Wesley, 1994.
[7] Tony Stockman. Interactive sonication of spread-
sheets. In Proceedings of the. International Confer-
ence on Auditory Display (ICAD), 2005.
-8-
34
 Using audio aids to augment games to be playable for blind people

David C. Moffat and David Carr

The eMotion-Lab
School of Engineering and Computing
Glasgow Caledonian University
Scotland, UK

email: David.C.Moffat@gmail.com

Abstract. One potentially important use of audio technology is to make video games for blind people. Visual impairment makes
almost all current video games totally inaccessible. We augmented a simple shooting game with audio aids to provide information
about the direction and distance of moving targets, and tested it to see whether people could play the game without seeing the screen.
Players managed to play the game quite well in the blind condition, and in some cases the audio aids improved their performance
when they could see the screen. The conclusion is drawn that audio aids can make mainstream games accessible for the blind.

1 Introduction 1.1.2 The lack of any audio direction or location

Blind and partially sighted people are excluded from playing
nearly all computer games. It is not often noticed or thought
remarkable, because up to recently games have been strongly
associated with a younger generation of enthusiasts. But games
are becoming a mainstream means of entertainment and
sociability, and it is no longer acceptable that any significant
sector of the population be excluded.
Especially when one considers that the elderly form a large
proportion of all people with visual impairments, it seems clear
that some efforts should be made to cater to this and other
disabilities.
Special games could be made for such people, but it would be
preferrable to augment normal games in such a way that the
blind can play them as well. This is challenging, but if feasible it
would give them access to many more games, and potentially
allow them to play along with normally sighted people, too.
In the study reported here, we explore the possibility of making
normal games playable for visually impaired people, by using
stereo audio output from a game to supplement or replace the
usual graphical output through a screen.
It is worth noting that making games playable for the blind in
this way would have benefits for other people as well. For
example, a game that does not need a visual display for output is
a game that might be played while walking along, using only a
PDA or mobile phone with stereo headphones.
More serious mobile applications may also benefit, by using the
same techniques that can be developed for audio games.
1.1 Problems for blind players
Visual impairements can make a game inaccessible in different
ways. Bierre et al [1] list the following categories of problem.
1.1.1 Only visual feedback to commands is given
When a player tells the game to do an action the only output
thats indicates what the player did is visual. Without such
feedback the game is unlearnable for the blind. The game menu
system is also inaccessible, without audio feedback to tell what
the menu options are and which option is currently selected.
Using synthetic speech output, textual output can be spoken to
the players, and this technology (of so-called "screen-readers")
is obviously important to the visually impaired who use
computers in any way. It can be used to augment the visual
feedback with audio feedback in games.
The lack of any audio indication of direction and orientation, in
a 3D game for example, results in blind players being unable to
tell where or what they are facing in a game.
Objects may be announced with speech output or special sounds
to represent them (so-called "earcons" by analogy with symbolic
icons for a visual interface). For example, an enemy monster
could emit a characteristic noise; but without direction
indication, or distance information, the player cannot take aim
with any normal weapon and shoot the monster.
1.1.3 Game hints or story elements are often only visual
Vital game or story information may be presented as objects on
the screen, or cut scenes and graphical animations. On-screen
text might not even be readable by a screen-reader.
This problem is particularly severe for adventure games, where
a lot of information is displayed using text only, making it
impossible for blind players to follow the game's story.
1.1.4 A lack of a "no 3D graphics" mode
Visually impaired people do not buy graphics cards for general
computing purposes, since they have no use for them.
But new video games require the latest graphics cards to work,
and typically do not provide an option to run the game without a
graphics card. Therefore the games will usually not even
execute on blind people's computers.
1.1.5 Game documentation in text form only
With the game manual only in text format both as a booklet and
as a text file on the games DVD visually impaired gamers are
unable to learn how to operate and play some video games
without the help of someone who can read the documentation
out loud to them. This means that every time when visually
impaired people wish to consult the game's documentation, they
will need someone with them. With the storage space available
to developers in the form of DVDs and so on, game developers
could easily provide the documentation in audio form; or they
could provide it in a textual form that would be easily navigable
with the aid of a screen-reader.
1.2 Games for the blind
Some of the problems identified above can be solved with
speech synthesis output from the game, and for some problems a
screen-reader might be enough.

-1-
35
In the following listed examples of games for the visually
impaired, synthetic speech output occurs often as an audio aid.
There are other forms of audio aid that have been tried as well,
with some degree of success. By the phrase "audio aid" is meant
any type of audio output that is intended to help visually
impaired players to play the game.
1.2.1 Special games
Some developers make games especially for the visually
impaired. These are not intended to be played by normal
(sighted) people. They are audio games, without a graphical
aspect, designed so that blind people can play them well.
One developer that makes such games is GMA games [2]. They
have made the game Lone Wolf, for example, which is a
submarine warfare game, in which the player captains a
submarine on battle missions. The submarine uses speech output
to report from its onboard sonar system, to tell the player when
enemy submarines are within firing range. The technicalities of
aiming torpedos can be assumed to be left to the crew and to
automation. It is a clever idea to choose a game world in which
visual impairment would not be a disability. The technique of
using in-game automatic devices for tasks like aiming is used in
other accessible games, too.
In another game of theirs, Shades of Doom, which is based on
the original Doom FPS (first-person-shooter) game, the player
has a special "night scope" device that signals with earcons
(special sounds) that an enemy is within range [3]. Firing the
weapon then automatically hits the target. While this game was
based on a game for normal players, the game-play has been
changed by this manner of shooting, to accommodate the blind.
To change the game-play for an FPS game in this way is to
make it a special game that is quite different from the original.
In these special audio games, it can be seen how good game-
play can be brought to the visually impaired. It is inclusive to
bring them into the world of games; but it is still somewhat
exclusive in that they are then playing specially made games
that normal people would not generally choose to play.
For the visually impaired sector of the population to depend on
special games means that they would never have the same range
of games to choose from, either. The market is smaller than the
market for normal games; and the game developers will often be
normally sighted people, who would need special training to
understand the kinds of problems that visually impaired gamers
face. From earlier points made, it is apparent that normally
sighted developers suffer from blind spots of there own, in so
often assuming that all players can see as well as they can.
To leave special game development to visually impaired
developers, on the other hand, would make for much more
accessible games; but even fewer of them. It would mean a
minority of the game-playing population being served by a
minority of the game-making population, which would make the
games doubly expensive as well as relatively few in number.
For those reasons, but especially for social inclusion, to let
visually impaired play with normally sighted people, it is good
to attempt to modify normal games with extra audio output so
that the visually imparied can play them, too.
1.2.2 Games augmented with symbolic audio (earcons)
An example of a normal FPS game that has been modified to be
playable by blind people is AudioQuake, which was developed
by the AGRIP project [4], and is free, being based on the code-
base for the original Quake FPS game by id Software [5]. The
intention of the AGRIP project is to modify mainstream games
for visually impaired players; and allow them to modify them
further if they wish.
-2-
The player in AudioQuake has a RADAR device that beeps to
signify danger like enemy monsters. The beeps sound different
for friends, for enemies, and for other objects. They are doubled
to enable them to carry more information. The double beeps are
faster when the target is closer, and the second beep is a higher
pitch when the target is higher than the player.
Compared with the speech output in the games mentioned
above, these beeping earcons allow information to be conveyed
to the player quicker, which is important for an FPS game. By
giving the player more detailed information about the objects in
the 3D world, AudioQuake gives players the opportunity to use
their own spatial reasoning to make decisions. It still provides
some automatic aiming, however, in that weapons "lock onto"
targets that are within a certain angle from straight ahead. The
experience it provides with the RADAR device is reduced in
another way, too, in that only the nearest friend or enemy is
detected. To have too many earcons representing several other
objects at the same time was judged to be too confusing.
The game AudioQuake is quite close to providing an authentic
experience for blind players, but again the aiming part of the
game's play is different from that of original Quake. The earcons
used are also symbolic sounds, rather than naturalistic ones.
Real enemies and friends do not beep faster when they approach
you; nor beep higher when they are above you. A more
naturalistic way to convey that kind of location information
would be to use the human capacity for stereo hearing.
Another free game that makes good use of audio aids for blind
players is Top Speed 2, by Playing in the Dark [6]. It is a car
racing game, in which the sounds made by other cars passing on
the right or left play louder in one or other speaker. The stereo
perception of audio is also used to guide the player around the
track, by providing audio feedback for position. The centre of
the track emits a sound, so that the player hears it louder on one
side when the car moves off the central "racing" line. Rather
than use sounds for walls to warn the player away, as in
AudioQuake above, the road-centre sound is used to keep the
player in the middle of the track, even as it turns.
Another clever trick is to provide the driver with a co-driver, as
in rally races. This means that the game can give a lot of
information to the driver about the road ahead, as well as timing
performance, in quite a natrual way, using speech synthesis. By
this means the game provides a fairly authentic experience
without needing the blind player to play a special game. The
modifications to the driving game are small enough to allow
normal players and blind players to play together.
Another computer game for normal players which has been
modified to be accessible to people of various disabilities,
including visual, is AccessInvaders [7], which is a variant of the
classic arcade game Space Invaders.
When set to be played by a visually impaired player, the game is
simplified to have only one column of enemies; and they cannot
kill the player. Instead, the player loses only if an enemy reaches
the ground. The aliens emit a "sonar-like" sound, and it is spatial
so that its position can be inferred. In this last point,
AccessInvaders uses spatial sound like the game Top Speed 2
does, allowing the player to use both ears in a natural way.
In simplifying the game for the blind, however, it becomes a
special game again, quite different in game-play to the normal
version of the game. Since the designers want to encourage
blind people to play games with sighted people, they have
another way to make this possible. They introduce a
handicapping system, in which some players get a simplified
game with reduced experience, to make the game accessible to
them; and the scoring and teamplay still allow them to play
alongside normally sighted people.
36
1.2.3 Games augmented with naturalistic audio
The above games all make different compromises to give blind
players something like an authentic game-play experience,
without simplifying the game too much, and allowing some
degree of joint play with normally sighted players. They all use
audio aids to make this possible, and may use weapons that can
aim automatically or do other things to assist the player in ways
that are not alway realistic.
The games use earcons to signify events in the world, but they
tend to be symbolic rather than naturalistic. They use stereo
hearing in some cases, but typically in a one-dimensional
setting, like Space Invaders or driving, where the player only
has to go left or right. When the game requires more complex
maneuvering in 2D or 3D space, like the shooting games, the
audio aids tend to revert to carrying information symbolically,
or with speech.
It is a challenge to support a blind player in a shooting game,
with naturalistic audio aids that convey information in the way
that sounds do in real life. Not with synthetic, coded beeps, that
is; but with natural properties of sounds, like volume and stereo
detected direction.
The study reported below attempted to meet that challenge. A
traditional shooter game is augmented with audio to give the
player location information for objects (targets) in a naturalistic
way, to see how well blind people could play it. The game is not
simplified for them in any way.
2 Audio aided Asteroids
Asteroids is a classic arcade game in which a spaceship moves
over a 2D area rotating left and right (or anti- clockwise), and
using its jets to go forwards (thrust). The ship must shoot at and
destroy the asteroids moving through space around it (see
Figure 1), and if the asteroids break into pieces, then they must
be destroyed also. When an asteroid goes off the screen, it
isntantly "warps" to appear at the opposite edge. If any asteroid
collides with the ship, the player loses a life and starts with a
new ship in the centre of the screen. The aim is to destroy the
asteroids in the shortest time.
AudioAsteroids is the version of the game developed for this
study. It is set to be fairly easy to play, even for beginners, so
that the effects of audio aids can be seen on a range of players.
However, it does have two levels. Players must complete both
levels in order to finish the game. In the first level (L1) the
asteroids wait to be shot at; and in the second level (L2) they are
moving across the screen, and so harder to hit.
2.1 Symbolic sounds for different objects
The main feature of AudioAsterdoids is that each asteroid emits
a characteristic sound. The sounds used were a cello string being
plucked about twice per second, and the rattle of a small card
box of paperclips. The manipluations of these sounds were
intended to convey information about the location of the
asteroids to the player.
These sounds are clearly not naturalistic, because real asteroids
do not sound like cello strings. They are symbolic sounds –
"earcons" – whose meaning can only be understood once it has
been explained. This is a necessary compromise, because
objects in space in fact make no noise at all. Other space games
and science fiction films make the same compromise when they
have loud explosions and other sound effects in space. Sound in
space is physically impossible, because there is no air for sound
waves to travel through; but it is shown in films because, like
background music, it gives an "emotional atmosphere."
-3-
Figure 1: The Asteroids game
To convey location information, however, AudioAsteroids does
use more naturalistic properties of sound. As shown below, the
game tries to solve the main problems that blind people have in
playing normal games.
2.2 Speech for the menu system
The first problem that visually impaired players encounter is
starting the game and setting options, because they cannot use
the visual menu system.
The audio aid to solve this part of the problem simply uses a
synthetic speech output tool that speaks the options available at
each stage. This aid was tested more thoroughly in another game
which described scenes by reading out the screen text, and gave
quite complicated series of changing options. It was an
adventure game, and the menu audio aid did enable people to
play the game without looking at the screen; but we focus
attention here rather on the other audio aids for location.
Because the ship in AudioAsteroids can rotate and move through
space, players need to know the relative direction of asteroids,
and how far away they are. The two audio aids for these are the
directional aid and the distance aid.
2.3 The directional aid
Because we have two ears, we can estimate the horizontal angle
of a sound source quite accurately, depending on circumstances.
When the source is almost directly ahead, we can tell if it moves
through an angle of as little as one degree of arc ([8] cited in
[9]). This is mainly because of the difference in time of arrival
of the sound at the two ears, and consequent phase difference.
For higher frequency sounds, there is also a difference in
volume at the two ears, because one ear will be partly shielded
from the sound source by the head.
The auditory cortex of the brain uses these clues to inform us of
the location of the sound source. There are other clues available,
also. For example, the relative height of frontal sources can be
estimated if their sounds are high in pitch, because of the way
that the shell of the ear reflects certain high frequencies back
into the ear canal.
The shape of the ears also helps us to determine whether the
sound source is in front or not, and sources from the rear will
sound a little "muffled."
The way the ears register sounds, and the brain decodes the
signal, is highly complex and not fully understood. However
using some of the main properties as outlined above it is
37
possible to manipulate sounds in ways that make it appear to a
listener with stereo headphones that the source is moving from
left to right (or "panning") and even going behind the head.
2.3.1 Sound from behind the head
Each sound that is to be used to represent an asteroid is
processed to make a second version that sounds like it comes
from the rear. This was done with a software tool called
Maven3D [10], which includes algorithms to simulate the
"muffling" referred to above, that makes sounds appear to
source from behind the head.
There are therefore two sound files prepared for each asteroid:
the original one for when the asteroid is towards the front of the
ship (ahead of the ships left-right axis), and the muffled version
for when the asteroid is somewhere behind the ship.
2.3.2 Sound Panning
Whether in front of or behind the ship, further processing is
needed to feed the sound differently to each ear through the
headphones. The ear which is slightly further away from the
source is given a slightly delayed signal, which is thus pase-
shifted, and the signal's volume is lowered. This processing is
called "panning," and can be done in real-time by a software
library. The library chosen for this study was the Dark GDK
[11], which is available as a plug-in for the Visual C++
development environment that is popular with game developers,
and is now also included in VisualStudio Express.
To use the library function dbSetSoundPan is simple: the
first argument is the sound to be played, and the second
represents the left-to-right angle that it should appear to come
from.
Combining the two techniques of muffling and panning,
asteroids are given a direction using only sound. The player also
needs to get an idea how far away from the ship the asteroid is.
2.4 The distance aid
Where some games use earconic or symbolic methods to signify
distance of an object, including some mentioned above, our
intention was to use naturalistic attributes of sounds to carry
information, where possible. There are many attributes that can
play a role in humans' estimation of distance from source,
including reverberation of the local environment and Doppler-
shift effects coupled with knowledge of the source; but
AudioAsteroids uses only loudness.
2.4.1 Naturalistic sound at a distance
The most important way we tell the distance of an object
auditorily is from the volume of the sound. In fact we need to be
familiar with the type of sound to do this, so that we can
estimate how loud the sound is at source, and thereby estimate
the distance from it's apparent loudness at the ear. Because of
the variablility of sound loudnesses at source, and for other
reasons too, humans do not estimate distance very reliably from
sound alone.
The asteroids in AudioAsteroids are set to sound proportionately
louder the nearer they are, by a simple linear function. The
precompiled sound for the asteroid is unchanged to signify its
closest distance (of zero), at maximum volume (100%). As the
asteroid moves away, its volume falls off linearly with distance
until it reaches its minimum volume when it is the furthest away
it can be, which is the width of the screen.
2.4.2 Modification of the sound-distance scale
To set the relative proportion of loudest to weakest sounds for
the asteroids, a simple pilot test was done to see how big the
-4-
difference could be allowed to grow. Clearly the proportion
should be as large as feasible, to make the player's distance
estimation more accurate. If there were no difference then all
asteroids would sound the same distance away. But if the
proportion is too large, then only the nearest objects are heard.
The minimum volume is set at 60% of the maximum volume,
which was found to be a good level to allow the asteroid to still
be heard when it is far away, and not get drowned out by much
nearer asteroids.
No tests were done to confirm that players could accurately
estimate the distance of asteroids, because we did not expect that
they would be able to. The purpose of the distance audio aid in
the game-play is to tell the player which asteroid is closest, and
when it is so close that it presents a danger to the spaceship.
2.5 Audio cross-hairs
During pilot tests, it became apparent that players would be
almost unable to hit moving targets. They would generally aim
to were the target was, but it would move on in the time it took
the bullet to travel to the target's position. This trouble could
have been alleviated by making the weapon a laser, that would
shoot the target instantly, but that would be to change the game.
Instead, some intelligence was built into the ship's weapon to
assist blind players by predicting the future position of the
target. The direction aid indicates this predicted location to the
player, rather than where the target is currently.
A visual analogue to these "audio cross-hairs" is the auto-
targeting function that can be found in military fighter jets,
where the pilot sees cross-hairs over the target when in fact the
aim of the gun is ahead of the target to where it will be when the
bullets arrive there. To include this audio cross-hairs feature for
visually impaired players does change the game-play somewhat;
but it is not unrealistic, and normally sighted players would have
access to the same auto-predictor device if they would listen to
the audio aids as well.
The reason it is so difficult to aim at a moving target without
auto-prediction is probably because distance estimation is so
innacurate in human hearing. That estimation is crucial to the
calculations of where to aim, so that the moving bullets will
collide with the moving target.
With the audio aids in place, the game could be evaluated to see
how well they help players to play it in a blind condition, when
they cannot see the screen.
3 Blind playability of AudioAsteroids
The AudioAsteroids game was tested on four players of different
skill levels. The players were normally sighted, so that their
performance could be compared in the blind and normal
conditions. The weakest player was a novice, and so the
difficulty of the game was set to a low level to allow all the
players to nearly complete it. Their relative performances would
be measured by time taken to complete it.
3.1 Participants – Players
The participlants in the experiment (the players; the Ps) were of
varying levels of skill. Three were aged between 20 and 23, and
the oldest was 50 years old: she was the novice gamer, who had
little experience of playing computer games of any type.
Three players had experience of computer games, two of them
more particularly of Asteroids, and one player had some
38
experience of the audio aids used in AudioAsteroids, having experiences shown in Table 1. Each player is also consistent in
played a part in its development. The relative skill levels of the performance, as shown by the standard deviations. The least
players are shown in Table 1, in which players are shown to consistency is shown by the weakest player, again as would be
have experience by a plus (+) sign. expected.
To confirm these inspections with statistical analysis, t-tests
P1 P2 P3 P4
were drawn between players to see if the scores were
+ + + - Experience of games
+ + - - Experience of Asteroids
significantly different. They were. The (unpaired, two-tailed) t-
+ - - - Experience of Audio Aids tests between P1 and P2 yielded probabilities of 0.01 for shots,
and 0.66 for time, indicating that their times were not
significantly different, but that their shots surely were.
Table 1: The participants' levels of experience
It appears that P2 is the stronger player, wasting significantly
fewer shots.
3.2 Procedure Visuals S T
The game was played on a laptop computer, with headphones 1.1 26 64
attached for the sounds output. The spaceship is controlled with 1.2 24 54
1.3 25 49
an Xbox 360 wireless joypad.
1.4 26 63
The game was started for the players, who first played it five
1.5 24 54
times with only the visual output on screen. They then played it P1 mean 25 56.8
five times with audio on, but not looking at the screen (no stdev 1 6.46
visuals); and finally they played five times again, with both
t-test 0.01 0.66
visuals and audio.
The difficulty of the game was set to be easy enough to allow 2.1 23 61
most players to complete the game without losing a life. They 2.2 19 58
would differ mainly in time taken. The hope was that the game 2.3 14 67
would spread the players, not defeat them. 2.4 22 64
2.5 18 45
To play the game once, it was necessary to play through two
P2 mean 19.2 59
levels, in which L1 starts with stationary asteroids, but in L2 the
stdev 3.56 8.51
asteroids are moving targets. In either case, a minimum of 7
shots are needed to destroy all the asteroids and their fragments. t-test 0.00 0.01
Performance was noted by recording two figures for each play 3.1 30 80
of the game: the number of bullets fired, and the time taken. 3.2 25 61
3.3 32 78
3.3 Results 3.4 32 91
3.5 30 96
All players completed all levels, destroying all asteroids, and P3 mean 29.8 81.2
any fragments resulting from asteroids the first time they are stdev 2.86 13.55
shot.
t-test 0.08 0.03
Every player found it difficult to reach 100% accuracy, and they
nearly always wasted some shots, firing more than 7 times in a 4.1 47 121
level. Only one player lost a life (this was P3 in the Audio-only 4.2 59 133
condition, which is the hardest); so we ignore lives lost. 4.3 27 65
4.4 37 133
The players differed in their completion times, both because of
4.5 27 104
skill level, and because of the difficulties of playind "blind." P4 mean 39.4 111.2

Those results are the aim of the experiment, but first we validate
the experimental setup with the results of all players in the
normal condition, with Visuals only, as in the traditional version
of the game, which is normal Asteroids.
3.3.1 Confirmation of player skill levels (Visuals)
The game was fairly easy for all players to complete without
losing lives; but there were significant differences in shots
wasted and in time taken to complete.
We first verify that the players' reported skill levels are
consistent with their results, and that individual players perform
quite consistently within each condition.
Table 2 shows the results for all players and all games in the
first condition, with only the visuals on. P1's third game is
signified by the row "1.3" and so on. The mean scores for each
player are shown, and their standard deviations. The scores are
the number of shots (S) fired to complete both levels in the
game, and the time (T) taken to complete them, in seconds.
By inspection it is clear that P1 and P2 are the best players of
the game, followed by P3 and lastly P4. This is just what would
be expected from the players' relative skills as suggested by their
Table 2: Confirmation of player skill levels
The other t-tests shown in Table 2 (comparing P2 against P3,
and P3 against P4) were again unpaired, but they were one-
tailed. This was because we expect that P2 should get higher
scores than P3, and P3 higher than P4, due to the Ps' differences
in experience.
The t-tests confirm that P2 (an Asteroids player) is more skilful
than P3 (a general gamer, but new to Asteroids), based on shots
and on time. P3 does not waste significantly fewer shots than P4
does (the novice), but is significantly quicker.
This validation and consistency of the players' relative skill
levels for the Visuals-only condition, which is comparable to
classic Asteroids, means that the further results can be relied
upon, and the significance of any variation in results can be
assessed by comparison with the variation for a player within
each condition.
3.3.2 Development of performance measure
To help in further performance comparisons between players,
the S and T scores (for shots and time) are combined into a

-5-
39
summary performance measure, P, which measures the P2 does not do as well as P1 in the blind condition, despite
divergence from an optimal or extremely strong performance. being the better player, which can be put down to P1's prior
The P measure is larger for weaker players, who waste more experience of the audio aids. It is notable that P2 does not
shots and waste time. P is smaller, close to zero, for stronger perform much better than P3 in the blind condition (one-tailed,
performances that waste few shots and complete in near record unpaired t-test; p = .38), although the gap between them was the
time. P is calculated by the following formula: largest one in the Visuals condition. One hypothesis is that P2
cannot any longer use strategies learned in previous Asteroids
1 1 experience, and is thus reduced to re-learning the game in blind
P = ⋅3⋅ S−14  ⋅T −34
2 2 mode. With that specialist knowledge of Asteroids neutralised,
P2 is not much better than a general gamer like P3.
The lowest value that S can possibly have is 14, over both levels On the other hand, now P4 does much worse than P3 (one-tailed,
of a game; so S-14 represents the number of shots wasted in a unpaired t-test; p = .002), whereas she was closer before. It
game. The fastest times that any player achieved in any level of appears that there are two factors for this result. The first factor
any game total 34 seconds for both levels, which is therefore a is that P4 cannot cope well with the heavy task imposed on her.
fair indication of the optimum time that can be reached by a She is experiencing a cognitive overload, which the experienced
player who completes both levels. By inspection, the average gamers are not: she still has to think explicitly about the nature
firing rate of all players was about 3 seconds per shot, meaning of the gaming context, including the joypad keys, while the
that the number of wasted shots should be tripled to compare the others are familiar with the context, if not the game itself, and
cost with wasted seconds. Finally these two "waste factors" are can therefore concentrate more on playing it.
combined with equal weighting (of ½ each) to arrive at a The second factor is related: P4 is getting tired. This suggestion
performance score. is consistent with her results in the final condition, as analysed
Using this notion of performance, players with different styles – below.
who fire more accurately but take longer, for example – can be
meaningfully compared. Despite the large drop in performance that all players suffer in
the blind (Sounds only) condition, it is remarkable that all of
The mean scores including performance (P) for each player in them, including the possibly overloaded novice player P4,
all conditions are shown in Table 3. manage to complete the game. Only one life is lost in all these
games, and that is by P3, not P4.
The neutralisation of the more skilful players' experience
Visuals M S T P suggests that they are having to learn new strategies. The game
P1 25 57 28 in the blind condition is almost like a new game to them. It was
P2 19 59 20 probably too much to hope that blind players would be able to
P3 30 81 47
play against sighted ones on an equal footing. But even if the
P4 39 111 77
mean 28 77 strategies or skills required for the blind version of the game are
different, the fact remains that blind players can play with
Sounds M sighted ones, albeit at novice level.
P1 34 125 75
P2 36 168 101 3.3.4 The audio aids do not impair performance (Both)
P3 48 146 107 In the final Visuals+AudioAids condition (BOTH in Table 3),
P4 67 384 255 performance is comparable to Visuals alone.
mean 46 206 Three players even improve: P1 improves from 28 to 20 (with
p = .06 for unpaired, two-tailed t-test), by shooting more
BOTH M
P1 19 58 20
accurately, P2 and P3 appear to improve slightly, but not
P2 20 50 17 significantly.
P3 24 79 38 The novice player P4 gets worse (no less accurate, but slower)
P4 38 140 89 in the final condition, compared with the Visuals only one. As
mean 25 82 noted above, this is consistent with the hypothesis of mental
tiredness due to the greater cognitive load the task has put her
Table 3: Ps' mean scores in each condition under.

According to the summary P scores in the Visuals condition, it
appears the P2 is indeed the strongest player of classic
Asteroids. The unpaired, two-tailed t-test gives a barely
significant probability of this of 0.08 however, so there is not
much in it. Unpaired one-tailed t-tests confirm that P2 is much
stronger than P3 (with p=.0006); and P3 is stronger than P4
(p = .04).
3.3.3 Performance in the blind condition (Sounds)
Referring to Table 3, it is clear that all players perform much
worse in the blind condition ("Sounds,"with only the audio aids
to guide them).
Player P1's performance falls to 75, which is comparable to the
novice performance in the Visuals condition (P4 in the top of the
table). Therefore, at least in this case, a skilful but blind player
can match a normally sighted novice at a shooting game, when
provided with helpful audio aids.
3.3.5 Moving targets are much harder, when blind
The performance of players at each level can be calculated, and
compared, to give a measure of how much performance suffers
when the targets are moving.
Figure 1 shows the performance for each player and condition in
L2 over performance in L1. When this measure is about 1, there
is no difference. As the graph shows, however, having the
asteroids move through space causes difficulties for the players.
The most severe difficulties are caused for P1 and P2, who are
the players most experienced in Asteroids. This suggests that
they have learned strategies for shooting moving targets that are
no longer applicable with only the audio aids for information.
P3's performance cost for moving targets, on the other hand,
does not suffer in the blind condition, suggesting that this
experienced gamer, new to Asteroids, has no particular shooting
strategies to lose.

-6-
40

P1
P2
Player
Visuals
P3 Sounds
BOTH
P4
0.0 1.0 2.0 3.0 4.0
Perf. cost of m oving targets
Figure 1: Performance in L2 compared to L1
– bigger numbers mean performance falls more
Again, the suggestion is that the audio aids provide an gaming
environment that permits blind people to play; but that they will
have to develop strategies quite different from the ones that
work well for normally sighted people. Some comments from
the players were to the effect that playing AudioAsteroids was
like learning the game from the beginning again.
The information presented auditorily is not exactly the same as
the visually presented game state. There is one small advantage
for the audio aids, in that they relate asteroid positions to the
ship's orientation; whereas, when viewing the screen, players
must continually look between the target and the ship, in order
to decide precisely when to shoot. But the audio aids suffer from
bigger disadvantages that that.
One particular difficulty is when asteroids warp from one side of
the screen to the opposite side, when they move over the edge.
Graphically, these warp events can be seen and predicted easily;
but there is no audio feedback for blind players.
The audio aids could be confusing in other ways, too. When
asteroids break up, for example, the remaining pieces generate
their own, different, sounds because they are new targets. There
could be up to four different sounds simultaneously.
By observation during play, it was noticed that unsighted
players tended to shoot at the nearest asteroids. This could be
because they generate the loudest noises; but also because they
present a greater danger; or because they are easier to hit.
3.4 Limitations of the study
One obvious limitation of the present study is that it only used
four participants. They repeated their plays of the game several
times each, but still it means that whole sectors of the gaming
population, with different levels of experience, are represented
by only one participant.
As a result of the small umber of participants, some of the
conclusions drawn have had to be tentative.
A more serious limitation is the way the experiment was
designed, with all plays of the conditions occurring in blocks. It
allows the possibility of a learning effect, in which the players
are more skilled in the last block (of both visuals and audio aids)
-7-
because they have experienced the game in earlier blocks. If the
plays had been interleaved, in order to control for any learning,
then some of the conclusions suggested earlier could have been
stated more strongly.
Another point is whether the results would extrapolate from
unsighted to blind players.
The participants were "unsighted" rather than "blind" because
they study could not see the screen, yet were not actually blind.
It is a further question whether really blind people, such as those
blind from birth, could also play the game. It might be that they
would lack the gaming history, and not well understand the
context of shooting asteroids, for example.
4 Conclusions and further work
The purpose of the study was to see if unsighted people could
play a shooting game, where location of world objects is crucial,
with only naturalistic audio aids to guide them.
With certain compromises, including the addition of audio
crosshairs to predict the right direction to aim, and non-
naturalistic sounds to asteroids which would be silent in space, it
was shown to be feasible for unsighted people to play the game.
In other respects than the audio aids, the game was the same as
the classic Asteroids game, and normally sighted people can
play it with its game-play unchanged. This demonstrates that
even quite dynamic games could be instrumented to enable blind
people to play them, without much compromise.
More tentatively, it was possible to suggest the performance that
blind players could achieve with the audio aids. It appears that
they can learn to play well enough to play against normally
sighted players at novice level. This is significant because it
means that blind people are not socially excluded from the
increasingly important world of gaming; nor sidelined into some
small gaming world of their own, playing only against other
blind people.
Consequently, game developers may feel encouraged to add
audio aid technology to their games in order to make them
authentically playable to the visually impaired. Just as websites
are increasingly being designed to make them more accessible to
the visually impaired, the same developments may occur with at
least some genres of computer game. While it is more difficult
to make games accessible, at least it seems feasible.
Note that the audio aids used in AudioAsteroids were a first
attempt, and that further work could find better versions of
them. More attributes of sounds could be used, and we have
little doubt that player performance could be improved from
these initial levels – the only question is by how much.
As well as helping the visually impaired, the use of audio aids
such as in this game offers interesting possibilities for sighted
people who just happen to be temporarily without a visual
display. The ever increasing use of mobile phones and similar
technology, while walking about town for example, will create a
need for audio information rather than visual output.
It would be interesting to see whether sighted players could
learn to use the audio aids while viewing the screen. Would they
return to their normal strategy after having adapted to the blind
condition; or would they be able to combine strategies and
modalities, and become better players than they were before?
The bandwidth for audio presentation of real-time information
should also be investigated, and compared with the quantities of
information that can be visually perceived in the same time.
Finally, there are possibilities for future work that analyses the
changes in strategy encouraged or demanded by the audio
version of the game. When unsighted, the players definitely
41
seemed to play different strategies. Analysis of collected video
clips would be one way to pursue this question.

Acknowledgements

We thank Brian McDonald, David Farrell, Scott Gannon, and

members of the School's Games Interest Group for their helpful
comments on this work.

References
[1] Bierre, K., Chetwynd, J., Ellis, B., Hinn, D. M., Ludi, S. &
Westin, T. "Game Not Over: Accessibility Issues in Video
Games." In Proc. of the 3rd International Conference on
Universal Access in Human-Computer Interaction,
Lawrence Erlbaum (2005).
[2] GA Games. http://www.gmagames.com/
Accessed on 4th June, 2008.
[3] Peterson, B. Guide to Shades of Doom by GMA Games.
http://www.audiogames.net/pics/upload/shadesofdoom.doc
Accessed on 4th June, 2008.
[4] The AGRIP Project. http://www.agrip.org.uk/about/
Accessed on 4th June, 2008.
[5] id Software. Quake.
http://www.idsoftware.com/games/quake/quake/
Accessed on 4th June, 2008.
[6] Playing in the Dark. Top Speed 2. A free computer game at
http://www.playinginthedark.net/
Accessed on 4th June, 2008.
[7] Grammenos, D., Savidis, A., Georgalis, Y., & Stephanidis,
C. (2006). "Access Invaders: Developing a Universally
Accessible Action Game." In K. Miesenberger, J. Klaus, W.
Zagler, & A. Karshmer (Eds.), Computers Helping People
with Special Needs, Proceedings of the 10th International
Conference, ICCHP 2006, Linz, Austria, 12 – 14 July (pp.
388–395). Berlin Heidelberg, Germany: Springer.
[8] Middlebrooks, J.C. & Green, D. M., "Sound localization by
human listeners," Annual Review of Psychology 42, pp.
135-159 (1991.)
[9] Lu, Y.-C., Cooke, M. & Christensen, H. "Active binaural
distance estimation for dynamic sources." Proceedings of
InterSpeech-2007, Antwerp, Belgium (2007).
[10] Maven3D. Software package for 3D audio editing.
http://www.venturaes.com/emersys/index.html
Accessed on 4th June, 2008.
[11] DarkGDK. Software dame development kit.
http://gdk.thegamecreators.com/ and
http://www.microsoft.com/express/samples/GameCreators/
Accessed on 4th June, 2008.

-8-
42
 Beowulf field test paper
Mats Liljedahl
mats.liljedahl@tii.se
The Interactive Institute, Sonic Studio, Acusticum 4, SE-941 28 Piteå, Sweden
Abstract. A practical field test, covering some of the parameters governing audio based games, designed for mobile applications
utilizing new techniques with the intention of allowing greater interpretive freedom for the player. The tests are realised through an
audio-based, simple game application: ‘Beowulf’.
1. Introduction
Audio may represent a substantial percentage of our total
perception, but it has been chronically under-represented in the
development of modern communication devices and
applications, even when these devices logically have audio as
their prime media.
Graphics and visual interface solutions continue to dominate
both the media’s attention as well as the lion’s share of the
development resources. This seems especially strange in relation
to the explosion of technical development in the mobile phone
industry. Despite audio-based information being the principle
drive and ‘raison d’être’ of all mobile phones, the vast
percentage of technical innovation to date is designed to emulate
the personal computer and is focused on improving the display
and the graphic quality.
Another perceived problem with this focus on graphics is the
way in which it limits the user. Large resources are continuingly
being put into the creation of more and more realistic game
graphics. This is true across the board, whether we are talking
about game consoles, home computers or more compact
devices.
Like so much in modern media, this trend has its origins in film;
Hollywood, especially, has a tendency to saturate its more
popularistic offerings (Marvel Comic based films, Indiana Jones
and especially the Star Wars franchise, for example) with faster
and more complex action, showing ‘bigger, better, faster’
content and graphics. This visual over-kill utilizing, for the most
part, photorealistic 3D environments in real time presents a real
danger to the user experience:
In the early days of cinema, there just weren’t the technical
resources and budgets to create the vast array of fantasy
creatures, monsters and overwhelming, catastrophic visions we
see in modern films, so the early film makers were forced to be
creative with other means; monsters were shown through
shadows, sound suggested storms, light created infernos, etc.
The advantage of this was that it allowed the audience great
freedom to invest their own imagination in the experience and,
as we have argued in an earlier paper [3], this actually improves
the audience satisfaction.
Beowulf was designed to take a huge step away from these
‘Total-information’ experiences and, once again, give the user
the freedom to invest heavily in their own experience of the
game.
So, we wished to challenge and explore both the under-use of
audio on mobile platforms and to strive for and focus on a
realistic, but open, experience for the user.
Two hypothesis where formulated:
1) An audio-based game for mobile devices has potential as an
-1-
Nigel Papworth
nigel.papworth@tii.se
alternative to the current graphics-based applications.
2) A computer gaming experience based on audio rather than
graphics to convey the game’s world will make it less dictatorial
and allow freedom and space for interpretation.
To test our hypotheses the project described here was
formulated. It draws on two trends in modern technology:
The first is the growing sophistication of mobile, multimedia-
capable devices.
The second is the use of ambiguity and multiple possible
interpretations as resources for design [1][2].
In addition to this, the project draws from earlier works in the
field of audio-based game for the visually impaired [4][5][6]. It
should be noted however that the project described here is, in
contrast, geared towards a normally sighted audience.
In the project, the audio-based game application “Beowulf”
serves as tool for research into the relatively broad spectrum of
questions arising from the hypotheses. Ultimately the goal of the
project is to gather indications on these two questions:
• Are audio-based games, delivered on mobile platforms,
interesting to a broad audience? Historically, these kinds of
applications have been mainly targeted at the hearing
impaired.
• Can a game experience, interpreted and disambiguated
through a virtual environment and communicated via a rich
soundscape, present the player with a new and unique game
experience?
In 2006 a first, smaller test was made with an early prototype of
the ‘Beowulf’ game. This was in order to get an initial proof of
concept for the application and some early feedback on our
basic ideas. The ideas behind the Beowulf application and its
design are described in [3]. The result of this first test was then
used in forming the second step of the project and revising the
design of the software and gameplay described here.
The focus of the project is firmly on the design process and the
user experience.
The questions it is designed to raise and address are intended to
be broad & general rather than narrow and specific.
In this paper we describe the test, the design of the computer
game application itself and the paper-based questionnaire used
in the field test. It is from these elements and the experience of
monitoring the tests in person, that we have drawn the lessons
learned and the results and conclusions.
2. Lessons Learned From Test One
The first test conducted in 2006 was used as a pilot study. Based
on the experiences from this study, the design of the game
application was revised.
43
[Title of Paper]
2.1. Lessons Learned – Sound Design
2.1.1. Health status
Beowulf’s current health status must somehow be
communicated to the player by audible means. It was decided to
use four different breathing sounds for this. A light normal
breathing means Beowulf has 100% health, a very heavy,
wheezing breath means he is almost dying. We initially tested to
let 100% health be represented by no breathing sound at all.
This turned out to be confusing for the players. Initial tests gave
clear indication that suddenly presenting the breath sound,
seemingly from nowhere, when Beowulf had lost enough health
for the first breathing sound to get activated, was very confusing
for the players. Several beta testers did not know how to
interpret the breathing sound when it suddenly appeared. Is it a
creture breathing or me myself? The sollution to this confusion
was to have the breathing sound present from the start and let it
dynamically Beowulf’s health status.
The current health status is also dynamically reflected in the
sound of the sword swing. The more health Beowulf looses, the
slower the sword swing. When Beowulf is nearly dying, you
even hear the sword cling against the floor and Beowulf groan
from the effort of swinging the sword. In addition the length of
the file is used to determine how often Beowulf can swing his
sword – he can only re-swing the sword if the previous swing
sound has played to the end. Good health has a short sound file,
bad health has a long sound file.
2.1.2. Room ambiences
Several of the ambient sounds of the caves where modified in
order to make each cave more unique. The reason for this was
more to facilitate a sub-consious memory of the various places
in the game world than to increase the effectiveness of
navigation.
2.1.3. Consistency and uniqueness
From the evaluation of the first test we learned the importance
of two sound attributes, namely what can be called ‘consistency’
and ‘uniqueness’.
For most of the sounds we use several slightly different sound
files in order to create variation and avoid hearing fatigue due to
repetition. We learned how extremely important it is to keep the
variations as consistent as possible when it comes to volume,
reverberation, equalisation etc. The player must be able to
perceive all the variations in the same way, be able to
experience them as coming from the same source. In particular,
we had to adjust volume for several of the variations of creature
sounds.
The Beowulf soundscape is filled with sounds and it becomes
vital to make it possible for the player to discriminate between
both different categories of sounds and individual sounds. The
uniqueness of the various sounds becomes crucial. We had to
rework several of the sounds of attacking creatures to make
them intuitive and obvious. In same cases it was enough to
increase the volume or decrease the amount of reverberation to
make the sound appear to emanate from a position closer to
Beowulf. In other cases we had to totally redesign the sound to
achieve the desired quality.
Another aspect of this is the ability for the player to discriminate
between what can be called “first person sounds” and “third
person sounds”. In the revised version used in the second test we
added several “first person sounds”, i.e. sounds produced by
Beowulf. The breath-sound is one example. In the initial beta-
tests it turned out that some players misinterpreted the breathing
sound and thought it emanated from a creature or other non-
player character. We had to rework the breathing sound in order
-2-
to move it closer to Beowulf to make it appear as if it came from
the own body.
2.1.4. Music
Initially the idea was to use only sound effects in the Beowulf
game. The game play situations added in the revised version
gave the game two possible endings: success or failure. Failure
was signalled by the death sound of Beowulf. Success was
signalled by the death sound of Grendel. Early on it turned out
that it was not obvious for the players which ending was which
of these two. There was an obvious need to more clearly signal
to the player in what way the game ended. In the end we opted
for the addition of two pieces of music, an archetypal “funeral
march” for failure and a likewise archetypal “hero’s theme” for
success. This gave implicit but clear enough information about
the outcome.
These pieces of ‘end’ music game a slightly unbalances feel to
the total game experience, therefore it was decided to also add a
introductory piece of music to serve as a framing overture.
2.2. Lessons Learned – Graphic Design
There are inevitable limitations and problems with audio based
games. One of the biggest problems with the first version of the
game was the difficulty of signalling and clarifying when the
player walked into a wall. In the game, when ‘Beowulf’ hits a
wall a “bump”-sound is played.
Several early testers had a clear problem with efficient
navigation in the cave environments when the wall ‘bump’
sound was their only indicator of an obstacle directly in front of
them. The repetitive attempts to circumvent this obstruction led
to the ‘bump’ sound repeating a large number of times, which
the players found annoying and detrimental to the overall game
experience. It was therefore decided to give a visual clue as to
where the walls in the caves are located.
In the revised version, when Beowulf enters a cave, total cave is
now highlighted in grey. In this way the player can easily
determine where the walls are. The tunnels are only one unit
wide and the audio signal is, therefore, far more easily
interpreted.
Figure 1. Enlightened cave
Note that the tunnel openings in the caves are not visible. To
determine where the openings are, the player must either try to
walk beyond the edge of the cave, risking just bumping into the
wall, or better still listen carefully to find the openings.
For the test described here, where the playing time was set to
max 10 minutes, it was also decided to preview the whole map
of the game world, including caves and tunnels, for five seconds
during the music overture.
3. The Test
The over-all questions of the project, described in the
introduction, was broken down into a set of more concrete sub-
parts in order to achieve a level of measurability. These partially
overlapping questions are:
• Does an audio-based game make sense to a first time user?
• Is an audio-based game a suitable vehicle to deliver game
content on a mobile platform?
44
[Title of Paper]
• Does the level of pre-game information colour the player’s
experience?
• Can an audio-based game give a satisfactory game
experience?
• Does a non-visual game world present the player with
navigational problems or other hinders to a working game
experience?
• Can a game emulate visual combat situations using only
sound?
• Can a player correctly interpret visual sounds without the
aid of visual reference?
• Does a sound based environment encourage a player to
contribute more to the game experience?
Here follows more detailed descriptions of the Beowulf game
application, the questionnaire and the implementation of the test.
3.1. The Game Application
‘Beowulf’ is an audio based game designed to present an
absolute minimum of graphical content and as much audio
stimulation as present technology and the game definitions will
allow.
The game itself is based on a heroic, epic poem probably written
sometime between 700 and 1000 A.D. This is sometimes
referred to as England’s national epic, being the earliest
recognised example of the English language in existence.
In the Beowulf game, we have lifted out one episode of the
poem and translated this into a simple gameplay scenario; the
game terrain is a series of caves in which the poem’s main
adversary, the monster Grendel, is to be found. These caved are
described to the player through a pure audio landscape created
from realistic sound sources.
The poem narrates how the Scandinavian hero Beowulf defeats
Grendel in this lair. This is the player’s task in the game.
The caves in the game contain other creatures including
wolverines, snakes, bats & rats. These will attack Beowulf and
injure or kill him if he fails to swing his sword.
In the game scenario, a gust of wind blows out our hero’s torch
as he enters the first cave, and darkness descends. The player
must now trust to his/her hearing and, with only an extremely
primitive top view map to help, navigate a route to the monster.
The player’s task is to successfully survive all the hazards
between the start point of the game and the monster’s location.
He/she must interpret the myriad sounds that fill the
environment both to enable this successful navigation and to
define and execute the confrontational combat situations.
Finally when Grendel has been found, it must be dispatched by
determined use of Beowulf’s sword.
For a more detailed description of the game and its design, see
[3].
3.2. The Questionnaire
Given the nature of the questions addressed by the project and
after testing several alternatives, a two-part questionnaire with
one more quantitative and one more qualitative part was created.
3.2.1. Questionnaire – quantitative part
This part consists of 24 statements that all can be answered on a
seven-step (1 – 7) scale ranging from ‘do not agree’ (1) to
‘entirely agree’ (7). Most of the statements were created with a
double-jeopardy system, where each statement appears a second
time in reverse (i.e. The game was boring / The game was fun).
This was to ensure consistency within the answers and to flag
for any wild-card guessing.
The statements can be divided into four distinct groups:
1. General appreciation of the game concept – 13 statements.
2. Experience of presence and immersion – three statements.
-3-
3. Game controls and navigation – six statements.
4. General computer game habits – two statements.
3.2.2. Questionnaire – quantitative part
For the second part of the questionnaire the subjects returned to
the game application and were asked to describe six distinct
places in the game environment. They were asked to, for each
place, describe the physical environment plus the emotions it
evoked in as great detail as possible. All subjects described the
same six places and jumped from one place to the next by
clicking buttons in a short cut window presented by the test
leader.
Figure 2. Game application window (left) and short cut menu
used for the questionnaire (right).
3.3. Test Implementation
In the test, we let four separate groups of subjects first play the
Beowulf game and then fill out the questionnaire. The tests were
conducted on identical laptops (Apple MacBook Pro) with all
subjects using Koss Portapro headphones to ensure consistent
audio parameters.
The four groups comprised 12 subjects in each. Group 1 were
music conservatory students aged 20-25. Group 2 were high
school students from the south west of Sweden aged 13-15.
Group 3 were Art and Design students aged 20-25. Group 4
were high school students from the North east of Sweden aged
13-15.
We wanted to have a mechanism that could give us some
controlled indication on how much of the total experience come
from the game itself and how much depended on other factors.
One such mechanism could be the information given to the
subjects prior to playing the game. Each group of 12 was
divided into two groups: Six subjects were given only the
minimum of information necessary to navigate the game. The
other six subjects were given a longer and more emotive
description of the kind of terrain and fauna they were likely to
meet in the game. (It was thought that this would colour the
subject’s own, later description of the game experience).
Minimal instructions to play the game - English translation:
‘You will be playing an adventure game where your task is to
kill a monster hiding somewhere in the game world. You move
by using the arrow keys. With the up key, you take a step
forward in the same direction as the blue arrow. With the left
and right keys you rotate to change the direction you are going.
With the down arrow you swing your sword, the only weapon
you have. By listening to your breathing you can judge your
health status.’
More elaborated, emotive instructions for comparison - English
translation:
45
[Title of Paper]

English translation: 4.1.1. General computer game habits

‘You will be playing an adventure game where you task is to kill One surprise was that the majority of the test subjects did not
a monster hiding somewhere in the game world. You move by consider themselves regular game players, even though they all
using the arrow keys. With the up key, you take a step forward in fall within the normal demographic for regular gamers. Only 18
the same direction as the blue arrow. With the left and right keys of 48 responded 5, 6 or 7 to the statement ‘I often play computer
you rotate to change the direction you are going. With the down games’.
arrow you swing your sword, the only weapon you have. By
listening to your breathing you can judge your health status.
The game environment is a dark, blue grey system of caves and
passages filled with mist and noxious fumes. The cold and damp
caves are of different sizes and shape, some are slippery as ice,
others are filled with sharp stalactites and stalagmites. You will
meet dripping water; you will be walking on half-chewed bones,
remains of fossils and other nefarious things. The cave system is
peopled by animals and other creatures, both real and demonic-
Wolverines, bats, snakes and toothsome rats. The air is
repulsive and there is a sour and unpleasant smell.’

Each subject was given ca. ten minutes of playing time and was
then asked to fill out the first part of the questionnaire.
They were then instructed to return to the game environment
and by using the six short-cut buttons in the application, visit six
specific locations. These they then described in the second part Figure 3: response to statement: I often play computer games.
of the questionnaire.
4.1.2. General appreciation of the game concept
4. Results -Despite this lack of gaming experience the general reaction to
The subjects have filled out the questionnaires in a consistent the application was favourable. 27 of 48 subjects responded 5 or
manner, which makes us confidant that the test has worked and more to the statement ‘It was fun to play’:
that the results are reliable. A number of check-points supports
this:
• There is a strong negative correlation (-0,658) between the
two opposite-pole statements ‘I do not like to play
computer games’ and ‘I often play computer games’.
• A strong correlation (0,784) between the statements ‘This
is a game I want to improve on playing’ and ‘This is a
game I want to play more times’.
• Strong correlations between the statement ‘It was fun to
play’ and all of the following statements: ‘The game idea
(game play) is good’, ‘This is a game I want to improve on
playing’ and ‘This is a game I want to play more times’.
At the same time there is a relatively strong negative
correlation to the statement ‘I was bored early on’.
Generally there are strong correlations between the
statements in the category ‘General appreciation of the
Figure 4: response to statement: It was fun to play.
game concept’.
• The correlations between the statements in the category
-A high number unexpectedly thought the application would
‘Game controls and navigation’ indicates that the
make an excellent game for home computers despite its having
questionnaire have worked and contains valid data.
next to no graphic content.
The only category that do not show strong correlations between
statements are ‘Experience of presence and immersion’. This
category contains the most subjective and illusive of the 24
statements and the subjects may have had problems relating to
the statements given. On the other hand is the aspect of
immersion and presence covered in the qualitative second part
of the questionnaire, which makes this a smaller problem.

4.1. Results from part one of the questionnaire

The shear number of statements posed in the questionnaire (24),
mean that we will be presenting only a selection of the results in
this paper, and show only a few of the graphical representations
of the answer frequency. In these diagrams, blue bars indicate
males and pink bars indicate female. The figures at the bottom
of each bar indicates the instruction sub-group, where sub-group
1 was given only the minimum instruction and sub-group 2 was Figure 5: response to statement: This is an ideal game for a
given the more elaborated instruction. home computer.

-4-
46
[Title of Paper]
-This is especially interesting when compared to the same
question, but now focused on mobile phones (the intended
platform for the application).
Figure 6: response to statement: This is an ideal game for a
home computer.
It can also be seen that generally the female response to the
game was more consistent and slightly more positive.
4.1.3. Experience of presence and immersion
15 out of 48 test subjects did not find the audio environment
highly convincing but only 6 subjects gave the statement ‘I was
totally immersed in the game and the game task’ a score of 3 or
less.
4.1.4. Game controls and navigation
41 out of 48 test subjects responded that they were able to
understand and play the game immediately.
26 out of 48 were able to understand and react to the combat
situations without problem.
14 out of 48 stated that they tired quickly of the game, with a
spread of 4.5 to 7.
33 out of 48 expressed an interest in having the game on their
mobile phone. The strongest reaction to this was from group 4,
the High School students from the North-East of Sweden aged
13-15, with all 12 scoring positive on this.
It should be observed that this particular group has a highly
developed mobile phone social-culture and a number of the
subjects were simultaneously SMS-ing each other during the
running of these tests.
Generally the response to the application was favourable. The
negative responses the game received were focused around 5-6
individuals or ca. 12.5% of those tested.
4.2. Results from part two of the questionnaire.
There was a mixed response to the descriptive part of the
formula, with some subjects taking a long time to answer and
writing complex descriptions, to some subjects hardly reacting
at all.
Generally it can be said that nearly all of the subjects added
elements of their own creativity to the descriptions of the
environments.
These elements fell into five distinct groups:
1. Temperature (warm, cold, icy etc.)
2. Smell (unpleasant, rotten, damp etc.)
3. Colour (grey, red etc.)
4. Emotional content (lost & alone, scary etc.)
-5-
5. Descriptive (foggy, sunny, dark etc.)
There was no correlation between the quality and complexity of
the description and the information the groups had received at
the start. Both those subjects who had an emotionally
descriptive introduction and those that had a factual introduction
showed mixed levels of creativity in their interpretation of the
environments dependant on the individual. Since the amount and
nature of information prior to the game experience do not seem
the have any greater impact on that experience, it can interpreted
as the answers and responses to the questionnaire emanates from
the gaming experience itself and that very little coloration, if
any, resulted from the pre-information.
Only 2 subjects out of 48 used no emotive descriptive element in
the second part of the formula. These two described only what
they heard directly from the game application itself.
All others described at least some elements of temperature,
colour, smell etc. that emanated from their own interpretive
powers.
Some subjects showed an almost poetic interpretation of the
soundscape:
‘…a deep ravine that you can’t see the bottom of, beyond this
unending pine forests and high mountains…’
‘It’s freezing & smells of sand.’
‘A dove flies away from a birch tree, chased away by a large
crow.’
‘Smells like old-cut forest.’
‘Smells like blood and I am really scared of what is going to
happen.’
‘…smells like damp stone, maybe even concrete.’
‘…threateningly red, bloody, hard, someone is hiding from the
dogs…’
‘Wet ground, cold on my toes…tough!’
‘…damp, humid, smells like wet forest, green-grey & cool, fresh
air!’
‘…brown, grey with mossy/slimy walls…’
‘…weak red light…smells of bodies and death…’
‘…and you think a wolf is howling, but its only the wind!...’
The average answer was around 19 words with the shortest
being 3 words and the longest being 65 words.
5. Conclusions
5.1. Questions and answers
The test carried out has given indicative answers to the
questions formulated at the start of this project:
Q: Does an audio-based game make sense to a first time user?
A: The test showed that although the thresholds did vary from
subject to subject, there was no real problem with understanding
the game concept and executing said gameplay in the
application.
Q: Is an audio-based game a suitable vehicle to deliver game
content on a mobile platform?
A: Despite the fact that in practise there are outstanding
technical issues to be solved, the game present no conceptual
problems and was seemed suitable for a mobile platform by the
test group.
Q: Does the level of pre-game information colour the player’s
experience?
47
[Title of Paper]
A: The test results show no evidence for that the information
given prior to playing the game should color the game
experience. One interpretation of this is that the game
experience itself was strong enough to shadow the experience
from the, relatively short, pre-information phase of the test.
Q: Can an audio-based game give a satisfactory game
experience?
A: Although not to everyone’s taste, a significant percentage of
the test group were sufficiently positive towards the application
to confirm that it can.
Q: Does a non-visual game world present the player with
navigational problems or other hinders to a working game
experience?
A: The test did show that there are issues to be concerned about
regarding navigation, however these presented no major
problems for the test group and are not sufficient to sabotage an
application.
Q: Can a game emulate visual combat situations using only
sound?
A: The test showed that this was entirely feasible and only a few
of the test group expressed any problem in understanding and
successfully utilizing the combat gameplay.
Q: Can a player correctly interpret visual sounds without the aid
of visual reference?
A: The test showed that this was not an issue. The players might
vary in their personal interpretation, but at no time did these
variations interfere with the gameplay.
Q: Does a sound based environment encourage a player to
contribute more to the game experience?
A: The high gameplay marks suggest that the subjects were
prepared to accept a much lower level of gameplay content than
in a visually based environment. It should be noted that the
gameplay in Beowulf is neither particularly sophisticated nor
complex. Whether this impression was caused by the novelty of
the game environment or through the addition of self-generated
game content by the user is a discussion for another paper and
needs further user testing.
5.2. Feasibility of audio based games
We can observe that there are only small technical limitations to
creating audio-based games on personal computers. In this
project we have used a large number of existing and off the shelf
tools and technologies. The narrowest bottleneck for the area is
not technical but more related to lack of knowledge about sound
and sound design.
There are no limitations when designing audio based games
though there are added demands when solving problems where
one would normally rely on a graphic pointer.
As these games are highly interpretive, great care must be
shown in the design process, as it is only too easy to give
unfocused and misleading signals to the player through audio
content.
There are no real limitations to the (sighted) player’s potential
and ability to interpret and accurately play audio based games,
as long as the design process has been intelligently and correctly
executed. The test subjects had no preconceived negativity to
the idea of a non-visual game. And if there was any scepticism
prior to the test this was soon dispelled when the game was
played.
5.3. Ability of subject to navigate play
-6-
Despite the game having minimal instructional information and
a very limited graphic interface and feedback, the game was
played successfully by all subjects…although there were
different levels of entry with some subjects grasping the concept
immediately and others needing some minutes to grasp the
principles of navigation and concept, all subjects were able to
control the character, use the sword and move through the
environment. Most of the test subjects were even able to
complete the set task and ‘kill’ the monster.
5.4. Problems illuminated by this test
Paradoxically, the principle problems with audio exclusive
games lie in their principle strengths. The ability of audio
applications to trigger self-generated, complimentary visual
content in the user also means that the game designers have little
control over what the user is ‘seeing’. This places great demand
on the game play and sound design. This, in turn, highlights the
need for a sound design methodology in this area. An absolute
majority of the work carried out in this field is still
experimental. With an informed set of “do’s and don’ts” the
work could be lifted to higher levels.
6. Acknowledgments
Stefan Lindberg, Interactive Institute Sonic for all the sounds.
Martin Nordlinder for dedicated work with software
development.
Pupils and staff at Central Skolan, Arvika
Students at Ingesund University College of Music
Pupils and staff at Carlshöjd School Umeå
Students at Umeå School of Art
Stuart Cunningham for advice, help and feedback
References
[1] Gaver, W. W., Beaver, J., and Benford, S. 2003. Ambiguity
as a resource for design. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (Ft.
Lauderdale, Florida, USA, April 05 - 10, 2003). CHI '03.
ACM, New York, NY, 233-240. DOI=
http://doi.acm.org/10.1145/642611.642653
[2] Sengers, P. and Gaver, B. 2006. Staying open to
interpretation: engaging multiple meanings in design and
evaluation. In Proceedings of the 6th Conference on
Designing interactive Systems (University Park, PA, USA,
June 26 - 28, 2006). DIS '06. ACM, New York, NY, 99-108.
DOI= http://doi.acm.org/10.1145/1142405.1142422
[3] Liljedahl, M., Papworth, N., and Lindberg, S. 2007.
Beowulf: an audio mostly game. In Proceedings of the
international Conference on Advances in Computer
Entertainment Technology (Salzburg, Austria, June 13 - 15,
2007). ACE '07, vol. 203. ACM, New York, NY, 200-203.
DOI= http://doi.acm.org/10.1145/1255047.1255088
[4] Lumbreras, M. and Sánchez, J. 1999. Interactive 3D sound
hyperstories for blind children. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems: the CHI Is the Limit (Pittsburgh, Pennsylvania,
United States, May 15 - 20, 1999). CHI '99. ACM, New
York, NY, 318-325. DOI=
http://doi.acm.org/10.1145/302979.303101
[5] Röber, N., Masuch, M. 2004. Auditory game authoring.
DOI= http://games.cs.uni-
magdeburg.de/audio/data/Roeber_2004_AGA.pdf
48
[Title of Paper]

[6] Friberg, J. and Gärdenfors, D. 2004. Audio games: new

perspectives on game audio. In Proceedings of the 2004
ACM SIGCHI international Conference on Advances in
Computer Entertainment Technology (Singapore, June 03 -
05, 2005). ACE '04, vol. 74. ACM, New York, NY, 148-
154. DOI= http://doi.acm.org/10.1145/1067343.1067361

-7-
49
 Control of Sound Environment using Genetic Algorithms
Scott Beveridge, Don Knox,
Glasgow Caledonian University, Glasgow, Scotland
scott.beveridge@gcal.ac.uk, d.knox@gcal.ac.uk.

Abstract. Sonification - the production of sound to represent some form of data or information, has been
applied in various fields including analysis of financial, meteorological and physiological data. A system overview
is presented which is based on analysis of sociospacial behaviour via video capture of a given environment. The
example given is a busy commuter environment such as a train station. Activity in the environment is mapped
to a set of musical performance parameters, and also forms the basis for controlling an optimisation function
based on genetic algorithms (GA). The aim is to develop a socially reflexive audio environment, where those
present unconsciously interact with the input, and output, of the sonification process. The output of the system
is a series of musical chords, optimised as regards their musical fitness as defined by three consonance criteria.

1 Introduction concentrates upon sonification of gestural information

- specifically socio-spatial behaviour. The aim is to
Sonification can be used as a tool to enhance our un-
develop a system that unconsciously influences passive
derstanding of abstract data. It is particularly useful
users, hence creating a socially reflexive environment.
in applications where the complexity of data is likely to
It is hoped that by carefully mapping input parame-
cause visual overload, or is at a level of abstraction that
ters, audio information can be used to create an envi-
requires some form of translation to make it useful. Ex-
ronment where human interaction with the system is
amples of the data sets that have been subject to soni-
self controlling. Ideal situations where this may be ap-
fication are financial [1], meteorological [2, 3] and phys-
plied include large sensate environments like train sta-
iological [4, 5]. Data sets can be categorised as those
tions, where during rush hour hundreds of commuters
which relate to remotely located data internet traffic,
could be subject to sedate music in order to provide a
financial indices, weather patterns, or ‘localised data’
more relaxed commuting environment. A system such
- skin conductivity, brain patterns, gestural informa-
as this raises interesting questions about the nature of
tion [5]. To produce meaningful interpretation of input
performance. Traditionally, groups of musicians build
data it is necessary to implement a schema or mapping
upon relationships and communication to facilitate the
framework. In its simplest form the framework can
creative process of making music. The system explores
be based on direct linear relationships between move-
the concept of a large scale musical performance where
ments and musical themes [6], however more compre-
the communication and interaction takes place between
hensive and versatile mapping results when an algo-
‘musicians’ who are unaware of their role in the pro-
rithmic design approach is taken [7]. Algorithmic de-
cess. As well as being of interest from a performance
sign encompasses many methodologies. These include
perspective, it is possible that processing data in this
stochastic and serial techniques - which manipulate
way will reveal patterns of information which are not
pre-defined musical parameters using either random
immediately evident using traditional statistical anal-
or structured processes, and generative processes such
ysis methods [8].
as those based upon genetic transformations and cel-
lular automata. Generative processes employ organic
techniques which produce unique, sonically rich mate- 2 The System
rial [5]. Past implementations of this approach have
2.1 Data capture
concentrated mainly on aesthetic uses. These involve
the active interaction of participants with the system Image processing is used as a means of extracting the
in the context of a performance environment. Situa- necessary information from a particular environment.
tions of this type see individuals, influenced by aural Input data is obtained with a visual capture system
feedback, becoming aware of the system and seeking suspended above the target space (see Figure 1). The
to then modify the source information based on their plan view provided by this arrangement gives wide ge-
subsequent actions [5]. ographic coverage and sufficiently detailed movement
This paper presents initial work on a system which information that allows observations to be made in a

50
[Control of Audio Environment using Genetic Algorithms]
wide range of locations.
Figure 1: Plan view from webcam
Video frames are captured using an inexpensive web
cam, and manipulated using Matlab image process-
ing tools to remove unnecessary information (floor de-
tail and static objects). This type of approach has
previously been used only in small scale performance
contexts, requiring specialised motion capture systems
[9, 5]. Subsequent image processing (see Figure 2) and
statistical analysis allow extraction of various param-
eters which describe the nature of movement in the
environment (see section 2.2).
Figure 2: Figures identified by vision capture system
2.2 Mapping
In order to acheive sound output that is in some way in-
dicative of events occuring within the environment be-
ing studied, correspondences must be formed between
measured behavioural activity and musical variables.
Wherever possible direct correspondences between the
data domain and sound domain should be made. How-
ever the data may not display such immediately obvi-
ous relationships, and in such cases metaphors may
be used [10]. The system draws parallel data streams
that derive direct relationships with dynamic intensity,
tempo and timbre and translates more complex rela-
tionships by statistical analysis and a generative algo-
rithm.
2.2.1 Acoustical parameters
In line with work carried out in gestural interaction [5]
a framework for direct sonification activities has been
implemented – see table 1.
Activity/Trigger Sonification
Number of people Dynamic Intensity
Speed of movement Tempo
Direction of movement Timbre
Table 1: Map of environment activity to sonifica-
tion parameter
This framework provides a comprehensive but po-
tentially complex set of acoustical parameters which
represent the activity within the environment. To
simpify these variables and so make the data set more
useful a higher level representation is required. The
objective is to obtain a qualitative measure of these
features in order to simplify audio generation. A re-
cent study by Lu et al [11] has been adopted to provide
this representation. The research involves a framework
for the emotion classification of music which examines
audio on the basis of the features extracted in Table 1.
The system uses signal analysis techniques to extract
intensity, timbre and rhythm (ITR) features which are
used in a classification process based on guassian mix-
ture models (GMM). The result is music which is la-
belled in terms of the two-dimensional stress-energy
model proposed by Thayer [12] (see Figure3). The
model, which was adapted from the original proposed
by Russell [13], forms quadrants equating to content-
ment, depression, exuberance and anxious/frantic.
Classifying features in this manner provides a con-
text for further decisions regarding music generated by
the system as the emotion labels are directly analagous
to the activity with the environment. This represen-
tation will not be sonified directly, instead the system
generates audio with opposing features to counteract
the activity within the environment.
-2-
51
[Control of Audio Environment using Genetic Algorithms]

e
ner
gy/
arous
al parameters. In an experiment where music with vary-
ing tempo and loudness was played as background mu-
sic within a supermarket environment it was found that
in-store traffic flow could be modified with fast tempo,
e
xube
ranc
e a
nxi
ous
/fr
ant
ic high intensity music proving the most effective in in-
creasing the passage of consumers through the sales
space. The system proposed in this paper can not only
s
tre
ss/
val
enc
e
dynamically alter these musical variables in real time
(and in response to user input) but also incorporate
structural musical features which results in novel au-
dio with the potential of controlling group behaviour.
c
ont
ent
ment de
pre
ssi
on
2.2.2 Structural Features
The mood classification scheme proposed by Lu et al is
based solely on acoustical parameters and does not take
into account more complex structural features which
may attribute to emotional expression. Musical as-
Figure 3: Thayer’s stress/energy model, showing the
four mood clusters; contentment, depression, exuber- pects such as mode, interval and melodic contour are
ance and anxious/frantic necessary in expressing emotion and provide a basis
for higher cognitive constructs associated with music
listening.
It has long been established that expectation plays
In our busy commuter environment we may observe
a pivotal role in a listener’s experience of music [20].
a large number of individuals moving quickly in a uni-
This theory is based on the assumption that while lis-
form direction. This behaviour indicates high intensity,
tening to music an individual will form expectancies
fast tempo parameters with distinct timbre character-
about its continuation. If these presuppositions are
istics dependent on direction. The classification frame-
violated it evokes a corresponding emotional reaction
work proposed by Lu et al (figure 4) places audio with
[21]. Narmour proposed a refinement of this concept
these features within the exuberance category. The
in the implication-realization (I-R) model [22] which
system will use this classification outcome to construct
judges melodic expectancy based on the Gestalt-based
audio with opposite characteristics and so generating
principles of proximity, similarity and good continua-
audio with features which would place it in the con-
tion. The I-R model is based on a three note archetype
tentement category and produce an affective balance
which focusses on the distance and direction relation-
to the original observed behaviour.
ships between the intervals. By measuring melodic
Considering musical features in terms of emotion is contour on the basis of these principles the model has
useful for a number of reasons. Studies conducted on been found to predict melodic expectancy with reason-
group psychology and consumer behaviour suggest that able accuracy.
individuals are very sensitive to changes in environ- This system copes with these structural aspects
ment [14, 15] and that music is a critical factor in their by implenenting an algorithmic approach. Parame-
construction. Spatial aesthetics or ‘atmospherics,’ is ters within the genetic algorithm are modified in real-
the term used to describe the concious designing of time in a mechanism which seeks to optimize structure
space to create certain effects in buyers [16]. Based within specific parameters. The emotion classification
on atmospherics indivduals are likely to display one of process plays an essential part in this process by ensur-
two behaviours. Approach behaviour involves such re- ing a context with which to voilate expectancies and
ponses such as physically moving toward something, hence generate emotion. The initial implementation
affiliating with others in the environment through ver- of this system uses this framework to optimize chord
bal communication and eye contact, and performing triads which form the basis for generated audio.
a large number of tasks within the environment [17].
Avoidance or negative reinforcement behaviour [18] in-
3 Sound Generation
cludes trying to get out of the environment, a tendency
to remain inanimate in the environment, and a ten- The system is built upon an evolutionary design model
dency to ignore communication attempts from others inspired by the Survival of the Fittest concept proposed
[19]. A study by Millman [15] shows how manipulation by Charles Darwin in 1859. In biological terms, this
of these responses can be acheived by altering musical relates to the competition for predominance amongst

-3-

52
[Control of Audio Environment using Genetic Algorithms]

stage where crossover and mutation take place. The re-

sultant children are then subject to same fitness mech-
anism which identifies the fittest subset and the pro-
cess continues. The goal is to incrementally increase
the overall fitness of the individuals with the aim of
achieving the best possible result in a specified time-
frame. The outcome is an optimum chord (as regards
the consonance criteria) which is output as MIDI data.
Figure 4: The hierarchial mood detection framework The optimisation process is influenced by parameters
of Lu et al provided by the statistical analysis module, which have
been mapped from data provided by the visual capture
peers. In the field of mathematics it is more com- system.
monly known as an optimisation problem. Optimisa-
tion problems employ specialised heuristics to exam-
4 Initial Results
ine large, potentially infinite solution spaces with the
aim of finding optimum results. The system uses a The chord structures generated by the system reflect
genetic algorithm methodology for optimisation pro- the current state of the environment being observed
posed by Manzolli et al [23]. Genetic algorithms apply via vision capture. The consonance measures between
the biological processes of reproduction and selection the notes of each chord, tend towards musically stable
to a population (or solution space) to achieve optimi- intervals (3rds and 5ths) in favourable conditions (for
sation. In the system the initial population consists example very crowded conditions during rush hour),
of randomly generated four note chord sequences. In and dissonance when less favourable. The optimisation
keeping with the original implementation [23], these process can be further controlled by altering initialisa-
chord sequences are assigned a fitness value based on tion parameters which affect the initial population size
three specific consonance criteria: harmonic, melodic and maximum number of generations. Figure 5 shows
and voice. These consonance criteria are based on the an initial population size of 30 undergoing 15 genera-
idea of coincidence of harmonics [24] and relate to the tions of evolution. Both the average population fitness
Gestalt principles of proximity and similarity. This and best individual fitness tend towards a maximum
method examines the first 16 harmonic components of value as the optimisation process progresses. The cri-
each individual note including the fundamental to give terion upon which the optimisation process is based
a measure of the quality of note intervals. is completely dynamic and depends upon behavioural
• In the case of harmonic consonance the four notes information as defined by human activity in the envi-
in the chord are compared with each other and ronment measured by the vision capture system.
a score is assigned based on the commonality of
their spectral components.
• Melodic consonance is used to find the chord
which best relates to a given input note. This is
achieved by comparing the harmonic components
of each note in the chord with the input note.
After all four comparisons are made the chord is
given a score which relates to the note with the
maximum overlap value.
• Voice consonance describes the relationship be-
tween the elements of the chord with respect to
a set of note ranges. These sets are defined by
a fuzzy set formalism which assigns scores based
on individual membership functions. Voice conso-
nance ensures adequate spacing between the notes
and hence a tendency towards consonant intervals
of 3rds and 5ths.
These criteria are added together to determine the
overall fitness of each individual. The fittest individ- Figure 5: Population evolution
uals are then chosen to progress to the reproduction

-4-

53
[Control of Audio Environment using Genetic Algorithms]

5 Future Work classification strategies will be developed which incor-

porate both acoustical and structural features. De-
The current implementation of the system creates a
velopment at this level will lead to more emotionally
unique and distinct evolutionary cycle for every dis-
relevant representation and more accurate forcasting
crete set of parameters captured from the behavioural
of crowd ‘feelings’ which will result in a more effective
information observed in the environment. Future ef-
self controlling mechanism. The unification of these
forts will look toward the creation an initial popula-
factors forms the basis for an EPRSC-funded project
tion which will evolve during the entire life of the pro-
with which the authors are involved. In addition, the
cedure. Processing larger populations would allow ad-
role of extra environmental parameters will be exam-
ditional evolutionary parameters to be altered, in ef-
ined. These could be variables that are not specifically
fect the population would grow in parallel with (and
related to human behaviour, but which influence the
in response to) the changing behavioural data at the
environment. For example environment temperature,
input. So in addition to fitness criteria, further evolu-
humidity, light level and ambient noise can be mea-
tionary criteria such as population bounds, mutation
sured and included in the generative process.
criteria, cross-over method and selection function could
be modified. Population manipulation in this manner
would result in more smooth and continuous musical References
gradients, resulting in musical output with smoother [1] F. C. Ciardi, “smax: A multimodal toolkit for
transitions between themes. stock market data sonification,” Proceedings of
ICAD 04, Tenth Meeting of the International
Conference on Auditory Display, Sydney, Aus-
I
nit
ia
lPopul
at
ion tralia, 2004.
[2] N. Reeves, “The cloud harp project.” Available at
http://www.cloudharp.org/index.htm, Accessed
April 10th, 2006.
Ge
net
icAl
gor
it
hm
Fi
tnesscr
it
eri
on [3] A. Polli, “Atmospherics/weather works: A multi-
i
nfl
uencedby channel storm sonification project,” International
Conference of Auditory Display (ICAD), Tenth
st
at
ist
ica
lcrowd
Meeting of the International Conference on Au-
anal
ysi
s ditory Display, Sydney, Australia, 2004.
[4] E. Miranda and A. Brouse, “Toward direct brain-
computer musical interfaces,” Proceedings of the
2005 conference on New interfaces for musical ex-
pression, Vancouver, Canada, pp. 216–219, 2005.
[5] K. Beilharz, “Wireless gesture controllers to af-
fect information sonification,” ICAD, Limerick,
Modi
fi
edPopul
at
ion Ireland, CD-ROM proceedings July, 2005c, 2005.
[6] H. Lrstad and J. Eacott, “The intelligent street:
responsive sound environments for social interac-
Theopt i
mumor tion,” Proceedings of the 2004 ACM SIGCHI In-
Fi
nalPopul
at
ion ‘
fi
tte
st’chordofthe ternational Conference on Advances in computer
gener
a t
iveproce
s s entertainment technology, pp. 155–162, 2004.
[7] K. Beilharz, “Criteria and aesthetics for mapping
Figure 6: Proposed system for evolution of the intial social behaviour to real time generative structures
population for ambient and auditory display (interactive soni-
fication),” INTERACTION - Systems, Practice
A main area for future investigation is the need to and Theory: A Creativity and Cognition Sympo-
ensure the musical output of the sonification process sium, Sydney, Australia, pp. 75 – 102, 2004.
is meaningful. This is in keeping with the stated goal [8] J. R. Minkel, “The computer
of creating a socially reflexive environment - where in minds the commuter.” Available at
this example, the aim is to create a more relaxed envi- http://focus.aps.org/story/v13/st26, Accessed
ronment for the commuter. Toward this, new emotion October 13th, 2006, 2004.

-5-

54
[Control of Audio Environment using Genetic Algorithms]

[9] M. Droumeva and R. Wakkary, “Sound intensity [24] H. Helmholtz, On the Sensations of Tone.
gradients in an ambient intelligence audio dis- Thoemmes Continuum, english ed 1875 edi-
play,” Conference on Human Factors in Comput- tion ed., 1875.
ing Systems, pp. 724–729, 2006.
[10] A. de Campo, C. Frauenberger, and R. Holdrich,
“Designing a generalized sonification environ-
ment,” Proceedings of the ICAD, 2004.
[11] L. Lu, D. Liu, and Z. Hong-Jiang, “Automatic
mood detection and tracking of music audio sig-
nals,” Audio, Speech and Language Processing,
IEEE Transactions on, vol. 14, no. 1, pp. 5–18,
2006.
[12] R. E. Thayer, The Biopsychology of Mood and
Arousal. Oxford University Press, 1989.
[13] J. A. Russell, “A circumplex model of emotions,”
The Journal of Personality and Social Psychology,
vol. 39, no. 6, pp. 1161 – 1178, 1980.
[14] R. E. Milliman, “Using background music to affect
the behavior of supermarket shoppers,” Journal of
Marketing, vol. 46, no. 3, pp. 86–91, 1982.
[15] R. E. Milliman, “The influence of background mu-
sic on the behavior of restaurant patrons,” Jour-
nal of Consumer Research, vol. 13, no. 2, p. 286,
1986.
[16] P. Kotler, “Atmospherics as a marketing tool,”
Journal of Retailing, vol. 49, no. 4, pp. 48–64,
1973.
[17] B. H. Booms and M. J. Bitner, “Marketing ser-
vices by managing the environment,” Cornell
Hotel and Restaurant Administration Quarterly,
vol. 23, no. 1, p. 35, 1982.
[18] M. M. Bradley, “Emotion and motivation,” Hand-
book of psychophysiology, vol. 2, pp. 602–642,
2000.
[19] R. J. Donovan and J. R. Rossiter, “Store atmo-
sphere: an environmental psychology approach,”
Retailling: Critical Concepts, 2002.
[20] L. B. Meyer and L. B. M. Meyer, Emotion and
meaning in music. The University of Chicago
Press, 1956.
[21] D. Huron, Sweet anticipation: music and the psy-
chology of expectation. MIT Press, Cambridge,
Mass., 2006.
[22] E. Narmour, The Analysis and Cognition of Basic
Melodic Structures: The Implication-Realization
Model. University of Chicago Press, 1990.
[23] J. Manzolli, A. Moroni, F. Von Zuben, and
R. Gudwin, “An evolutionary approach ap-
plied to algorithmic composition,” Proceedings
of VI Brazilian Symposium on Computer Music,
p. 201210, 1999.

-6-

55
 Genie in a Bottle: Object-Sound Reconfigurations for Interactive
Commodities
Daniel Hug
Interaction Design Department
Zurich University of the Arts
daniel.hug@zhdk.ch

Abstract. Everyday commodities are increasingly enhanced with information or communication technologies and become more
complex and interactive. Electroacoustic sound can be a powerful way of shaping the aesthetic and functional aspects of such artifacts
beyond the modification of the physical aspects. This is a challenge for sound design, which traditionally has prospered mainly in
linear audiovisual media. The nature of objects requires new approaches to sound design, both because physical objects and sound
have a close relationship and also because artifacts are embedded in complex sociocultural contexts. Thus sound design for interactive
commodities is one element that contributes to the hermeneutic affordances of complex, interactive commodities. This paper outlines
the sociocultural significance of objects, their relationship with sounds, and how sound design can re-configure this relationship.

1 Introduction even R. Murray Schafer could not have anticipated3 . Tradition-

Fueled by recent developments in information and communica-
tion technology (miniaturization, increase of processing power,
reduced costs), cultural changes (information society, nomadic
lifestyle, merging of work and leisure) as well as the ever-
increasing need for new markets, the vision of the ”ubiquitous
computer”, aptly termed ”everyware” by Adam Greenfield, is be-
coming everyday reality [19]. Literally every thing is a possible
resource for deploying new technologies and interactivity. Our
everyday life is permeated by networked technologies and com-
plex, seemingly autonomous, devices. Some of these devices op-
erate entirely without human intervention, others require some de-
gree of operation or manipulation. The latter we call ”interactive
commodities”. The term commodity stands for the pragmatic, ev-
eryday nature of these artifacts [20].
1.1 The Role of Sound for Interactive Commodities
Studies dealing with the probable future ”prime fathers” of in-
teractive commodities, the mobile phone and the personal digital
assistant (PDA), suggest that sound is going to play an impor-
tant role in the design of small, disappearing, ubiquitous devices.
For instance, Brewster describes strategies to overcome the lack
of screen size limitations with sound [10]. In general, sound is
a powerful means to communicate information about continuous
or dynamic processes, to supply ambient information related to
a place or activities, to improve an ongoing continuous control
activity and to shape the functional and aesthetic experience of
artifacts [16]. Therefore its application in interactive objects of
everyday use suggests itself and is investigated in initiatives like
the European COST IC0601Action on Sonic Interaction Design1
or the European research project CLOSED2 , which aims at de-
veloping suitable measurement tools and criteria for the design of
sound of interactive artifacts.
1.2 A New Challenge for Sound Design
Thanks to miniaturization of electroacoustic components sounds
can be integrated relatively easily into interactive commodities.
This creates possibilities for a new kind of ”schizophonia” that
ally, the work done in the context of auditory display research fo-
cuses on functional aspects and how they can be related to sound.
The research and the resulting sound design criteria are geared to-
wards efficient communication of system or data status or changes
and are mostly concerned with desktop computers or highly spe-
cialized application domains such as medicine or geology [24].
Our hypothesis is that the nature of interactive commodities
entails new, largely unstudied aspects, and that new criteria for
sound design are required. What these aspects and criteria are,
how they can be studied, and finally, how guidelines for design
can be derived from this study, remains to be investigated. This
paper outlines these aspects and shows possible directions for fu-
ture research and design.
2 Meaningful Objects
It suggests itself that any study on how sound can be designed to
create meaningful interactive objects should include an in-depth
understanding of the role objects play in the everyday life of in-
dividuals and society. This is of course a vast domain for study
and difficult to frame. The following presents a few spotlights that
demonstrate the importance of understanding how objects become
meaningful and how this relates to sound design. The elabora-
tions are not concerned too much with details like formal aspects
of objects and focus on higher level cultural implications.
2.1 Semiotic Approach
From a semiotic point of view objects are either the correlative
of signs or they are signs themselves. Nöth states that artifacts
have emblematic (meaning oriented) as well as non-emblematic,
pre-semiotic (practical, use-oriented) aspects. According to some
semioticians also the latter is seen as semiotic aspect. Nöth further
explains that according to Heidegger artifacts carry references to
the practical use, the determination to be used for something, im-
printed in them. Practical things can also be understood as refer-
ring to their value of use as indices. Such primary and denotative
meanings can be accompanied by secondary, e.g. aesthetical or

1 See http://www.cost-sid.org 3 Schizophonia is the term coined by R. M. Schafer to denote the sepa-
2 Closing the Loop on Sound Evaluation and Design, see ration of sound from their - natural - sources by means of electroacoustics.
http://closed.ircam.fr and [28]. For him this concept carries only negative connotations [31].

56
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities
connotative, meanings [27]. In his semiotic writings, Barthes of-
fers two broad connotational categories for objects. The first is
the ”existential” connotation, which understands objects as obsti-
nate, inhuman or even antihuman. The other is the ”technolog-
ical” connotation which is concerned with objects as consumed,
reproduced and functional products [4].
A special aspect in the creation of meaning in objects is ma-
terial. In his dialogue ”Hippias Maior” Plato describes a discus-
sion between Socrates and Hippias about what makes the beauti-
ful beautiful. One proposition is that certain materials like gold
make ordinary things beautiful, another proposition is that some
materials are more functional and thus make an object appropri-
ate (cited in [9]). Materials like marble and (fake) gold suggest
a royal or noble atmosphere in shopping malls, wood can signify
closeness to nature, and so forth ([9], [6]).
2.2 Sociocultural Objects
Artifacts play an important role on a sociocultural level. Based on
interviews with over 300 people Csikszentmihalyi and Rochberg-
Halton investigated the meaning of everyday artifacts beyond
mere functionality. They describe how objects become part and
are the result of the process of cultivation, that is ”the process
of investing psychic energy so that one becomes conscious of the
goals operating within oneself, among and between other persons,
and in the environment” ([14], p. 13). According to Csikszent-
mihalyi and Rochberg-Halton things embody goals, make skills
manifest and shape the identities of their users. Objects thus are
embodiments of intentionality. Hence they alter patterns of life,
they reflect and define the personality, status, social integration
etc. of both, producer and owner. In addition, things evoke emo-
tions through interpretations in the context of past experiences,
thus becoming signs or symbols of one’s attitude. Moreover they
can mediate conflicts within the self.
Through the ”objectivity” and permanence of objects such
identities can be shared. This is the precondition for the social-
izing effect of things and of their ability to provide role mod-
els. Csikszentmihalyi and Rochberg-Halton also shed light on
the role of the functional aspect of objects for culture, describing
how even the use of things for utilitarian purposes is inseparable
from the symbolic context of culture. Artifacts socialize people
to certain habits or ways of life and represent these as signs [14].
Socio-semiotic studies describe the practice of collecting, the
accumulation of objects as dowry, the expression of self in rela-
tion to society in bricolage, the rhetoric of the displayed artifact,
shopping centers as super-objects and stages for objects and social
actions, the mystification and commodification of culture through
souvenirs, and the complex culture around jewels. An excellent
collection of essays related to these questions can be found in
[29]. Baudrillard finally points out the relationship between ob-
jects, production systems and society, describing the development
of artifacts from a static role in a traditional pre-modern society
to their emancipation to flexible, functional entities, and how sys-
tems of objects and their consumption reflect socio-ideological
circumstances [6]. All these aspects contribute to the meaning
making related to artifacts.
2.3 Beyond Semiotics
It shines through some of these points that semiotics in a struc-
turalistic sense is limited as an analytical tool when dealing with
the world of concrete physical objects. Barthes states that the
semiotic study of objects is still at an early stage, one reason being
that no pure significant system of objects exists without the inter-
-2-
vention of language. Thus, he resorts to language as paradigm for
understanding objects [4]. Objects challenge the semiotic idea of
communication by signs and code through their immediate, com-
plex, archetypical multi-sensorial presence. They are not delim-
ited to relatively restricted semiotic frameworks like media such
as film, photography or even architecture. It is not surprising that
many accounts of semiotics of objects actually refer to the presen-
tation of objects in media such as advertisements (see e.g. [4]).
Joan M. Vastokas calls for a ”semiotics of visual phenomena”
(and we shall add auditory phenomena), taking into account the
full spatial, temporal, and gestural dimensionality of the artifact.
She proposes a narrative concept of artifact, understanding the
procedural component of an artifact as being born through inten-
tionality of its creator(s), going through a life of use and abuse
as ”meaningful and expressive object in itself, and as a ritual per-
former in social and cultural life”, finally ”dying” and being dis-
posed, exhibited in a museum or recycled ([37], p. 341). She
describes several essential points to consider in the study of arti-
facts from a sociocultural perspective:
”(1) The meaning of artifacts, including works of
visual ”art”, is constituted in the life of the objects
themselves, not in words or texts about them; (2) the
artifact is not an inert, passive object, but an interac-
tive agent in sociocultural life and cognition; (3) the
signification of the artifact resides in both the object as
a self-enclosed material fact and in its performative,
”gestural” patterns of behavior in relation to space,
time, and society; (4) the processes, materials, and
products of technology, especially those of a society’s
dominant technology, function as cultural metaphors
at many levels and in many sociocultural domains; and
(5) theoretical insights derive, not from theorizing in
the abstract, but from direct observation and experi-
ence of the phenomenal world of nature and culture.”
([37], p. 337)
2.4 Active, Animated and Magic Objects
These considerations lead to an aspect of the study of artifacts
which seems to be counterintuitive at first glance: The notion of
the object as actor. Latour proposes in his actor-network theory
(ANT) that human beings and non-human or even inanimate ob-
jects are interlinked in mutual interactions, both being actors in
the process. In this view, action is not limited to what intentional,
meaningful humans do. According to Latour any thing that does
modify a state of affairs by making a difference is an actor [26].
A theoretical approach which is somewhat related to ANT, but
more focused on human agency, is activity theory, introduced by
Leontiev in the late 1970’s. Instead of departing from a relational
approach (between symmetric nodes in networks, ”actors”, that
can be people, machines, or other things) activity theory proposes
a primacy of activity over both object and subject, originating in
purpose, need and intentionality. As for the role of artifacts, they
are described as the product of cultural needs, embodying our in-
tentions and desires. Thus they mediate between people and the
world and in this sense things have agency [22].
What has been outlined in these reflections seems to come to
a new level in interactive commodities. In her book ”The Sec-
ond Self” Sherry Turkle described the computer as an ”evocative
object for thinking about human identity”. It serves as projection
surface for our desires and fears and is often considered animate in
some way - not only by children - exhibiting behaviors indicating
some kind of reasoning and agency. The main criterion for alive-
57
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities
ness, autonomous motion, is being replaced or extended by the
notion of psychological autonomy. Looking at the more recent de-
velopments in the 20th Anniversary Edition of ”The Second Self”
Turkle states that these characteristics have become commodified,
are part of our everyday experiences. But although computational
technology has lost much of its uncanniness we still tend to per-
sonify and project our self onto it. In the case of the increasing
autonomy and complexity of computational artifacts this tendency
becomes even stronger. And in some cases like wearable comput-
ers or computational implants the border between computer and
human is increasingly blurring on a very concrete level [35].
According to Daniel Chandler the quality of purposiveness and
autonomy in artifacts arises from the whole being more than the
sum of the parts when technology becomes too complex to con-
trol. Technological artifacts seem to have a will of their own and
we tend to anthropomorphize them. The resulting technological
animism credits an inanimate entity with consciousness and will4
[11].
From here the step to a notion of a magical quality in complex
computerized artifacts can be easily made. According to Arthur
C. Clarke’s third law ”any sufficiently-advanced technology is in-
distinguishable from magic.”5
3 The Relationship Between Sound and Objects
We have stated above that the study of objects in all their com-
plexity matters for the sound design of interactive commodities.
This is the case both because sounds and objects are often closely
related and because of the power of sound to modulate an objects
identity. Moreover sounds can evoke a certain object and thus its
sociocultural significance described above.
In the following a closer look at this relationship is provided.
3.1 Sound and Physical Properties
Sounds are directly connected to an artifact’s physical properties.
In our everyday experience the acoustic properties of materials
and objects provide us with information on their quality. We
might see a transparent object, but we will only be able to tell
whether it is glass or acrylic after tapping it with our finger. We
can distinguish metals in the same way or determine whether a
piece of wooden furniture is made of hardwood or plywood or
whether it is in good or bad condition. Many studies have been
concerned with the ability of detecting material properties, shape
and size through sound, as well as processes of interacting mate-
rials (see e.g. [18] for a comprehensive overview). Nonetheless,
many aspects of recognition are still not completely understood,
as most of these studies deal with simplified sound events, such
as mallets hitting metal plates [38].
Opposed to typical laboratory setups, the sounds we hear in ev-
eryday life are often composites of several sound sources. Each
single vibration in the human hearing range merges into one
sound gestalt and is perceived as a complex entity. With increas-
ing complexity, the actual elements that cause a sound are not
discerned at all anymore. In what is called everyday or ecological
4 In our everyday myths and narratives this topic reappears often under
a somewhat humorist veil, also referred to as ”resistentialism”. This term,
coined by Paul Jennings, stands for a humorous theory in which inanimate
objects display hostile desires towards humans, a ”fact” apparent in expe-
riences such as cars not starting when one’s in a hurry or the bread always
falling on the side with butter on it.
5 To be found in a 1973 revision of his compendium of essays, ”Profiles
of the Future”.
-3-
listening we hear an approaching car rather than four wheels, an
engine and the various vibrations of the car’s body [17].
In terms of perceiving physical qualities, the intersensorial link
between audio and haptics is quite strong. For example, Kayser et
al. demonstrated somatosensory and auditory interaction and the
conditions for its effectiveness, namely temporal coincidence and
inverse effectiveness [23]. And Avanzini and Crosato have suc-
cessfully demonstrated how sound can modulate the haptic per-
ception of stiffness [1]. Some studies deal with the audio haptic
relationship in food consumption, for example the factors asso-
ciated with judging apples to be mealy [3]. In the less scientific
but nevertheless relevant domain of film and game design, sound
often is used to substitute or denote haptic sensations of protag-
onists, and is essential for suggesting authenticity of objects on
screen and suspending disbelief [15].
3.2 Critique of Sonic Causalism and Naturalism
But the relation between sounds and physical objects and pro-
cesses is more complex. On closer investigation we can discover
a dialectical relationship between objects and sounds: While ob-
jects are permanent and concretely graspable, sounds are tempo-
rary and evasive, yet still they often have an almost intimate rela-
tionship with the physical world, which is the basis for the natural-
izing effect of sound in media. But the natural link between sound
and causing physical event has to be questioned. Sounds also have
an existence which is detached from their original source, am-
bivalent, sometimes carrying a rather vague notion of material in
them, sometimes being totally abstract. Chion criticizes the sim-
plistic assumption of the (scientific) discourse about sound ”nat-
urally” representing a certain cause. This myth leads to a general
research focus on sounds that are empirically verified as ”well
identified” and thus supposedly meaningful and useful for design.
However, the list of ”well identified sounds” is relatively short,
often restricted to archetypes or clichés, and very case dependent.
Their successful identification depends on a specific recording of
a specific cause (e.g. a slamming door). And not a small number
of the sounds in such a list are well identified because we have
learned a specific connotation through the consumption of me-
dia like film. It seams evident that resorting to some empirically
”well identified sounds” would mean to abandon the richness and
diversity of the sonic world in favor of a statistical ”average”.
Chion also points out the linguistic ambiguity when speaking
about sounds and their cause: A sound of a piano can be the tone
emitted when pressing a key on a piano or it can be the sound
resulting from hitting the piano with a hammer. Or the sound
of a piano can actually result from a synthesizer. And although
we might identify a certain sound as coming from a wooden box
we can not say that one particular sound of a wooden box exists.
Depending on where and how we exert physical force on the box
it will sound differently. In summary: it is usually not possible
to claim that a sound is the sound of something specific, or, vice
versa, that every thing has its one sound [12].
Steven Connor describes the dialectic relationship between
sounds and objects as being an ”immaterial corporeality”: ”One
apparent paradox of hearing is that it strikes us as at once intensely
corporeal - sound literally moves, shakes, and touches us - and
mysteriously immaterial. (...) Perhaps the tactility of sound de-
pends in part on this immaterial corporeality, because of the fact
that all sound is disembodied, a residue of production rather than
a property of objects.” ([13] p. 157)
Last but not least, the relationship between objects and sounds
can be even viewed from a totally different angle. Chion states,
that we can find certain irregularities, frictions, traces of impacts
58
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities
in abstract sounds that can give it a material, bodily quality. He
calls these notions of materiality ”indices sonores matérialisants”
([12], p. 102).
3.3 The Sonic Metaphysics of Objects
In the sound related discourse the notion exists that sound can
be the ”voice” of objects in an actual, immediate manner and not
merely metaphorically. This notion again is dialectic. Connor
states: ”When we hear something we do not have the (...) sen-
sation of hearing the thing itself. This is because objects do not
have a single, invariant sound, or voice.” ([13], p. 157) At the
same time however, sound rarely comes completely apart from its
source. Connor continues that ”to think of a sound as the ’voice’
of what sounds (...) is also to think of the sound as owned by and
emanating essentially from its source, rather than being an acci-
dental discharge from it.” ([13], p. 157) The following dictum is
ascribed to Oskar Fischinger: ”Sound is the soul of an inanimate
object.” (cited in [15], p. 330, author’s translation) John Cage
is reported to have hit the objects wherever he went in order to
investigate their inner nature.
An old example of experiencing sound as voice of things can
be found in Homer’s Odyssey. Homer describes Odysseus, hav-
ing returned home and competing in an archery contest: ”Then
his right hand took the string and made it vibrate, the cord sung
beautifully and clearly, like the call of a swallow.” (cited in [12],
p. 102, author’s translation) This is more than just a metaphor:
To some extend it is the string which is acting, singing, even if
excited by the plucking.
”Through noise nature vibrates of sense”: With this poetic for-
mulation Barthes describes the ability of sound to give things
a voice. According to him, listening is always connected to a
hermeneutics that aims at understanding the dark, the blurred or
mute and to make the sense ”behind” appear. Listening to these
sounds is an essentially religious experience, connecting the lis-
tening subject with the hidden world of the gods [5]. This at-
tribution of sound to an expressive quality of objects and to the
soul and voice of things is an obvious connection to the topics of
antropomorphization and animism described above.
3.4 The Cultural Signification of the Sounds of Ob-
jects
Barry Truax states that sound mediates the relationship between
listener and environment [34]. This includes also the sounds pro-
duced by the cultural-technological artifacts we create. For in-
stance, Mark M. Smith describes how in travel accounts of an-
tebellum America sounds became the signs of positively con-
notated, pre-industrial work, linking sound to an increase of
wealth and population. He also reveals significant differences be-
tween the industrialized North and the slavery dominated South.
Both areas and cultures had a distinct soundscape with different
keynote sounds and soundmarks6 [32]. The sounds of certain ar-
tifacts and machines thus contribute to the identification with a
class, political orientation, etc.
Through a comparative study of noise abatement campaigns
from the early 20th century, Karin Bijsterveld points to class being
an important element involved in definitions of noise and noise
pollution. The cultural struggle about sounds is also a struggle be-
tween intellectual and working classes. The philosopher Theodor
6 The
terms ”keynote sound” (sounds that are heard by a particular so-
ciety frequently enough to form a characteristic sonic background) and
”soundmark” (a sound with a special meaning for a community) have been
coined by R. Murray Schafer [31].
-4-
Lessing is reported being one of the first to organize an anti noise
campaign. Bijsterveld points out that ”the sound of technology
is a key aspect of technological culture, because sound has been
highly controversial and deeply invested with symbolic signifi-
cance.” ([8], p. 165)
New technologies like the automobile or industrial machines
bring with them new sounds that become symbols of progress for
some and a primitive nuisance for others. A group that embraced
noise and loud powerful sounds of technology with almost reli-
gious devotion were the Italian Futurists. In his manifesto ”The
Art of Noises” Luigi Russolo states: ”We find far more enjoyment
in the combination of the noises of trams, backfiring motors, car-
riages and bawling crowds than in rehearsing, for example, the
’Eroica’ or the ’Pastoral’.” (cited in [30], author’s translation)
These examples show how the discourse about sound strongly
reflected the societal structures and significant changes brought
about by industrialization and technological changes in general.
Many comparable accounts could be found today, referring to
ghetto blasters and mobile phones. Since the industrial age and
the introduction of electroacoustic technology sounds have be-
come more pervasive than ever. And this trend will increase due
to the technique of electroacoustic enhancement of commodities
described above. The study of the sociocultural history of sound
reveals that the importance of sound goes far beyond purely func-
tionalist purposes e.g. of providing feedback in an interface.
4 Implications, Directions and Strategies for
Sound Design
We have described how sounds relate to artifacts, how they be-
come meaningful and even can give inanimate things expressive
qualities. And we also have described how both artifacts and their
sounds play an important sociocultural role. Thus, to say that
sound has great potential for conveying information about the na-
ture of artifacts and their hidden processes or properties is correct
but falls short of grasping the full complexity of the role of sound
in artifacts. Sound does not simply convey an information but
it offers the listener resources, affordances, clues for an interpre-
tative act. The dialectic relation of sound with physical objects
which is combined with the dual nature of artifacts - being at the
same time abstract signs and concrete, physical realities - pro-
vides endless possibilities for complex combinations. The result
is a complex interleaving of levels of interpretative clues through
sound. In the following we will elaborate on how sounds can be
designed with these aspects in mind, focusing on the relationship
and possible reconfigurations of objects and their sounds.
4.1 A Narrative Approach to Sound Design
In the introduction we have described the transformation of inan-
imate objects to procedural, interactive objects and the resulting
narrative potential. A technologically complex object with the
ability to sense, process, store and communicate can be seen as
an actor in a narrative of interaction. However, there is hardly any
theoretically grounded know-how and only a very small number
of practical examples which employ a narrative notion expressed
through sound in the interaction with computerized objects. In
order to establish criteria and a methodological framework to in-
vestigate the possibilities of this new direction of sound design for
interactive commodities, we propose to start with film and game
sound design because they provide a rich source of material de-
scribing how the narration of object interaction can be designed
sonically. Maribeth Back suggests that it could be worthwhile to
investigate design practices of sound designers of narrative me-
59
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities

dia. She describes how sound can help to create micro-narratives, etc.) in order to fit them into the ”carrier” sound8 . Particularly
using both cultural experience (codes, sound as sign) as well as interesting to note here is that such sounds, detached from their
physical experience [2]. original source through electroacoustic recording, can become fa-
In the following we will outline a few considerations and possi- miliar and strange at the same time. As mentioned above, some-
ble directions for design drawn from film and game sound design. what contrary to many accounts of everyday listening, the identi-
fication of a sound source can be subjected to quite a big deal of
4.2 Strategies for Object-Sound Reconfigurations interpretation and uncertainty, especially when the sound is rep-
resented in a recording9 . However, this separation of sound and
We have mentioned the possibility of referring sonically to an ob- referent is a powerful device for design because it allows to cre-
ject or a specific action with an object. The sound produced when ate products which have a distinguished and novel sonic identity
an object is used or operated (or just touched in some way) can without being entirely alien.
become not only the signifier of that object but also of its function,
use context and sociocultural significance and turn into a signifier 4.3 The Use of Sound to Animate and Characterize
of metaphorical or associated qualities of the object. Let us con- Objects
sider the example of the sound of a hammer hitting a nail: It is
Animation, Fantasy and Science-Fiction film is certainly also a
an index of a hammer as well as hammering and can symbolize
rich source for the study of objects as expressive ”characters”. In
strength, aggression, work, DIY-culture, communism and so forth
these genres we can find countless examples of objects that are
or it can even be an enunciation of headache.
provided a personality and emotional expressivity through sound.
These levels of meaning creation can become literally mixed
Beauchamp states that ”virtually any character or object can be
in sound through the strategic layering and intertwining of various
personified by adding speech, movement, and the expression of
sounds into a new sound gestalt. This is a common practice in film
emotions.” ([7], p.21) Even in ”realistic” movies the sound de-
sound: Through layering of sounds, combining concrete, identi-
sign often goes beyond just naturalizing or physicalizing an ob-
fiable sounds with each other or even with more abstract sounds,
ject, adding a narrative component. According to Flückiger ob-
meaning potentials7 are transferred between them. Film sound
jects in movies often are not only made credible but also animated
designers for example use this method to create richer meaning
through sound. The purpose of beeping computer displays, the
potentials into seemingly simple sounds. For example, the water-
extensive sonic endowment of supernatural occurrences or alien
drops falling onto an ant colony in ”A Bug’s Life” (John Lasseter,
creatures is to distract from their mocked nature, to suggest an
1998) are sounds of splashing water, combined with sounds of
actual existence, life and function [15].
rockets and explosions [7].
An example for sonically antropomorphized and animated ob-
Another example of the ability of sounds to redefine the nature
jects is the sound design of Darth Vader’s Tie Fighter in ”Star
and meaning of objects displayed on screen are also the typical
Wars”: The distorted scream of a human being was mixed with
cartoon sound effects that have been created for producers like
various jet engines, producing a nail-biting sound while contain-
William Hanna and Joseph Barbera or Tex Avery. Here signifier
ing an eerie humaneness: ”His ship and its sounds are extensions
and signified are subjected to extreme reconfigurations: Instead
of this merger (between flesh and machine, note from the author),
of sonifying literal physical processes, metaphorical or symbolic
and thus they scream in pain.” ([39], p. 109) But also fantas-
sounds are used. Emotional expression and interpretative clues
tically animated objects become more credible when having the
are achieved by the associative nature of the sound and the spe-
appropriate sounds. Think of the living dishes in ”Beauty and the
cific cartoon aesthetic based on analogy, contrast and exaggera-
Beast” (Gary Trousdale, Kirk Wise, 1991). And sound designer
tion. An example is the sound of an anvil being hit with a heavy
Ben Burtt used morphological analogies to human language sen-
hammer used in a scene where a character gets hit by something
tences and baby talk to shape the beeps of R2D2 from ”Star Wars”
like a baseball bat [7]. Many new sound icons, which stand for a
(George Lucas, 1977) [33].
very specific meaning which is entirely artificial and established
Sound plays also a role in marking ordinary objects as extraor-
only through repeated use, have emerged from animated film.
dinary or magic. This can be the case with simple, inanimate ob-
Examples are many film sound clichés: falling objects whistle,
jects that not only become ”alive” through use, but also reveal ex-
strong fist blows in fights have a metallic impression, quick run-
traordinary qualities through sound: An example of a rather sim-
ning produces a sound of a ricochet (which, again, is often the
ple object being enriched and personified in this way is the jade
same, stereotyped sound, and not an arbitrary, realistic recording
sword in ”Crouching Tiger, Hidden Dragon” (Ang Lee, 2000).
of a ricochet), and so forth.
The sword emits glass-like singing sounds, conveying a fine, pre-
These sounds are not questioned by the viewer-listener. Instead
cious identity, fragile and sharp at the same time. When used
the naturalizing power of sound suppresses the notion that the au-
inappropriately it starts to oscillate which is accompanied by a
diovisual event could be impossible or unrealistic, at least during
wobbling sound, as if it was responding in annoyance.
the actual experience of the film. This ability of sound to create
Sonnenschein suggest the use of ”archetypal templates” to cre-
credibility is often used for fantastic or incomprehensible things
ate nonhuman sounds that we can relate to. We might ”find
and processes. Ben Burtt, the sound designer of Star Wars, first
squeals, squeaks or hisses from compressed air hoses; groans
established credibility for fantastic sound effects by finding ”an-
from old wooden doors; and a laughter-like craziness from bend-
chors” in familiar sounds like animals (in which case there is also
ing a saw.” ([33], p. 61) Often sounds of animals are not only
the aspect of animism to be considered, see below) or familiar ma-
used to animate objects but also serve as sign carriers and conno-
chinery. Using these sounds he could inject the fantastic objects
tational devices. Lion growls, yapping chimps or cat purrs can be
and processes depicted on screen with the necessary amount of
familiarity and credibility [39]. These sounds then often are ma- 8 The carrier-modulator principle known from FM synthesis serves

nipulated (pitch shift, time stretch, filtering, amplitude envelopes well here as a metaphor for how layered sounds often work.
9 Quite often students of our classes in sound design are not able to
7 Van Leeuwen proposes this term instead of the static code to express identify the sound sources of recordings their fellows made, unless the
the contextual dependency of meaning making. [36] sounds are very typical or strongly contextualized.

-5-
60
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities
layered over or mixed into ”carrier sounds”, suggesting strength,
wackiness or seductiveness [33].
4.4 The Limitation of the Semiotic Paradigm
Several of the examples given are described through semiotic
analysis, referring to concepts of sign, index, icon, symbol and
so forth. In product sound quality research and design the semi-
otic viewpoint is becoming increasingly established. It suggests
that sounds of objects convey meaning by a significative rela-
tionship to experiences or schemas [21]. But similar to objects,
sounds can never be pure signs. Even in its most abstracted, elec-
troacoustic form sound always carries an immediate, naturalizing
component in it. This effect is at work in film, where sounds
are taken for granted and are accepted as the ”natural” accompa-
niment of events on screen, although the audio-visual product is
entirely constructed. But this effect is also potentially at play -
and very important to be aware of - in sounding physical artifacts.
The semiotic analysis actually works well in audiovisual me-
dia like film because movies are spatially and temporally finite
artifacts, consumed in a strictly standardized way. Linguistic con-
cepts (and semiotics are deeply permeated by a linguistic notion
of meaning making) can be applied, the codes are known and new
codes can be easily developed and decoded, based on previous
knowledge. And, most importantly, there is always an obvious
presence of an authorship, of an actual communication through
the medium of film.
In the interaction with actual, physical, computerized com-
modities, however, the experience changes fundamentally. Al-
though there is a designer involved she is not designing the entire
experience of using the interactive artifact. The designer can not
predict and control the interpretative and appropriative strategies
people will chose as they go about their situated actions in ev-
eryday life. Also the sound designer disappears as the initiator
of a communicative act. In such a situation sounds will be per-
ceived as ”naturally happening”, even if they are not, and the in-
terpretative act will be influenced by this. Already with a rather
simple mechanical process such as a squeaking door there is an
intuitive judgement of an inherent quality of the hinge, which
will be interpreted by the listener on a immediate level, before
any higher level sign systems take effect. The communication
is ad-hoc and interpretative, the sign is constructed through per-
sonal experience rather than cultural conventions10 . According to
Krippendorff semiotics has fundamental limits when dealing with
such dynamic design issues: It is rooted in a two-world ontology,
dividing between a world of signs or signifiers and a world of ref-
erents or objects and it excludes human agency in the sense that
meaning always evolves in an iterative and interactive process of
ongoing interpretation [25]. This suggest that the analysis and de-
sign of sounds for interactive commodities should take a critical
distance from semiotics (without abandoning its valuable contri-
butions) and put an emphasis on hermeneutics, which acknowl-
edges that interpretation and understanding, not only of texts, but
of the world, is essentially a dynamic circular movement between
a single element and the whole, and can not be crystallized to a
specific structure.
10 This may change of course if we are confronted with mass-produced
artifacts, where a certain class of similar devices create structural stan-
dards comparable to film or game genres. But this will only shift the
emphasis towards structurally coded meaning, the ad-hoc interpretation
of the sounds and their naturalizing, immediate expressiveness is still
present.
-6-
5 Conclusion
Designing interactive commodities poses new challenges on sev-
eral levels of design. Such objects often are integrated in everyday
life practices which can not be framed by traditional task and goal
oriented methods. Despite certain routines there are always new
configurations of contexts and events as we go about our daily
lives. Instead of simply decoding pre-structured signs in artifacts
which requires a relatively high level of standardization as it can
be found e.g. in mainstream film, we often interpret the world
in an iterative, ad-hoc manner. This is especially true if the arti-
facts we encounter are complex, networked and to some extend
autonomous and intelligent.
Sound design in this context can be seen as designing an ex-
pressive channel for such animated artifacts and creating micro-
narratives of interactions. This means that we see artifacts as char-
acters, as actors in a micro-dramaturgy and sound both tells us
something about the temporal and dynamic development of the
process as well as about the inherent semantic complexity in the
nature of the artifact which can range from hidden or metaphoric
material qualities to emotional expressions of a kind of ”genie in
a bottle”.
5.1 The Aesthetic Dialectics of Sonically Augmented
Objects
In the design of the sounds for such artifacts the sociocultural
significance of artifacts in everyday life plays an important role.
This happens on two levels:
Firstly, ”objective” qualities can be rendered into artifacts, for
example with indexical sounds pointing to certain artifacts and
thus to a range of potential meanings arising from them. Phe-
nomenologically, sound as a resource for meaning creation in the
context of physical artifacts has the characteristics that it is al-
ways rooted in some sort of physical motion in time. The pos-
sibility of electroacoustic processing and reproduction of sound
is not eliminating this archetypal connection, rather it extends
and modulates it, significantly altering the perception of an ob-
jects properties and its meaning potential. For instance we have
described the practice of mixing concrete and abstract sounds in
film sound. Thus sound design for interactive commodities builds
upon the fundamental perceptive link between material objects
and processes and sound. At the same time it extends this fun-
damental level to socio-semiotical and psychological aspects of
the design domain. By enhancing artifacts with electroacoustic
sounds that are derived from interactions with other artifacts an
effect can be achieved that is related to the dialectical relationship
between objects and their sounds described in this paper.
Second, these new artifacts become sociocultural devices
themselves, subjected to individual and collective interpretative
practices. This requires a sensitivity from the designer towards
techno-cultural history and how artifacts grow into the fabric of
society, shape discourse and reflect, as well as support, cultiva-
tion.
5.2 Goodbye Authenticity - Welcome Schizophonia
The relationship between sounds and objects has always been di-
alectic. Often sounds reveal the true ”authentic” quality of an
object but through engineering of sounds for industrial products
this quality of sound is questioned. Does the convincing sound of
a car door really display its high quality or is it just engineered by
changing the resonance body or maybe even by using materials of
lower quality but with better acoustic properties? In any case the
sounds of objects are perceived as ”natural” and authentic to some
61
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities
extend, which implies a certain ethical dimension in the discourse
about sound design for artifacts. The possibility to integrate a
even wider range of sounds by means of miniaturized electroa-
coustic devices and to control them through computer technology
makes the relationship between physical object and its sound ar-
bitrary11 . Never before there was so much control over the sonic
appearance of an artifact.
R. Murray Schafer was deeply concerned about the increasing
schizophonia resulting from the separation of sounds from their
”natural” sources. But schizophonia is already the normal cultural
condition we live in and have learned to deal with (most of the
time at least). While we might still have some resistance towards
the schizophonic reality, the generation of teenagers, walking on
the streets listening to distorted and filtered pop songs from their
mobile phones, obviously enjoying it, will most likely have no
problems at all with this notion.
As designers we have to embrace this new aesthetics of the
sonically extended artifact as a new field where practices and vo-
cabularies are still to be developed. To just pick one example con-
sequence indicated above: It might be a good idea not to stick too
tightly to sounds that are ”well identified” in laboratory settings,
thinking of them as the only way of conveying a clear meaning.
For designers it will be necessary to deal with the fact that most
of decontextualized (recorded!) sounds are ”badly identified” or
at least reinterpreted. They are the sound designer’s raw material
and the question is: What creative and interpretative potential lies
in them?
6 Future Work
In terms of design many methodological questions remain to be
answered. One central issue is the conceptual and practical inte-
gration of the macro and micro levels of the design. The macro
level deals with the overall experience of the interaction with the
object and how it is embedded into sociocultural context. This
requires a suitable method of design and evaluation that takes into
account cultural factors and interpretative processes.
On the micro level the challenge for the sound designer is the
integration of sounds into each other, into the artifacts and into
the interaction dynamics. For example, an open question is how
”hybrid sound gestalts” can be designed, that merge with a device.
How far can we stretch an objects (sonic) identity? What sounds
are acceptable on a metal, plastic, wooden object? The meth-
ods and tools for the fine-tuning of the fit of the electroacoustic
enhancements and the objects own sounds to processes and dra-
maturgies exist and are widely used in film and game sound de-
sign: Manipulation of pitch, amplitude, spectrum over time with
envelopes, adjusting the mix, crossfading, masking, layering or
using semantically complex effects like reverb, delay, filtering,
and so on. And designing sounds for narrative purposes means
that we have to design transition points, find linking strategies be-
tween sounds to create continuity, and model the dynamic time-
space relationship of interactive applications. This requirement
has already lead to advanced game sound middleware such as
FMOD12 or Wwise13 . Finally, there are several techniques for
”material transfer”: Convolution, manual filtering and manipu-
lation of resonance ”by ear”, or physical modeling. Technically
very simple and often very successful is the subtle layering of fil-
tered recordings of physical processes onto other sounds to sug-
11 Every recording of a sound in fact already is a distortion of the au-
thentic link between a physical process and the resulting sound.
12 http://www.fmod.org
13 http://www.audiokinetic.com
-7-
gest a certain material quality.
But the relevant questions are: What happens if such ”transnat-
ural” materials are present in an actual, physical object, which
has already a material identity? How can design methods, which
mostly are used in immersive media, be adapted to the context of
actual physical objects with their non-immersive qualities? And
of course this sound design is not ”free”, in many ways it is much
more restricted by external factors than the sound design for tra-
ditional audiovisual media. These factors are technical limita-
tions as to what sounds can be chosen (e.g. resonance body of
the device, loudspeaker specifications), but also how the object is
handled or used (hitting, shaking, heating, etc.). In short: Fur-
ther investigation is also needed into synchresis and diegetics of
sound for interactive commodities. We propose to draw on mate-
rial from game sound research as a good starting point as it is able
to link the narrative audio-visual world of film with interactive
media and physical interfaces.
Finally, we will have to investigate strategies for defining and
modulating the ”threshold of attention” by shaping how sounds
integrate into a soundscape and how they may emerge from it
in a meaningful way. Traditionally, much research has been con-
ducted in order to evaluate the perceived urgency of alarm sounds.
In reverse we suggest that it should also be possible to shape
sounds in such a way that they easily blend into the background
of our attention, becoming a keynote sound and being available
to listening-in-search or listening-in-readiness [31]. In addition,
designing meaningful, rich, subtle sounds with a certain degree
of individuality, freshness and unpredictability in terms of what
they are supposed to mean will be a good foundation to actually
prevent them from being annoying, helps to create ”hi-fi” sound-
scapes (in the Schaferian sense), and supports the human ability to
interpret and appropriate whatever we encounter and make sense
out of the alien.
To make progress in these directions a collaboration between
the still quite separate communities of sound and music comput-
ing, auditory display, and traditional sound design is required.
The design of rich sounds for interactive commodities will have
to integrate the two design cultures and their strengths: Dynamic
and interactive control on the one hand and rich, detailed sonic
semantics on the other.
References
[1] Federico Avanzini and Paolo Crosato. Haptic-Auditory
Rendering and Perception of Contact Stiffness, volume
4129/2006 of Lecture Notes in Computer Science. Springer,
2006.
[2] Maribeth Back. Micro-Narratives in Sound Design: Con-
text, Character, and Caricature in Waveform Manipulation.
In Proceedings of the 3rd International Conference on Au-
ditory Display, Palo Alto, California, 1996.
[3] P. Barreiro, C. Ortiz, M. Ruiz-Altisent, V. De Smedt,
S. Schotte, Z. Andani, L. Wakeling, and P.K. Beyts. Com-
parison Between Sensory and Instrumental Measurements
for Mealiness Assessment in Apples. A Collaborative Test.
Journal of Texture Studies, 29:509–525, 1998.
[4] Roland Barthes. Semantik des Objektes. In Das semiologis-
che Abenteuer. Suhrkamp, 1988.
[5] Roland Barthes. Der entgegenkommende und der stumpfe
Sinn. Suhrkamp, 1990.
[6] Jean Baudrillard. The System of Objects. Verso, 1996.
62
Genie in a Bottle: Object-Sound Reconfigurations for Interactive Commodities

[7] Robin Beauchamp. Designing Sound for Animation. Else-
vier, Burlington, MA, 2005.
[8] Karin Bijsterveld. The diabolical symphony of the mechani-
cal age. In Michael Bull and Les Back, editors, The Auditory
Culture Reader, pages 165–189. Berg, 2003.
[9] Gernot Böhme. Der Glanz des Materials - Zur Kritik der
ästhetischen Ökonomie. In Atmosphäre. Suhrkamp, 1995.
[28] K. Obermayer K. Franinovic Y. Visell et. al. P. Susini,
D. Rocchesso. Closing the Loop of Sound Evaluation and
Design. In ISCA Workshop on Perceptual Quality of Sys-
tems, 2006.
[29] Stephen Harold Riggins, editor. The Socialness of Things
- Essays on the Socio-Semiotics of Objects. Mouton de
Gruyter, 1994.

[10] Stephen Brewster. Overcoming the lack of screen space on [30] Luigi Russolo. L’art des Bruits, Luigi Russolo, textes établis
mobile computers. Personal Ubiquitous Comput., 6(3):188– par Giovanni Lista. L’Age d’homme, 2001.
205, 2002. [31] R. Murray Schafer. The Soundscape: Our Sonic Environ-
[11] Daniel Chandler. Technological or media deter- ment and the Tuning of the World. Destiny Books, New
minism, 1995. Available from World Wide Web: York, 2nd edition 1994 edition, 1977.
http://www.aber.ac.uk/media/Documents/tecdet/tecdet.html [32] Mark M. Smith. Listening to the Heard Worlds of Antebel-
[cited 24.1.2008]. lum America. In Michael Bull and Les Back, editors, The
Auditory Culture Reader, pages 137–163. Berg, 2003.
[12] Michel Chion. Le Son. Editions Nathan, Paris, 1998.
[33] David Sonnenschein. Sound Design - The Expressive Power
[13] Steven Connor. Edison’s teeth: Touching hearing. In Veit
of Music, Voice, and Sound Effects in Cinema. Michael
Erlmann, editor, Hearing Cultures - Essays on Sound, Lis-
Wiese Productions, 2001.
tening and Mordernity. Berg, 2004.
[34] Barry Truax. Acoustic Communication. Ablex, 2nd edition,
[14] Mihaly Csikszentmihalyi and Eugene Rochberg-Halton.
2000.
The meaning of things - Domestic symbols and the self.
Cambridge University Press, Cambridge, 1981. [35] Sherry Turkle. The Second Self: Computers and the Human
Spirit. MIT Press, 20th anniversary edition, 2004.
[15] Barbara Flückiger. Sounddesign: Die virtuelle Klangwelt
des Films. Schüren Verlag, Marburg, 2001. [36] Theo van Leeuwen. Speech, Music, Sound. Palgrave
Macmillan, 1999.
[16] Karmen Franinovic, Daniel Hug, and Yon Visell. Sound
embodied: Explorations of sonic interaction design for ev- [37] Joan M. Vastokas. Are artifacts texts? Lithuanian woven
eryday objects in a workshop setting. In Proceedings of the sashes as social and cosmic transactions. In Stephen Harold
13th international conference on Auditory Display, 2007. Riggins, editor, The Socialness of Things - Essays on the
Socio-Semiotics of Objects, pages 337–362. Mouton de
[17] W. W. Gaver. What in the world do we hear? An ecological
Gruyter, 1994.
approach to auditory event perception. Ecological Psychol-
ogy, (5):1–29, 1993. [38] G. B. Vicario. Prolegomena to the perceptual study of
sounds. In D. Rocchesso and F. Fontana, editors, The Sound-
[18] B. L. Giordano. Everyday listening, an annotated bibliogra-
ing Object, pages 17–31. Edizioni di Mondo Estremo, 2003.
phy. In D. Rocchesso and F. Fontana, editors, The Sounding
Object, pages 1–16. Edizioni di Mondo Estremo, 2003. [39] William Whittington. Sound Design & Science Fiction. Uni-
versity of Texas Press, Austin, 2007.
[19] Adam Greenfield. Everyware: the dawning age of ubiqui-
tous computing. New Riders, 2006.
[20] Daniel Hug. Towards a hermeneutics and typology of sound
for interactive commodities. In Proc. of the CHI 2008 Work-
shop on Sonic Interaction Design, Firenze, 2008.
[21] Ute Jekosch. Assigning Meaning to Sounds - Semiotics in
the Context of Product-Sound Design. In Jens Blauert, edi-
tor, Communication Acoustics. Springer, 2005.
[22] Victor Kaptelinin and Bonnie A. Nardi. Acting with Tech-
nology. MIT Press, Cambridge, Massachusetts, 2006.
[23] C. Kayser, C. I. Petkov, M. Augath, and N. K. Logothetis.
Integration of touch and sound in auditory cortex. Neuron,
48:373–384, 2005.
[24] Gregory Kramer, Bruce Walker, Terri Bonebright, Perry
Cook, John Flowers, Nadine Miner, and John Neuhoff.
Sonification report: Status of the field and research agenda.
1999.
[25] Klaus Krippendorff. The semantic turn - A new foundation
for design. Taylor and Francis, 2006.
[26] Bruno Latour. Reassembling the Social - An Introduction to
Action-Network- Theory. Oxford University Press, 2005.
[27] Winfried Nöth. Handbuch der Semiotik. J. B. Metzler, 2.,
vollständig neu bearb. und erw. aufl. edition, 2000.

-8-
63
 Saturday Night or Fever?
Context Aware Music Playlists

Stuart Cunningham, Stephen Caulder & Vic Grout

Centre for Applied Internet Research (CAIR), Glyndŵr University,
Plas Coch Campus, Mold Road, Wrexham, LL11 2AW, North Wales, UK
{s.cunningham | s.caulder | v.grout}@glyndwr.ac.uk

Abstract. Context awareness provides opportunities for enhanced user experience, interaction and customisation of electronic
devices, particularly those which hold large data sets of information which may often only be relevant to a user in certain scenarios.
In this work, we examine how context awareness can be applied to the automatic generation of music playlists on mobile music
devices, such as MP3 players and mobile phones. We hypothesise that the type of music which a person might wish to listen to will
often be influenced by external factors such as the time of day, the ambient temperature, amount of ambient or background noise,
their current amount of physical activity, and their emotive state, to name a few.

We detail the results and data sets of preliminary investigation into several human movement scenarios, emotional status and external
factors. These results are obtained by employing the cost-effective Wiimote controller to record acceleration profiles. The Wiimote is
assessed against a professional level, high-cost, motion capture device to identify if such portable devices are useful in everyday
scenarios. Base values for subject locomotion were investigated for the Wiimote device and verified and analysed using the Qualisys
3D-motion capture system. This was done for setting a base line for the subject forward velocity, but also to allow the research into
further complex locomotion studies for this project. A model of the playlist generation system is provided, which can be used to
simulate responses to various type of context-informing input.

It is noted that the system has been implemented using a fuzzy rule base system (FRBS). This will allow the initial construction
based on a knowledge base relating a suggested emotional state (E-state) based on various inputs. The longer term concept is to
investigate the adaptable nature of the initial knowledgebase and allow it to adapt to an individuals actual emotional state
preferences. It is suggested that further research into the implementation of a Self-Learning Fuzzy Rule Based System (SL-FRBS).

1. Introduction choices. A diagrammatic overview of the complete system we

propose is given in Figure 1. Not only this, but in order to
The generation of an automatic playlist is a useful resource for provide such a customised system, we propose that techniques
portable music players, especially given the storage capacities of of fuzzy logic and self-learning systems are suitable to
currently available which are continually increasing. This means efficiently carry out this task.
that the average listener with an MP3 or other digital music
player will often have a database which has a membership of
tens of thousands of songs, possibly even more! The factors
which influence the choice of music a listener wants to listen to
will be influences by many factors, but especially the user’s
current mood or emotional state, their current activity (if any),
and the range of other external and environmental factors around
the listener, such as the temperature, amount of background
noise, time of day, etc. Rather than have a user cycle through
such large databases to find music which they want to play,
automatic playlist generation, in an ideal scenario, attempts to
select and play music which it has determined the user would
currently like to listen to.

Existing recommendation and playlist generation systems rely

upon the ability to make correlations and measurements between
songs in a music database in order to automatically generate or
order the music in such a manner that the listener is
automatically provided with music which (it is hoped) that they
will like. To date, this has mainly been achieved by analysing
the user’s listening trends and habits as well as analysing the
songs in the database to extract information about the musical Figure 1: Context-Aware Playlist Generation
content itself.
2. Related Work
In this work we propose, and provide initial results of, the
development of an automatic playlist generation system which Implementing automatic playlist generation is not a new field
not only implements these previous approaches identifying user and has a long history relating to the organisation and
trends and content, but also consider the current context of the recommendation of music tracks present on a music player. A
listener and how this might influence their desired musical detailed overview of alternative, historical approaches for

64
automatic playlist generation is beyond the scope of this work,
however, we briefly present an overview of the field and refer
the reader to the references made in this section should he or she
wish to gain a deeper knowledge of playlist generation
techniques [1, 2, 3, 4, 5, 6, 7].
Initially, recommendation and playlist generation systems relied
on abstract or meta-data level information and user preference in
order to order the music tracks. These systems are not hugely
different from Automated Collaborative Filters (ACFs) [8] in
that they track and correlate user preference and build up table
of similarity based on musical information such as artist and
genre [1, 2, 3]. Although the can take simplistic forms by
counting number of plays, favourite artists, etc. the processes of
learning user preferences purely based on these factors can also
take more complex forms [3].
More recent work has been focussed on extracting and analysing
content present within the user’s music collection and making
decisions based upon similarity metrics or correlations, available
as a result of content analysis. A more notable example of this
type of analysis is in the field of audio thumbnails [4, 5, 6]. Such
content information can then be coupled with the meta-data and
user preferences mentioned previously in order to provide, what
is generally agreed to be, a more suitable and effective system
for playlist generation and music recommendation [4, 5].
In similar work to this paper, Reynolds et al. propose systems
more advanced than the more conventional approaches to
playlist generation mentioned earlier. Their work supports the
theories that contextual information is also highly valuable and
appropriate when suggesting or ordering musical tracks for the
listener. In fact, they mull over many of the factors which we
propose to be crucial in our own work, and explain in more
detail later in this paper. Reynolds et al. consider variables such
as temperature, activity and location to be incorporated as meta-
data, and also indicate that the mood of the listener is another
key variable which must be considered in automatic playlist
generation. Their work also presents an excellent overview of
the history of automatic playlist generation and the links
between music and mood or emotion [7]. A detailed exploration
of emotional states, measurement and music goes beyond the
intended scope and context of this paper, however the reader is
further referred to the work of Meyers who an in-depth
exploration of the links between emotion and music [9].
initially of interest comes from two main sources: the user or
listener and the environment in which the listener exists. This is
further ratified by Reynolds et al. who also consider contextual
input parameters from these two domains [7].
3.1. The Listener
Information which can be extracted from the user is arguably the
most useful data which can be acquired if one wishes to
determine contextual information regarding the listener’s current
emotional state and level of activity. This is illustrated in more
detail in Figure 2 which is presented as a subset of the previous
diagram in Figure 1.
The emotional state of the user is highly likely to influence the
type of music which they wish to listen to. Listeners who are
happy or contented are likely to desire their favourite music
tracks and music which is from genres which is known to have
positive effects on happiness and reflect and stimulate their
current emotional state. Equally, a listener who is unhappy
might wish to listen to slower, calmer music that fits with their
current mood. However, this is not to negate the fact that a sad
or unhappy listener might listen to upbeat, happy music in order
to change their mood. Therefore, the listening requirements
cannot be based purely upon determination of emotion, or at
least, multiple inputs are needed to determine if a sad listener
wishes to remain sad or wants to be cheered up. We propose that
mechanisms such as skin conductivity and heart rate might be
acquired directly from the user, before being sent to a decision
making tool which also took into account other parameters and
semantic knowledge of the music database.

We approach playlist generation from a similar set if initial

hypotheses and notions of employing contextual knowledge. We
see our work as a natural of extension from Reynolds et al. A
limitation of their work was that there was only minimal
identification of the practicalities and implementation present. In
our work we begin to examine how to measure and put into
practice context-aware playlist generation based on a number of Figure 2: Sample Human Input Parameters
bio-physical measurements.
The other major factor interest at this stage is the amount of
3. Context movement or physical activity the listener is engaged in. If the
user is moving a lot then it is reasonable to suppose that they
might be exercising or engaging in an focussed physical
Mobile devices have become increasing computationally
exertion, in which case they would be likely to desire music
powerful and as well as functioning as digital music players,
which reflects this physical motion and might feature strong,
contain more and more peripheral devices such as cameras,
driving beats and tempos which are relatively high, greater than
touch screens and accelerometers. The Apple iPhone and iPod,
120 beats-per-minute (BPM), for instance. An easily available,
in particular, is a high profile example of such a device,
low cost device which is already on the market and can be used
although there are other competing products available and in
to detect three-dimensional (3D) motion is the Wiimote,
development. The power, connectivity options and range of data
illustrated in Figure 3, the remote control device designed for
sources becoming available in mobile devices means that a
use with the Nintendo Wii games console. The Wiimote can be
range of mechanisms can be devised to allow the extraction of
used independent of the Wii console to communicate with
contextual information [10]. Principle contextual information

65
Bluetooth-enable devices, such as computers and provides
valuable motion information via its accelerometers [11].
Figure 3: Nintendo Wiimote Controller
The ability of mobile devices to provide motion data is initially
a very interesting concept when it comes to considering
applications for these features, beyond those originally
conceived for the device. This combined with the ability to
attain other real-world and user information from sensors and
direct input means that information regarding the user’s current
context can be described and learnt from observations of
subjects using the devices.
3.2. The Environment
Factors in the environment around the user too, are likely to
have an effect on many factors which influence the listener’s
habits and preferences when playing music. A number of
parameters related to the physical environment as well as
environmental metrics can provide suitable input to a decision-
making system. Figure 4 provides illustration of how
environmental inputs fit into the playlist generation system.
Figure 4: Sample Environmental Inputs
Consider firstly, physical phenomena around the listener, such
as the ambient temperature, humidity, acoustic noise, and levels
of light, for example. When high temperature and high humidity
are present, we suggest that two scenarios are possible. The first
may be that the user will be tired and at rest (determined via
motion detection) in which case music of lower tempo and
which provides a more relaxing experience may be required.
However, an alternative may be that the user wants music to
stimulate them perhaps because they are exercising or deriving a
positive, happy feeling from the strong temperatures (such as
when on the beach during a holiday). This can be further refined
by measuring the amount of light. If light levels are low then it
is most likely night time and, again, in combination with
movement and ambient temperature the user might wish to
either dance or relax and chill-out. The amount of ambient noise
is useful firstly to help ensure that the listener is provided with a
constant, desired volume level proportional to the amount of
noise in the environment (within reason). It might also be used
to determine if the user is inside or outside.
4. Initial Implementation & Results
To begin to assess the ability to gain contextual information
from sensors and the usefulness of this information in deriving
an emotional state (E-state) we implemented a small-scale
system for playlist generation which would analyse a number of
input factors and apply these in ordering a small music database
consisting of eight songs, shown in Table 1.
Table 1: Music Database used in Testing
ID Artist Song
0 Daft Punk One More Time (Radio Edit)
1 Fun Lovin’ Criminals Love Unlimited
2 Hot Chip Over and Over
3 Metallica Harvester of Sorrow
4 Pink Floyd Comfortably Numb
5 Sugababes Push The Button
6 The Prodigy Breathe
7 ZZ Top Gimme All Your Lovin’
4.1. Defining Input Parameters
Through some initial studies of the input parameters we are able
to define and categorise a number of states against which any
incoming data to the recommendation system can be matched in
order to tag the current input state and make decisions based on
this state attainment. This applies across all of the sensor input
available to the recommendation system. It should be noted at
this stage that the states currently defined are not necessarily
absolute at this stage and are often indicative. These are easily
refined and extra levels of granularity or simplicity can be easily
introduced by adjusting the size of the membership sets and/or
adjusting the state value thresholds.
Initially we define four locomotive states, which can be
extracted from the Wii controller or another similar device such
as the motion data from an iPhone/iPod or from a higher level
source such as the Qualisys motion capture system. These four
possible states and their associated values are defined in Table 2.
These values are attained from input directly attained from the
user of the system using such a device.
Table 2: Locomotive States
Standing Walking Jogging Running
0 m/s 1.1 m/s 2,2 m/s 3,5m/s
Similarly, we must define some standard states and ranges for
any other parameters which we intend to implement at this
stage. Due to limitations of equipment availability and time, we
decided to focus on additional inputs from the environment at
66
this stage, rather than other factors directly read from the listener provide hands-free and low cost motion feedback to the FRBS.
or user. Therefore, we define states and parameters for a number The Wii controller provides acceleration feedback for the three
of environmental factors, which are presented in Tables 3 to 5. the x, y and z axis. Initial usage of the device is to obtain an
approximate value for the forward linear velocity V from
the acceleration ∆a across a period of time ∆t as
Table 3: Temperature States
Cold Warm Hot
V = ΔaΔt . (1)
15-18 oC 20-23 oC 27-30 oC We experimented with extracting locomotion data from the
Wiimote controller and compared this to the data which can be
extracted from a full-blown motion capture system; the
Table 4: Lighting States
Qualisys. Figure 6 shows how placement of the Wiimote on
Dark Grey Day Light Light Sunny subjects was achieved and Figure 7 provides an image of a
0-3 2-4 3-5 4-7 6-10 subject being tracked by the Qualisys system.

Table 5: Weather Condition States Subject

Heavy Rain Light Rain Drizzle Dry
0-2 2-5 4-6 5-10 Initial
test
At this stage, the parameters are loosely defined from empirical
and historical knowledge of the individual and are not yet finely
tuned for the playlist generation system. In order to make the
input functions more realistic, usable and suitable to the listener, Wii
they must first be fuzzified. controller

4.2. Fuzzy Logic Model

To make a decision on the emotional state of the user, based
upon the input data sets, we employ a fuzzy logic system,
specifically a Fuzzy Rule Based System (FRBS). Fuzzy set
theory is proposed by Zadeh [12]. The concept of fuzzy logic is
the ability to formalise approximate reasoning. An application
of this theory is FRBS which utilises the concept of fuzzy rules.
Such systems comprise of two main features: An inference
engine and knowledge base. Figure 5 shows an illustration of a Figure 6: Locomotion Extraction with Wiimote
generic FRBS system. Mamdani and Takagi-Sugeno-Kang
(TSK) FRBS were considered in this study, with the later TSK
being implemented as the main FRBS in this paper. Our initial
test FRBS is implemented using the Matlab FIS environment.
The Matlab Fuzzy Logic toolbox provides a useful tool for the
initial realisation of the fuzzy model.

Knowledgebase

Input
Decision
Fuzzification Making Defuzzification
logic

Output
Figure 5: Schematic of FRBS

A fuzzy set L defined on a Universe of discourse U may be

characterised by a membership function μL(x) and is defined
over the interval [0, 1]. For this case the range is based on
measurements taken for a single subject. The following figures
define how the input parameters are configured in the fuzzy
logic model.
The definition of membership of locomotive states is shown in
Figure 8 and was obtained by empirical observations and
experimentation with motion detection using both the Wiimote
and a Qualisys system to capture motion. The Wiimote
controller was selected for this study due to its suitability to
Figure 7: Motion Capture with Qualisys
The Qualisys is a high-end motion capture system which uses an
array of infra-red detection cameras to track reflective markers
that can be placed on the subject of interest. Motion vectors are
plotted for each marker on the subject and these can be recorded
and visualised in real-time. Detailed information of the Qualisys
system can be found at www.qualisys.com.

67
As can be seen, the Qualisys system is more cumbersome than
the Wiimote and actually requires a number of cameras to track
motion accross a very fine range. However, the Wiimote is
much less intrusive and can be attached to a belt or put in a
pocket. Furthermore, provided the Wiimote sunject remains
within the transmission radius of the Bluetooth transceiver
(either 10 meters or 100 meters in laboratory conditions), the
user will have complete freedom of movement. This makes the
Wiimote not only better for recording natural movement, but
much more practical for deployment into real-world scenarios,
such as playlist generation.

Figure 10: Lighting Set

Figure 8: Fuzzy Locomotion Set

Ambient temperature is illustrated in Figure 9 and membership

criteria were gained by measuring a range of location conditions
and consulting average temperature conditions in the UK. It is
expected that for implementation in other countries, this kind of
contextual information is subjective and would be defined at
initialisation.
Figure 11: Weather Conditions Set

These factors are combined as inputs to the FRBS system to

allow an output of emotional state estimation as can be seen in
Figure 12.

Figure 9: Temperature Set

The set of lighting condition classification is shown in Figure 10

and possible weather conditions set is in Figure 11. Again,
membership criteria are based upon empirical observations.

Figure 12: Suggested TSK-type FRBS

68
The current rule base has been derived on the expert or
suggested emotional state of a subject for various states of
inputs. Input ranges for locomotion and temperature have been
determined experimentally, while Lighting and weather are base
on a incremental scheme and a initial triangular or trapezium
membership function was chosen initially and further research
on the rule base and fuzzy set distribution, membership
functions will provide refinement of the system performance.

4.3. Defining Output of Emotional State

Currently a single output state is generated from the fuzzy logic
model which is used to indicate the predicted emotional state of
the user based upon the input parameters. We define a set of 10
emotional states or outcomes which are broadly defined as
having membership of 5 categories. This can be seen in Table 6
and a schematic of the integration of output states is shown in
Figure 13.
Figure 14: Wii Acceleration Curves for Motion
Table 6: Emotional States
Depressed Unhappy Neutral Happy Zoned Comparative data from the Wiimote is presented in Figure 14
0-3 3-4 4-6 5-8 7-9 and also demonstrates locomotive state can be simply
established by correct interpretation of the accelerometer data
over a fixed-rate sampling period of 10Hz. The illustrations
from the Wiimote present the velocity of the user over samples
(time). As can be seen from the illustration, the states of
walking, jogging and running are clearly identifiable.

Figure 13: PlayList Organising Tool (PLOT)

5. Initial Results
5.1. The Wiimote as a Motion Device
The Qualisys motion capture system allowed comparative data
capture of the three Primary directions of subject motion. For
this paper we are primarily interested in forward velocity. It is
noted from the Qualisys results a potential coupling is possible
due to placement of the Qualisys Sensor. The data recordings
from the Wiimote and the Qualisys system were conducted
under the same conditions in order to fully investigate the
comparative effectiveness of each device.

Results measuring motion using the Qualisys are provided in

Figure 15 for walking, jogging and running states. The graphs
from the Qualisys plot forward motion, over a fixed distance,
across time for each of the three tests. The fixed distance is
covered over increasingly short times as the subject increases
speed from walking, jogging and running. Interpretation of time
comes from multiplying the number of samples by the fixed
sampling rate on the Qualisys used for each test.
Figure 15: Motion Data from Qualisys

5.2. Fuzzy Playlist Generation

We configured the system with a number of expert-informed
parameters which related emotional state to a sample collection
of input parameters such as locomotion and environmental

69
factors. We carried these tests out over ten subjects to begin Additionally, we constructed attached an emotional state range
with, who were experienced at using portable digital music to each of the songs in our small database to which the output
players in a variety of scenarios, and established the knowledge emotional state can be correlated. Table 7 shows the E-state
in the fuzzy system using the average of these tests. The range which, through pilot testing, we attached to each song.
scenario configurations used are given below in code along with
a textual description of the scenario and the average set of Table 7: Song Database with E-States
results relating to emotional state is presented in Figure 16. ID Song E-state E-State Median
Although an average response is employed, there were strong 0 One More Time (Radio Edit) 5-8 6.5
correlations between the majority of the subjects and their 1 Love Unlimited 4-6 5
perceived emotional state indicators, which indicate that the data
2 Over and Over 7-9 8
recorded is reliable for those circumstances.
3 Harvester of Sorrow 0-3 2
%1 Walking, temperature is hot, lighting is 4 Comfortably Numb 3-4 3.5
dark/grey and weather is light rain. 5 Push The Button 7-9 8
ip1=[1.5;28;2;2.5] 6 Breathe 0-3 1.5
E(1)=evalfis(ip1, a) 7 Gimme All Your Lovin’ 5-8 6.5

%2 Stationary, temperature is cold, Table 8 shows the results of each of the seven experimental
lighting is dark and weather is raining. scenarios presented along with the resulting playlist to be
ip2=[0.1;14;1;0.5] generated. The grade in the playlist G is determined by taking a
E(2)=evalfis(ip2, a) simple Euclidean distance measurement of the form

( p − q)2
%3 Stationary, temperature is Warmish,
lighting is brightening and weather is dry. G( p, q) = (2)
ip3=[0.1;21;9;9]
E(3)=evalfis(ip3, a) from the song E-state median and the current E-state of the
listener, based on the scenario. The playlist is shown as a ranked
%4 Running, temperature is hot, lighting is set of song ID numbers from the database for each E-state.
Daylight/Getting brighter and dry.
ip4=[3.5;30;5;9.5] Table 8: Playlist Order for Test Scenarios
E(4)=evalfis(ip4, a)
Scenario E-state Playlist order
%5 Walking, temperature is getting hot, 1 4.3 1; 4; 0; 7; 3; 6; 2; 5
lighting is dark and weather is drizzling. 2 0 6; 3; 4; 1; 0; 7; 2; 5
ip5=[1.5;16;1;6] 3 6.8 0; 7; 2; 5; 1; 4; 3; 6
E(5)=evalfis(ip5, a) 4 7.7 2; 5; 0; 7; 1; 4; 3; 6
5 3 4; 3; 6; 1; 0; 7; 2; 5
%6 Stationary, temperature is hot, lighting 6 3.8 4; 1; 3; 6; 0; 7; 2; 5
is grey and weather is dry. 7 6.5 0; 7; 1; 2; 5; 4; 3; 6
ip6=[1.4;32;2.5;9.5]
E(6)=evalfis(ip6, a) 6. Conclusions & Future Work
%7 Walking/Jogging, temperature is mild, We have demonstrated that the Wiimote can function as a highly
Daylight and it’s dry. useful instrument for measuring forms of human motion and in
ip7=[2;17;4.5;9] future plan to assess the functionality of other low cost motion
E(7)=evalfis(ip7, a) devices such as the accelerometers which have been integrated
into mobile music players such as the iPhone and iPod Touch.
Emotions=floor(E*100) These devices, and the Wiimote functionality, can be further
ratified by more comparisons with data extracted using high-end
motion capture hardware, such as the Qualisys, mentioned
earlier.

In terms of being able to attain a more reliable, and possibly

multiple-faceted, emotional state indicator would be to develop
a self-learning algorithm to provide an adaptive context list
generation. This is something which we intend to pursue in the
near future as it will provide a much more customised
interpretation of the user’s emotional states and musical
preferences. Given that emotion is such a personal and almost
unique experience for each individual, this is high in the list of
priorities for future developments. The current system operation
is to obtain an estimate for the emotional state of the subject and
to use this as a factor for the subjects play list organisation. The
list update is based on a predetermined time up-date interval.
The current E-state is used to modify the play list. We have used
a single output value based on a Multiple Input Single Output
Figure 16: Results of Emotion Scenario Indicator Testing
(MISO) model. Further research based on Multiple Input

70
Saturday Night or Fever? : Context Aware Music Playlists
Multiple Output (MIMO) would allow multiple meta-data
selection process based on beat or tempo of the music file, for
example.
The initial FRBS has been implemented and provides a baseline
model for the development of a more defined model. Fuzzy sets
and memberships functions need to be refined to allow further
optimisation of the system, extending the range of inputs to
allow a far more flexible system. Of interest is the modification
of the suggested rule base. An aspect is the ability to modify the
initial rule base with rules more selective of the subjects’
individual emotional states. Approaches being investigated
based on a self-learning strategy:
1. Modifying the fuzzy set definitions. Modification of the
fuzzy sets implies potential issues with the fundamental
nature of the linguistic meaning. Mamdani suggested that a
change in the fuzzy set definitions should be avoided
although potential minor or small modifications may be
possible [13].
2. Modifying the set of rules in the rule base. Given a fuzzy
relation is used to describe each rule in the rule-base of the
FRBS. Then given the general rule expression:
Rn : If m is Mn AND t is Tn AND l is Ln AND w is Wn
THEN u is Un
Where
Rn = the fuzzy relation of rule n.
M, T, L, W, and U are linguistic labels assigned to each
variable of rule n.
The general relation ‘R’ is constructed as the union of individual
relations
n databases, Audio Mostly 2007, Ilmenau, Germany, 27th – 28th
R = UR i . (3)
i =1
So that using Zadeh’s compositional rule of inference Zadeh the
output fuzzy set assuming for this case a Mamdani model then
u o = (m o × t o × l o × wo )oR . (4)
It is therefore feasible to modify the rule-base based on factors
such as the individual’s emotional preference, or additional
inputs such as heart rate. The system would then potentially
provide an emotional state ranking that is more reflective of that
individual.
Another point for development is to attach a number of E-states
to each song in the database, since certain types of music may be
suitable for two or more E-states. Through further investigation,
we propose to develop a primary emotional rating and then a
secondary and possibly third E-state. Furthermore, the E-state
suitable for each song can be investigated until a much deeper,
accurate estimation of the range of suitable E-states can be
determined.
The system presented at this stage is more limited than the
outlined system presented in section 1 of this paper. As
mentioned, this was due to equipment and time constraints,
therefore future work would include employing further sensor
systems as inputs to the playlist generation system particularly
those which are concerned with input from the listener, rather
than the environment. It can be argued that inputs from the
environment are factors which, we predict are likely to influence
the listener’s emotional state. However, to derive measurements
and indicators directly from the listener suggest a much more
robust and reliable evaluation of the emotional state.
Acknowledgments
The authors wish to thank Sue Taylor and Janet Hayes of Sports
and Exercise Sciences at Glyndŵr University.
References
[1] Aucouturier, J-J. & Pachet, F., Scaling up music playlist
generation, Proceedings of the IEEE International Conference
on Multimedia Expo, Lausanne, Switzerland (2002).
[2] French, J.C. & Hauver, D.B., Flycasting: On the Fly
Broadcasting, Joint DELOS-NSF International Workshop on
Personalization and Recommender Systems in Digital Libraries,
Dublin, Ireland, 18th – 20th June (2001).
[3] Platt, J.C., Burges, C.J.C., Swenson, S., Weare, C., & Zheng,
A., Learning a Gaussian process prior for automatically
generating music playlists. Advances in Neural Information
Processing Systems 14, pp. 1425–1432 (2002).
[4] Gasser, M, Pampalk, E., & Tomitsch, M., A Content-Based
User-Feedback Driven Playlist Generator and its Evaluation in
a Real-World Scenario, Audio Mostly 2007, Ilmenau, Germany,
27th – 28th September (2007).
[5] Logan, B., Content-Based Playlist Generation: Exploratory
Experiments, ISMIR2002, 3rd International Conference on
Musical Information (ISMIR), (2002).
[6] Kukharchik, P., Martynov, D., & Kheidorov. I., Indexing
and retrieval scheme for content-based search in audio
September (2007).
[7] Reynolds, G., Barry, D., Burke, T., & Coyle, E., Towards a
Personal Automatic Music Playlist Generation Algorithm: The
Need for Contextual Information, Audio Mostly 2007, Ilmenau,
Germany, 27th – 28th September (2007).
[8] Cunningham, S., Bergen, H., & Grout, V., A Note on
Content-Based Collaborative Filtering of Music, Proceedings of
IADIS - International Conference on WWW/Internet, Murcia,
Spain, 5th-8th October (2006).
[9] Meyers, O.C., A Mood-Based Music Classification and
Exploration System, MS Thesis, Massachusetts Institute of
Technology (MIT), USA (2007).
[10] Tamminen, S., Oulasvirta, A., Toiskallio, K. & Kankainen,
A., Understanding mobile contexts, Personal and Ubiquitous
Computing, 8(2), 135–43 (2004).
[11] Maurizio, V., & Samuele, S., Low-cost accelerometers for
physics experiments, European Journal of Physics, (28) pp. 781–
787, Institute of Physics (2007).
[12] Zadeh, L.A., Fuzzy logic and soft computing: Issues,
contentions and perspectives, Proceedings of the 3rd
International Conference on Fuzzy Logic, Neural Networks and
Soft computing, Iizuka, Japan (1994).
[13] Mamdani, E.H., Application of fuzzy algorithms for simple
dynamic process, Proceedings of the IEE, 121, pp.1585-1588
(1974).
71
 A Musical Instrument based on
3D Data and Volume Sonification Techniques
Lars Stockmann, Axel Berndt, Niklas Röber
Department of Simulation and Graphics
Otto-von-Guericke University of Magdeburg
Lars.Stockmann@Email.de
{aberndt|niklas}@isg.cs.uni-magdeburg.de

Abstract. Musical expressions are often associated with physical gestures and movements, which represents
the traditional approach of playing musical instruments. Varying the strength of a keystroke on the piano results
in a corresponding change in loudness. Computer-based music instruments often miss this important aspect,
which often results in a certain distance between the player, his instrument and the performance.
In our approach for a computer-based musical instrument, we use a system that provides methods for an inter-
active auditory exploration of 3D volumetric data sets, and discuss how such an instrument can take advantage
of this music-based data exploration. This includes the development of two interaction metaphors for musical
events and structures, which allows the mapping of human gestures onto live performances of music.

1 Introduction sic production, although recently a new research area

Over the past years, computers have contributed to
musical performances in several ways. Already in the
late 1960s, computers have been employed to control
analogue instruments. The GROOVE synthesizer de-
veloped by Max Mathews was one of the first com-
puter controlled analogue synthesizers [14]. Since the
introduction of the MIDI standard as a communica-
tion protocol, computers have been used as a means
for conduction and arrangement in many music pro-
ductions, but also as a bridge between input devices
and synthesizers. In this context, computer have also
been used to augment a performance by adding algo-
rithmically generated notes that fit musical structures,
as for example in Music Mouse [19] or MIDI composing
software like Bars & Pipes 1 . Intelligent instruments
like Music Mouse facilitate an easier, more intuitive,
approach to the creation of music for the musically in-
experienced. At the same time they offer new ways of
creating music – even for professional musicians.
In today’s productions, external synthesizers are of-
ten omitted. Their place is taken by virtual instru-
ments, such as Native Instruments’2 simulation of the
B3 organ or the virtual acoustic and electric piano.
Even standard consumer hardware is powerful enough
for their deployment, and they are used to imitate any
kind of instrument in realtime. In contrast to the
achievements in sound synthesis, input devices other
than MIDI-keyboards are still not common in mu-
solely focussing on new musical interaction methods
has been established. One example3 that is planned to
be commercially available in the near future is the re-
acTable system, which is described in [10, 12]. Like
Crevois et al., who developed an instrument called
Sound Rose (see [6]), Jordà et al. use a tangible inter-
face as a new intuitive way for live music performances.
Computer-based instruments are designed in a way
that a musical controller generates data that is passed
to a computer and therein mapped to a single acoustic
stimuli of a certain pitch and volume, or to parameters
that somehow control an algorithmic composing. The
advantage of this approach is that virtually any type
of data can be used as input for these instruments.
The mapping of arbitrary data to sound (including mu-
sic) is part of another very important area of research,
specifically sonification. It is often used in the devel-
opment of Auditory Display systems, and employed to
acoustically convey scientific data. While for a long
time, sonification has merely been a part of visualiza-
tion research, the techniques which were outlined by
Gregory Kramer (see [13]) have been developed and
successively improved to provide an enhancement, and
at places even a superior alternative, to visual represen-
tations in science (e.g. [7]). Especially when it comes
to the visualization of the inner and outer structures
of 3D volumetric data sets. The auditory channel can
be used to reduce the load of information that other-

1 Bars & Pipes www.alfred-j-faust.de/bp/MAIN.html 3 An overview of some musical controllers can be found
2 Native Instruments www.nativeinstruments.de at www-ccrma.stanford.edu/~serafin/NBF/Newport.htm

72
A Musical Instrument based on 3D Data and Volume Sonification Techniques
wise has to be absorbed by the visual channel alone.
The main challenge for sonification research is to find
an expressive, intuitive, and comprehensible mapping
from the data domain towards sound.
In our sonification system, we employ spatial inter-
actions to facilitate an intuitive method for an audi-
tory exploration of 3D volumetric data sets. It uses a
strictly functional mapping of data to complex sounds,
based on differences in pitch and volume. This sys-
tem is the basis for a novel computer-based instrument
that can be used without musical experiences. The in-
strument is designed out of two metaphors: The Tone
Wall metaphor allows a performer to directly gener-
ate a melody, while the Harmonic Field is used for an
computer-aided accompaniment. Both techniques can
be used at the same time. It produces diverse sounds,
and allows for a highly interactive performance. It can
be shown that spatial interactions inherent a great po-
tential for the use in computer-based instruments.
The paper is organized as follows: After an intro-
duction to the sonification of volumetric data sets in
Section 2, we advance by presenting our sonification
system in Section 2.1. This includes some technical de-
tails regarding our implementation. We then elaborate
in Section 2.2 how sonification and computer-based in-
struments connect, and how live music performances
can benefit from an instrument that uses our sonifi-
cation system. In Section 3 we describe musical data
can be derived from spatial gestures in volumetric data
sets. The Tone Wall metaphor (Section 3.1) specifies
the pitch, loudness, and timbre space for melodic pur-
poses. The Harmonic Field (Section 3.2) describes how
volume data can be used to represent harmonies, bro-
ken chord play, and musical textures. Section 3.3 is
concerned with a combination of both concepts for the
presentation of a one man polyphonic performance. Fi-
nally the results are discussed in section 3.4, which also
includes possible improvements for further research.
2 Volume Data Sonification
Data sonification is an underdeveloped, but growing
field of research. In this section we describe how soni-
fication can be applied to acoustically describe 3D vol-
ume data sets. Before we describe our method, we
discuss several advantages that make sonification tech-
niques at times superior to a more classic visual exam-
ination and presentation of scientific data sets. Exam-
ples are monitoring applications, or any type of unfo-
cused operations and processes. The generated acous-
tic stimuli can be heard without paying direct atten-
tion. This yields an improved mobility. Furthermore,
Kristine Jørgensen states that the presence of sound
increases attention, and eases the perception by inten-
tionally utilizing channel redundancy [11, 8]. A simple
example is a flash light that is augmented with a sound
while flashing. The proper use of acoustic stimuli in
combination with the visual representation also gener-
ates a deeper sense of immersion, especially in inter-
active 3D environments [17]. Gregory Kramer stated
that ’spatialized sound can, with limitations, be used to
[...] represent three-dimensional volumetric data’ [13].
One reason is that spatialized sound provides a direct
mapping to the physical 3D space.
3D volume data occurs in countless fields of research
and is used to represent the inner and outer structure of
objects or materials in a voxel representation. To find
an expressive mapping of the these voxels to sound is
one of the main challenges when designing a sonifica-
tion system.
Since the development of powerful graphics acceler-
ators, there has been much research on finding a good
mapping in the visualization domain, but only a few
attempts exist to exploit the possibilities of sonifica-
tion to convey 3D volume data. Minghim and Forrest
have suggested methods like the “Volume Scan Pro-
cess”, in which the density inside a volume probe is
mapped to the pitch of a generated tone [16]. David
Rossiter and Wai-Yin Ng traverse the voxels of a 3D
volume and map their values to different instrument
timbres, amplitudes and pitches [18]. Both systems
are controlled through a quite simple mouse/keyboard
interface. However, for the sonification of 3D volume
data, interaction must not be seen as requirement, but
as key aspect. In fact, it is the second most impor-
tant aspect after the mapping. A direct exploration of
the data by, e.g., moving the hand through an interac-
tive 3D environment can provide the user with a better
understanding of extent or local anomalies. Both ex-
amples of related work lack this ability of a responsive
user interface for 3D input like a realtime tracking sys-
tem, or need to compile the audio data before one can
listen to it. The next passage outlines our sonification
system, which focuses on direct interactions and an ex-
pressive mapping of the inner structure of 3D volume
data.
2.1 Spatial Exploration of Volume Data
As mentioned before, a sonification system can greatly
benefit from tracking devices that allow a direct ex-
ploration of the volume data. In the visualization do-
main, this is generally done using a certain Viewpoint-
Metaphor, such as the ones presented by Colin Ware
and Steven Osborne [23]. With respect to data sonifi-
cation, the eye in hand metaphor can be easily trans-
formed into the above described volume probe. Instead
of a spherical or cubical shape, our approach uses the
metaphor of a chime rod, as illustrated in Figure 1.
-2-
73
A Musical Instrument based on 3D Data and Volume Sonification Techniques
Figure 1: 3D Volume scan through chime rod
The rod can be moved freely through the 3D volume,
and is controlled by an interactor that is connected to
a 3D tracking device. The advantage of using a rod in-
stead of a spherical or cubical shape is, that the pitch
of a tone can be directly associated with a position
along the rod. Together with an amplitude-modeling
depending on the density value at a certain position, a
complex tone is generated. This allows for an intuitive
exploration of the inner structures of the volume data.
Unfortunately, the system could not be implemented
using a MIDI-controlled synthesizer. Instead, we de-
vised our own sound synthesis. A sound is rendered
depending on the density distribution of the volume
that is in close vicinity of the chime rod. The listeners
head is orientation-tracked, and the generated sound
is spatialized to provide an additional localization cue
for a more immersive experience.
The realtime tracking is achieved using a Polhemus
FASTRAK that allows four sensors to be connected.
The input data is processed in the client PC that, be-
sides the sonification and sound rendering also per-
forms the visualization. Figure 2 shows the setup of
our sonification system.
Figure 2: Setup of the sonification system
Using sonification and visualization at the same time
does not only induce the afore mentioned redundancy
that eases perception of the data by dispensing the in-
formation on two channels, but also allows for multi
variate data to be presented directly without the need
of switching between different representations. How-
ever, it is a crucial aspect of the system that the visu-
alization, which requires a powerful hardware, does not
interfere with the audio streaming, even if the system
is not equipped with the latest graphics accelerator.
Thus, we make great use of multi-threading running
the visualization on a low priority to ensure that the
audio stream is never interrupted. A scheme of the
whole sonification system is illustrated in Figure 3
Hardware Software
Input device
Input data Main loop
Position, an- Sonification
Tracking gle, button
device events Thread manage-
ment
3D-volume
data
Sound synthesis
Spectral shaping
Output device Audio data
Soundcard Audio Stream Spatialization
Figure 3: Schematics of the sonification system
For the sound processing and output in a multi-
threading environment we use an audio API that is spe-
cially designed for realtime audio applications [20] and
revised it for our purposes. The results were promising
and beared the idea to introduce music elements into
the system. In the next section we elaborate on how
sonification methods and computer-based instruments
are connected and show how our system can contribute
to the research field of the latter.
2.2 Volume Sonification in a Musical Environment
Hunt and Hermann who advance the research of the
model based sonification impose interaction with the
physical world to be a natural cause for acoustic feed-
back [9]. This feedback is used to gather information
about an object. E.g., a bottle of water that is shaken
reveals information about its contents. This cause-
and-effect chain can not only be used to convey ab-
stract information like the number of messages in an
e-mail inbox of a mobile phone [24] but is also a power-
ful paradigm for computer-based instruments. In the
broadest sense, one could consider these instruments as
merely a special case of sonification: The sonification
of interaction itself. In a musical improvisation interac-
tion can be seen as an expression of emotion and mood.
-3-
74
A Musical Instrument based on 3D Data and Volume Sonification Techniques
A computer that is asked to improvise could, of course,
not use mood or emotion as basis for its performance,
but arbitrary, or specially arranged data. Using mu-
sic to convey data can have some advantages. Often
sonification suffers from annoyance. Paul Vickers and
Bennett Hogg state that ‘Sonification designers con-
centrated more on building systems and less on those
systems’ æsthetic qualities’ [22]. Accoustic stimuli that
abide by the rules of music are generally more appeal-
ing for the listener than sounds that use arbitrary pitch
and timbre. It may even stimulate the interactive ex-
ploration of data, as the listener self-evidently becomes
a music performer by interacting with the dataset. She
or he will try to achieve the most pleasant musical re-
sult. A distinct variation in the data means a distinct
variation in music. Its location can be memorized more
easily when the performer ‘explores it intentionally’ be-
cause she or he feels that this particular variation fits
best in the current music progression.
However, finding a meaningful mapping of arbitrary
multi-dimensional data to music must be considered
highly challenging. Some approaches can be found in
projects like the Cluster Data Sonification or the Solar
Songs by Marty Quinn. In his Image Music4 sonifi-
cation, the user can interactively explore a 2D image
through music. However, nothing has been done yet
in the domain of 3D volume data. Furthermore, the
said examples are not intended for live music perfor-
mances. The interaction is limited to mouse input that
does not meet the high responsiveness demanded by a
music performer.
Besides the mapping, the method for interacting
with the system is crucial for its efficiency. Like the
afore mentioned sonification system computer-based
instruments mostly use either mouse/keyboard inter-
action, or are designed to be played with MIDI-
keyboards. These demand a certain skill in order to
be adequately handled. Systems using the elements
of direct interaction as a means for acoustic excita-
tion are scarce. Instruments like the Fractal Composer
introduced by Chapel, for example, provide a mouse
driven graphical user interface [5]. The system com-
poses music using the MIDI protocol in realtime that
depends on parameters which are set by the user. She
or he has no direct control over the melody or har-
mony that is generated. This induces a big distance
between the performer and the instrument. She or
he can only influence the composition on a fairly high
level. These systems are referred to as interactive in-
struments [4] or active instruments[5]. In contrast, the
reacTable and the Sound Rose mentioned earlier are
collaborative instruments that use direct interaction.
4 Design Rhythmics Sonification Research Lab www.
drsrl.com/
Indeed the tangible interface is very intuitive though
these attempts are momentarily limited to two dimen-
sional space. Besides the afore mentioned reacTable
and Sound Rose The “Morph Table” system that uses
morphing techniques presented in [25] is a good exam-
ple how this interface can be used for music generation
[2]. However, the music is also controlled on a rather
high level. The system generates transitions between
a source- and a target pattern which is applied on pre-
composed melodies and rhythms. It is not possible to
create a melody directly. Furthermore, it is limited to
two dimensions.
Chadabe describes a system called Solo that uses mod-
ified theremin’s (see [21]) as 3D input devices to guide
the system [3]. Again, the melody is generated al-
gorithmically. The performer controls variables like
tempo and timbre. The computer is used for sound
synthesis. Thus, this approach is similar to that de-
scribed in [5] and [2] as the performer has only a global
influence on the generated music. However, we think
that 3D input devices can be used to intuitively control
both, melody and accompaniment. Where the former
is generated through a direct mapping of the position
to pitch while the latter could benefit from semi auto-
matic composition or precomposed elements. This not
only opens the path for diverse improvisations but also
can be considered more immersive than just influencing
certain aspects of music that is otherwise algorhythmi-
caly generated.
Our system for interactive exploration of 3D volume
data is applicable in that it provides the necessary de-
grees of freedom to have both aspects in one instrument
as well as the responsiveness demanded for a live per-
formance. This makes it possible to develop metaphors
for music and sound generation. Two are described in
the next section.
3 Volumetric Music
Along the lines of traditional musical instruments,
computer-based musical instruments have to find in-
tuitive performative metaphors for musical events. A
typical example: To strike one key on the piano means
playing its corresponding pitch. The keystroke veloc-
ity regulates its loudness. The following sections will
describe and discuss this mapping of spatial gestures
to musical events and structures, in analogy to the pre-
viously discussed image and volume data sonification
techniques. The volumetric data represents thereby
the medium of interaction and defines the basis for a
music processing.
-4-
75
A Musical Instrument based on 3D Data and Volume Sonification Techniques

3.1 Tone Wall

A question that arises is: How can different tones be
represented in the 3D space? A very intuitive way is a
mapping along the vertical axis: low pitches go down,
high pitches go up.
But an expressive performance necessitates more
than the on/off switching of simple tones. It must
be possible to form them. One of the most impor-
tant means therefore is dynamics (i.e., loudness). In
correspondence to the keystroke velocity on the pi-
ano, we consider the tone space as a wall. The deeper
the performer/interactor punches through that virtual
wall (in z-direction) the louder the tone will be played.
Short punches produce staccato notes, whereas to hold
a tone, the interactor remains in the wall for as long as vel bre
desired.
oci
ty tim
An additional parameter is the punch velocity that dyn
am

affects the attack and onset behavior of the tone. A fast
punch causes a short attack (a very direct beginning of
the tone), and a more percussive onset performed in a
slow velocity results in a softer tone at the beginning
independent of its dynamic level.
Thus, the y- and z-axis open up the complete band-
width of expressive tone forming known from keyboard
instruments, like the piano, and the punch velocity is
a new means to specify details of the tone beginning.
However, it would be unwise to not additionally exploit
the potentials lying in the x-axis. Many instruments
allow the player to vary its timbre to a certain extent,
for which the x-axis is predestined. Different timbres
can be blended from left to right, e.g. from a very dark
sinusoidal waveform over relaxed, clear sound charac-
teristics up to brilliant and very shrill sounds. There
are no limitations in sound design in comparison to
traditional musical instruments. The complete Tone
Wall concept is illustrated in figure 4.
For more timbral variances and freedom, it is possi-
ble to fill the Tone Wall with volumetric data of vary-
ing density. It can be employed as static behavior or re-
act on interactions, e.g. like particles that are charged
with kinetic energy when they are hit by the interactor
device. Due to the freedom to apply any sound synthe-
sis methods, the Tone Wall interface is not restricted
to pitch based melodic structures, but also for more
complex sound structures and noises for contemporary
music styles.
3.2 Harmonic Field
In contrast to the Tone Wall concept, which speci-
fies an interface to create basic musical events, the
Harmonic Field is already a pre-composed musical en-
vironment, which can be freely explored by the per-
former.
pitch
ics
Figure 4: Tone Wall
It defines a number of regions (as illustrated in fig-
ure 5) with their own harmonic content, e.g. a C ma-
jor harmony in the grey area (harmony 1), a minor in
the yellow (harmony 2), a cluster chord in the area of
harmony 5, and so on. The performer can move his
focus via a head-tracking interaction over the regions
to change the harmony that is currently played; he lit-
erally looks to the harmonies to play them.
Each harmonic area defines a density gain towards
the peak in its center. The density allocation can, of
course, also feature more complex shapes, define mul-
tiple peaks, holes and hard surfaces. The values can
be used for fading techniques, such as those described
in [1]; high density can be implemented with a louder
volume than low density. But the harmonic field is not
restricted to static tones only. Chords can be orna-
mented by arpeggiated figures and compositional tex-
tures can be defined. Instead of using a simple in/out
fading, the texture density can be adapted: very sim-
ple, transparent textures at lower density areas and
rich in detail figures at higher densities.
Since harmonic areas can overlap, we applied a num-
ber of transition techniques—other than fading that
does not satisfy in any situation. Held chords are tran-
sitioned part by part. Each part is moving stepwise
towards its targeted pitch, where the steps are cho-
sen according to the underlying scale of the harmony
(e.g., major, minor, or chromatic scale). Instead of a
stepwise movement, the transition can also be done by
linear glissando. The transitional pitch is an interpo-

-5-

76
A Musical Instrument based on 3D Data and Volume Sonification Techniques
harmony 3
harmony 2
harmony 1
line
of si
ght
harmony 5
harmony 4
Figure 5: Harmonic Field
lation of the pitches of each harmonic area according
to their density weightings. The goal pitch is reached
when the old harmonic area is left, or a hole within
with zero density is found. With complex cluster-like
harmonies, the resulting metrumless clouds do wake
associations with György Ligeti’s Clock and Clouds for
women’s choir and orchestra.
Compositional textures, in any respect, are not
metrumless. They are well-defined sequences of pit-
ches/events in a certain tempo and rhythm. In the
case of different tempi, the transitional tempo is an in-
terpolation depending on the density weighting. Since
the textures are repetitive, the morphing techniques of
Wooller and Brown [25], and the interpolation tech-
nique of Mathews and Rosler [15] can be applied to
combine the figural material.
However, generative textures were not included at
the current state. Therefor, transition techniques for
generative algorithms have to be developed and are
classified as future work.
3.3 Poly Field
When performing music, it is always desirable of be-
ing able to handle both, melodic and harmonic data,
simultaneously. Thus, both interfaces, the Tone Wall
and the Harmonic Field, have to be accessible and con-
trollable by one person at the same time.
This is achieved by employing two input devices
which can be controlled independently. The user plays
melodic gestures on the Tone Wall using hand and
arm gestures and thereby controls the harmonic pro-
gression on the Harmonic Field through head gestures
and a simple look. Furthermore, tilting the head can
be used to steer timbral aspects of the Harmonic Field
play.
Since it turned out to be of some difference to play
melodic figures that harmonize with the Harmonic
Field play, a further quantization is implemented to
the Tone Wall. The scale that is playable on the Tone
Wall is matched to the current harmonic base and the
punch height is quantized to this scale.
3.4 Discussion
As with all musical instruments, it is necessary to in-
vest a certain amount of practice to learn the intu-
ition and motoric sensitiveness for a confident expres-
sive play. The intuitive correspondence between ges-
tural and musical events, especially in the case of the
Tone Wall interface, turned out to be very supportive
for a steep training curve. Nonetheless, a few practical
issues have to be discussed.
The interaction with the Tone Wall is subject to a
motoric limitation; it is quite exhausting to create fast
pace melodies with a proper play over a long period
of time. Tracking latencies (ranging between 8–10 ms)
and sampling artifacts (interaction sample rate is 60
Hz with two interactors) also slightly interfere with the
play and the possible speed of interaction.
Because of the absence of any visual reference points,
it is at times difficult to meet the intended pitches.
A calibration, according to the size of the performer,
can lower this problem; his body can provide several
reference points.
For playing melodic intervals, the interactor has to
leave the wall, jump over the unwanted pitches, and
punch back into it. Moving the interactor within the
wall would trigger pitches in-between. Thus, melodic
intervals are always adherent with short pauses. A
legato articulation is not possible within this approach.
Therefore, an interactor speed dependency has to be
incorporated: a pitch is only played if the interac-
tor’s velocity is below a certain threshold. Pitches
can be skipped by faster movements even within the
wall. Since this raises the problem of creating fast
pace melodies, this mode has to be detachable, e.g.
by a button on the hand interactor.
The same approach could be useful to reduce the
wah-effect when playing a pitch. The punch always hits
the low dynamics area at the wall surface first, and the
loud dynamics afterward. Hence, each tone fades in,
even with fast punches that do only effect a more direct
tone attack. Although the interaction sampling rates
-6-
77
A Musical Instrument based on 3D Data and Volume Sonification Techniques
used lower this effect, a velocity dependent sampling
of the interactor would make the dynamic level more
accessible.
However, all performative means of expression are
available and easy to perform—dynamics and empha-
sis, articulation, (de-)tuning, timbral and articula-
tional (glissando, triller etc.) effects.
For the Harmonic Field the composer is free to de-
fine any chords, assign them to any timbral instru-
mentation and figurative ornamentation, and combine
them by overlapping. He can actually define any com-
positional and timbral texture and it can be explored
freely by the player. The player, however, is fixed to
this predefined set, unable to create new chords and
textures interactively during the performance. Fur-
thermore, the three-dimensional space cannot be ex-
plored adequately using head-orientation alone, i.e.
looking at a harmonic area from a relatively fixed po-
sition, which allows only an exploration in 2D. The
player should be able to move freely in 3D space. This
raises conflicts with the Tone Wall metaphor. A pos-
sible solution is to position the Tone Wall always in
front of the player and reposition it when the player
moves through the Harmonic Field.
However, the combination of the Harmonic Field
with the Tone Wall interface open up a very large
musical bandwidth with more timbral freedom than
any traditional musical instrument can offer. The
three-dimensional setup of harmonic structures and
their density-dependent ornamentation textures are
also unique and provides an inspiring platform espe-
cially for performing contemporary music.
4 Conclusion, Future Work
In this paper we presented a gesture based approach to-
wards virtual musical instruments. We introduced the
conceptual basis, which is a novel interaction mecha-
nism developed for the interactive auditory exploration
of volumetric data sets. For their sonification we de-
vised the musical metaphors of the Tone Wall and the
Harmonic Field, and conceived their sonic behavior in
a way that the interaction with them produces musical
events and aesthetic structures, like tones, melodies,
timbre effects, chords, and textures. We discussed as-
sets and drawbacks of these metaphors and outlined
advancements.
3D interaction devices open up a multitude of new
possibilities for the design of computer-based instru-
ments. Their big potential lies in their intuitive as-
sociation with physical human gestures and musical
events, for which the interaction with virtual volume
data turned out to be the medium of choice. Future
work includes the development of further metaphors
and the integration of serial and generative concepts.
The volumetric interaction interface also opens up a
promising possibility for the conduction of music.
The musical volume representation concept is also
a novel view on musical structure and elements, en-
abling new compositional forms and means of expres-
sion. Here lies the biggest potential of new computer-
based instruments. It is unnecessary to imitate tradi-
tional instruments to create music that is performed
better with the real ones. If one wants to play a pi-
ano, violin, trombone etc. the real ones perform always
better. New instruments should not imitate them, but
stand for a confident self-reliance to open up new possi-
bilities for new music to constitute their right to exist.
References
[1] A. Berndt, K. Hartmann, N. Röber, and M. Ma-
such. Composition and Arrangement Techniques
for Music in Interactive Immersive Environments.
In Audio Mostly 2006: A Conf. on Sound in
Games, pages 53–59, Piteå, Sweden, oct. 2006.
Interactive Institute, Sonic Studio Piteå.
[2] A. R. Brown, R. W. Wooller, and T. Kate. The
Morphing Table: A collaborative interface for mu-
sical interaction. In A. Riddel and A. Thorogood,
editors, Proceedings of the Australasian Computer
Music Conference, pages 34–39, Canberra, Aus-
tralia, july 2007. Australian National University
Canberra.
[3] J. Chadabe. Interactive Music Composition and
Performance System. United States Patent Nr.
4,526,078, july 1985. filed sep. 1982.
[4] J. Chadabe. The Limitations of Mapping as a
Structural Descriptive in Electronic Instruments.
In Proceedings of the Conference on New Instru-
ments for Musical Expression (NIME-02), Dublin,
Ireland, may 2002.
[5] R. H. Chapel. Realtime Algorithmic Music Sys-
tems From Fractals and Chaotic Functions: To-
wards an Active Musical Instrument. PhD thesis,
University Pompeu Fabra, Department of Tech-
nology, Barcelona, Spain, sept. 2003.
[6] A. Crevoisier, C. Bornand, A. Guichard, S. Mat-
sumura, and C. Arakawa. Sound Rose: Creat-
ing Music and Images with a Touch Table. In
NIME ’06: Sixth meeting of the International
Conference on New Interfaces for Musical Ex-
pression, pages 212–215, Paris, France, 2006.
IRCAM—Centre Pompidou.
[7] W. T. Fitch and G. Kramer. Sonifying the body
electric: Superiority of an auditory over a vi-
sual display in a complex multivariate system.
-7-
78
A Musical Instrument based on 3D Data and Volume Sonification Techniques

In G. Kramer, editor, Auditory Display: Soni- [19] Laurie Spiegel. Music Mouse. http://retiary.
fication, Audification, and Auditory Interfaces, org/ls/programs.html, 2004.
Boston, MA, USA, 1994. Addison-Wesley. [20] Lars Stockmann. Designing an Audio API for Mo-
[8] C. Heeter and P. Gomes. It’s Time for Hyper- bile Platforms. Internship report, 2007.
media to Move to Talking Pictures. Journal of
[21] L. S. Theremin. Method of and Apparatus for the
Educational Multimedia and Hypermedia, winter
Generation of Sounds. United States Patent Nr.
1992.
73,529, dec. 1924.
[9] A. Hunt and T. Hermann. The Importance of
Interaction in Sonification. In ICAD 04—Tenth [22] Paul Vickers and Bennett Hogg. Sonification ab-
Meeting of the International Conference on Audi- straite/sonification concrète: An ‘Æsthetic per-
tory Display, Sydney, Australia, july 2004. spective space’ for classifying auditory displays in
the ars musica domain. ICAD 06 – 12th Interna-
[10] S. Jordà, M. Kaltenbrunner, G. Geiger, and
tional Conference on Auditory Display, Juni 2006.
R. Bencina. The reacTable. In Proceedings of
the International Computer Music Conference, [23] Colin Ware and Steven Osborne. Exploration
Barcelona, Spain, 2005. International Computer and virtual camera control in virtual three di-
Music Association. mensional environments. SIGGRAPH Comput.
Graph., 24(2):175–183, 1990.
[11] K. Jørgensen. On the Functional Aspects of Com-
puter Game Audio. In Audio Mostly 2006: A [24] J. Williamson, R. Murray-Smith, and S. Hughes.
Conf. on Sound in Games, pages 48–52, Piteå, Shoogle: Excitatory Multimodal Interaction on
Sweden, oct. 2006. Interactive Institute, Sonic Mobile Devices. In Proceedings of the SIGCHI
Studio Piteå. conference on Human factors in computing sys-
[12] M. Kaltenbrunner, S. Jordà, G. Geiger, and tems, pages 121–124, New York, USA, 2007.
M. Alonso. The reacTable: A Collaborative Musi- ACM.
cal Instrument. In Proceedings of the Workshop on [25] R. W. Wooller and A. R. Brown. Investigat-
”Tangible Interaction in Collaborative Environ- ing morphing algorithms for generative music. In
ments” (TICE), at the 15th International IEEE Third Iteration: Third International Conference
Workshops on Enabling Technologies, Manch- on Generative Systems in the Electronic Arts,
ester, U.K., 2006. Melbourne, Australia, dec. 2005.
[13] G. Kramer, editor. Auditory Display: Soni-
fication, Audification, and Auditory Interfaces.
Addison-Wesley, Boston, MA, USA, 1994.
[14] M. V. Mathews. The Digital Computer as a Musi-
cal Instrument. Science, 142:553–557, nov. 1963.
[15] M. V. Mathews and L. Rosler. Graphical Lan-
guage for the Scores of Computer-Generated
Sounds. Perspectives of New Music, 6(2):92–118,
Spring–Summer 1968.
[16] R. Minghim and A. R. Forrest. An Illustrated
Analysis of Sonification for Scientific Visualisa-
tion. In IEEE Conference on Visualization, At-
lanta, USA, oct. 1995.
[17] Niklas Röber and Maic Masuch. Playing Audio-
only Games: A compendium of interacting with
virtual, auditory Worlds. In Proceedings of 2nd
DIGRA Gamesconference, Vancouver, Canada,
2005.
[18] David Rossiter and Wai-Yin Ng. A system for the
complementary visualization of 3D volume images
using 2D and 3D binaurally processed sonification
representations. In Proceedings of the 7th confer-
ence on Visualization, pages 351–354, San Fran-
cisco, USA, 1996. IEEE Computer Society Press.

-8-

79
 Same but Different – Composing for Interactivity

Anders-Petter Andersson, Interactive Sound Design, Kristianstad University, Anders-Petter.Andersson@hkr.se

Birgitta Cappelen, AHO- The Oslo School of Architecture and Design, Birgitta.Cappelen@aho.no

Abstract. Based on experiences from practical design work, we try to show, what we believe, are the similarities and differences,
between composing music for interactive media compared to linear music. In our view, much is the same, built on traditions that
have been around for centuries within music and composition. The fact that the composer writes programming code is an essential
difference. Instead of writing one linear work, he creates infinite numbers of potential musics that reveal themselves as answers to
user interactions in many situations. Therefore, we have to broaden our perspectives. We have to put forward factors that earlier was
implicit in the musical and music making situations, no matter if it was the concert hall, the church, or the club. When composing
interactive music we have to consider the genre, the potential roles the listener might take, and the user experience in different
situations.

music for the installation. This, in order to be able to discuss in

What and Why what manner we mean the compositional work is similar and
Interactive media are increasingly becoming a significant part of different when comparing interactive music to linear music.
our daily lives, and the extreme ongoing developments in ORFI is a new audio tactile interactive installation (see Figure
mobile communication services are making auditive interactive 1). It consists of around 20 tetrahedron shaped soft modules, as
media in particular important. This is a big challenge for special shaped cushions. The modules are made in black textile
everyone who wants to take part in the creation and and come in three different sizes from 30 to 90 centimetres.
understanding of the new auditive interactive media. But to what Most of the tetrahedron has orange origami shaped “wings”
degree is it new? And to what degree does the composition of mounted with an orange transparent light stick along one side
interactive music build on traditional music composition? (see Figure 2).
In this paper we want to show how the composition of music
for interactive media is similar to, but also different from, linear
music composition. We would like to show to what degree, we
can reason and follow the same line of thoughts, when
composing interactive music as in linear music. We want to
show what perspectives, factors and conditions one has to
acknowledge when composing music for interactive media.

The ORFI Example

Figure 2: A user playing and interacting with ORFI’s wings.

The “wings” contain bendable sensors. By interacting with the

Figure 1: The ORFI landscape, the modules and the dynamic
video projection.
We would like to use the interactive installation ORFI 1 [1] as an
example, showing how we were thinking when creating the
1 ORFI is an interactive audio tactile experience environment created by
the group MusicalFieldsForever (Anders-Petter Andersson (concept,
music, composition rules, sound design), Birgitta Cappelen (field theory,
wings the user creates changes in light, video and music. ORFI
is shaped as a hybrid, a hybrid between furniture, an instrument
and a toy, in order to motivate different forms of interactions.
One can sit down in it as in a chair, play on it as on an
instrument, or play with it as with a friend. The largest modules,
in suitable size to sit on, have no wings with sensors, but
speakers instead. Every module contains a microcomputer and a
radio transmitter and receiver, so they can communicate wireless
with each other. The modules can be connected together in a
Lego-like manner into large interactive landscapes. Or, the
modules can be spread out in a radius of 100 meters. So one can
interact with each other sitting close, or far away from each
other. There is no central point in the installation, the field [2].
The users can look at each other or at the dynamic video, like a
concept, design, interaction design) and Fredrik Olofsson (concept,
music, software and hardware)) www.musicalfieldsforever.com

80
living tapestry, which they create together. Or they can just chill
out and feel the vibrations from the music sitting in the largest
modules.
The installation has a 4-channel sound system that makes
listening a distributed experience. ORFI consist for the time
being of 8 genres, or collections of rules, which the user can
change between. Our use of the term “genre” has references to
popular culture, such as music, and everyday activities made
when consuming the music, such as dancing [3, 4]. In ORFI we
explore 8 different musical genres:
ƒ JAZZ (bebop jazz band, dancing, ambient),
ƒ FUNK (groove, dancing)
ƒ RYTM (techno, club)
ƒ TATI (speech, onomatopoeic, movie)
ƒ GLCH (noise, club)
ƒ ARVO (ambient, relaxation)
ƒ MINI (minimalist instruments, playing with toys)
ƒ VOXX (voice recordings generated dynamically by
the user).
In this paper we have chosen to describe the compositions in the
JAZZ and MINI genres, because they represent oppositions
regarding genres and therefore serve as explanatory examples.
The many possibilities, such as many distributed wireless
modules, and many genres to choose between, reflect our goal to
facilitate collaboration and communication on equal terms,
between different users in different use situations
New Situations and Roles
One of the aspects we have put a lot of effort into when creating
ORFI is related to the use or consumption situation. We don’t
know and cannot control in what situation and for how long
ORFI will be played on and listened to. This differs in a
fundamental way from composing music for a stage
performance. In a stage performance one knows implicit that the
audience will sit in the dark with the face towards the stage
quietly listening for one or two hours. Radio listening is more
like our situation, but here one usually knows by knowing the
time, what everyday ritual the radio program is part of [5].
Music as an ambient sound tapestry in a home is more like our
situation, but here the user is limited to turning the music on or
off, or change to next tune. These actions represent a break in
continuity of the listening experience.
In ORFI the interaction shall be a seamless [6] part of the music
experience. The music therefore must invite and motivate
dynamically to interaction. To the co-creation of the music
experience. In this meaning, ORFI is more like the
improvisational musician, but in ORFI we can not count on the
user to have a professional musical know-how. It must be
satisfactory to both musical professionals and people with little
music competence, if we are to reach our ambitious goals.
In ORFI the audience changes continuously between roles in
different situations, from being a passive listener, to a musician
and a composer. Through long use the user gets deeper
knowledge about ORFI`s complexity and the user becomes
more like the improvisational musician, who with his
competence creates music on an instrument in real time. But the
user also comes nearer and becomes more like a composer over
time. The user becomes like a co-composer, which based on the
potential, the composer and writer of the software has
formulated, “composes” music by choosing and mixing music
together. In this way the real composer who has written the
software is present in the installation by continuously giving
musical answers, offering new musical possibilities to the user,
the co-composer. This is also a major difference between linear
an interactive music composition.
The Interactive Challenge
The composer of interactive music does not write notes on
paper or mix sound samples together to a linear track. He
creates music and software which totally or partly are the same,
depending on if the music elements are programs or sound
samples. This means that the composer composes potential
music, and software that controls the potential relation between
music elements. Music elements that will follow each other or
lie as layers on top of each others and be distributed in the 4-
channel system. All depending on what the user does, which
never can be predicted exactly. Writing software represent a
totally different potential than writing for an instrument, because
the computer can wait, remember and learn in a more or less
intelligent manner. Therefore one can write software so that the
installation or the interactive medium can behave more or less
like an active actor [7] instead of as an instrument. Playing a
traditional instrument a musical gesture on an instrument will
produce an immediate mechanical sound response [8]. Writing
software for a computer one can decide that a gesture or
interaction from the user, after a while will create a more
complex musical answer. This is more like the improvisational
musician, which after some time comes with his answer to your
solo play.
In ORFI we use both strategies in order to offer multiple
possibilities in all situations [2]. This means that the user
interacting with ORFI gets both a direct, immediate answer in
light and sound, as when playing on an acoustic instrument. But
after a little while he gets a complex musical answer to motivate
the user to further co-creation with ORFI. Examples of how it is
composed will be presented under “Interactive music
composition”.
ORFI is created so it continuously invites to collaboration in
different ways and through different media and forms [9]. Since
we have a very ambitious goal that ORFI shall work satisfactory
for most users in many situations over long time it is necessary
with an open concept of collaboration. Nothing is the right way
to do something. Nothing is wrong. It is as right to listen and
sleep in the interactive landscape as it is to throw the modules
between each other while playing. It is equally right to build and
shape one’s own interactive landscape, as it is to concentrate and
move the wings rhythmically for some minutes. To offer this
amount of openness we have put a lot of effort into the design of
ORFI in order to offer qualities like robustity, sensitivity,
ambiguity, musicality and consistency.
ORFI has to be very robust physically to handle being thrown,
stepped on and bended intensely. But it also has to be robust,
tolerant and sensitive in the software and hardware to register a
weak movement from a child’s hand and attempt to follow the
rhythm.
ORFI offers many visually, physically and musically
possibilities in many situation. It tries to answer and encourage
the users in different ways to musically interaction, in spite of
their competence, or lack of competence, in music. This means
that for some, in some genres, the lights rhythmical blinking
motivates the user to rhythmic interaction. But in other
situations, for other users, it is the complex dynamic graphic that
gives the user a visual image, and motivates the user to create
the musical narrative.
All these possibilities open up for various experiences for
different users depending on the individual’s competences and
experiences with ORFI. It has been very important for us to
design ORFI to offer many possibilities in every situation,
ambiguity [10]. But it has also been very important to give ORFI
a clear and unique identity. This so ORFI might act as a
convincing actor in a collaboration or improvisation. The
continuous change of roles the user can make. The many
possibilities the users are offered. The potential infinite uses and
81
the many consumption situations make the interactive
composition challenge much more complex than in linear music.
The fact that the nature of interactive music composition is
software, and not notes or samples, also makes it necessary to
structure the interactive composition in another way than linear
music.
Interactive Music Composition
So how have we met the interactive challenge of composing
potential music? How have we created algorithms and rules in
programming code that regulate the relations between musical
elements, the conditions for potential music? And how did we
compose music that motivated both professional musicians and
laymen to interact?
With concrete examples we will try to show how we have
composed the music for ORFI, exemplified by the two most
diverse genres.
We have chosen to structure ORFI’s software and music
composition into the following layers: sound node, composition
rule, and narrative structure (see Figure. 3).
Sound nodes are the smallest musically defined elements, such
as, tones, chords, or rhythmic patterns.
The sound nodes can be joined into sequences or parallel events
by composition rules, forming phrases, cadences, or rhythmic
patterns. The sound nodes can be joined into phrases or parallel
events by composition rules (algorithms). The user experiences
these phrases as narrative structures based on a genre.
Figure 3: Structure of the interactive music composition
software and interface in ORFI.
Figure 3 shows two users interact (bottom) with input sensors A
and B. The composition rules written in programming code,
selects the saxophone sound nodes (sax 1, 5, 7, 2) based on the
users interaction. Another composition rule creates the switch
from “ground 1” in high tempo, to “ground 2” in slow tempo, so
that it synchronise smoothly with the pulse without creating a
break. Over time the user creates a narrative structure of an 8 bar
jazz blues that motivates further interaction.
Sound Node
definition, mediation and qualities
We call the smallest musically defined elements sound nodes.
They are categorized into sound qualities like length,
instrument, pitch, harmony, tempo, meter, etc. Based on the
sound qualities and the composition rules, the program choose
and creates the narrative, e.g. a melodic motif, where the
expressive qualities depend on user interaction.
A sound node can be a linear sound file or programming code.
We have chosen to present the JAZZ and MINI genres, to show
our solutions in the two cases, sound samples vs. code.
The accompaniments (ground), horn riffs and saxophone sound
nodes in the JAZZ genre are sound files. The melodic patterns in
the MINI genre are programming code. The difference between
sound file and programming code is that the auditive result
formulated in programming code varies dynamically for each
interaction, while the sound file is essentially the same each time
it is played. This makes the programming code potentially more
flexible, since it can vary with user interaction.
creation of a node
Similar to traditional jazz, the blues in our JAZZ genre is
composed and recorded by a jazz ensemble [11]. Each musician
has recorded his instrument until the result is mixed down to a
jazz song.
After recording the music, we have cut and grouped the
recorded instruments into separate sound files. Then we have
arranged the files interactively by writing rules for ORFI [2].
Our arrangement builds on the style’s traditional "improvisation
on a theme".
nodes and node structure
In traditional jazz the musicians play on instruments with direct
response only. In ORFI the user might instead play on 20
physical soft modules. When interacting each module plays
three different saxophone sound samples depending on the
situation. The reason for this solution, is that we wanted to be
able to vary the expression from soft Ben Webster-like sound
nodes within Dorian scale, to hard, growling and dissonant
sound nodes outside the Dorian scale, and percussive saxophone
pad sound nodes. Which sound nodes the program combines and
how depend on if the users are active or passive, interact on their
own or collaborate, synchronise to the musical beat or not.
roles and experience
Similar to jazz, our interpretation has separate roles tied to the
different instruments. We use the tenor saxophone in the soloist
role, blues drums and walking base as accompaniment (ground),
and horn riffs played on saxophone, trombone and trumpet. An
important difference is that the interacting users continuously
can choose to change between the role of improvisator, soloist,
accompanist by choosing the module to play on etc. Therefore
the roles are potential and open for interpretation more than
definitely.
An example; When improvising, a saxophonist creates music
from pre-composed short motifs. He also creates phrases from
two contradictory curves of tension – amplitude and vibrato.
The amplitude curve goes from strong to weak (>). And the
curve for vibrato goes in the opposite direction from little to
much vibrato (<). We use the same strategy when composing
interactive music; This result in a sound node that potentially
has two gestures at the same time, a decreasing and an
increasing gesture. These contradictory curves of tension could
potentially function as the start tone, building up a tension in a
phrase, and as an end tone finishing the phrase creating a
release. The user, both laymen and musicians, can choose to
hear it as a tension or release depending on the situation.
In our JAZZ genre laymen can use the saxophone’s tension
creating curve for other purposes than the professional musician.
For instance speeding up a movement, rolling his body over the
soft modules spread out on the floor, while communicating with
a friend.
response and experiences
The melodic pattern generated by the programming code is
changing dynamically with the user interactions and other
melodic patterns playing at the same time. So when the user
82
interacts, the software realise one out of many possible
melodies.
Similar to playing on traditional instruments, the user gets direct
response when interacting. And at the same time the user
contributes to a musical whole.
The MINI genre gives immediate response in simple 2-6 notes
melodic motifs. The sound nodes also contribute to a complex
and musically satisfying response that motivate users to interact
with others over longer time.
Similar to traditional music our JAZZ genre uses different
instruments to create complex variations and contrasts between
the instruments. This is also the case for groups of sound nodes,
within an instrument.
code vs. tune
The sound nodes in the MINI genre are inspired by minimalist
music in the style of Steve Reich [12].
Similar to minimalist music our genre is characterised by
repetitions and small variations of short rhythmical and melodic
motifs, rather than large-scale development such as phrasing, or
sonata form. With less happening on a macro level the focus is
directed towards the surface and the micro level of small
changes in melody, rhythm and timbre.
What makes our MINI genre different is that every sound node
is a program (see Figure 4).
SynthDef(\pattSynth, {|out= 0, freq= 440, amp= 0.1, atk= 0, rel= 0.5,
max= 40|
var e= EnvGen.kr(Env.perc(atk, rel), doneAction:2);
var f= EnvGen.kr(Env.perc(0, 0.01), 1, 1, 0, Rand(0.95, 1.1));
var z= SinOsc.ar(freq*[1, IRand(0, 3).round(1).max(0.5)+Rand(1,
1.02)], f*Rand(10, 40), amp*0.1);
Out.ar(out, e*z);
}).store;
Figure 4: The sound node programming code for a synthesised,
marimba, written by Fredrik Olofsson in SuperCollider [13, 14].
Composition Rule
definition and mediation
Similar to traditional music the composition rule is composition
knowledge, the composer use to create the traditional musical
work.
It can for instance be knowledge about how to create relations
between tones, rhythm, melody, timbre and harmony in music.
Different from traditional music is that the composition rules are
programming code realised through use. For instance the sound
files in the JAZZ genre are controlled by the composition rules
in a program that consider both the musical and the interactive
development over time. Another example is the MINI genre
melodic pattern, where both sound nodes and composition rules
are formulated as programming code (figure 4) and where the
difference between sounds and rules is lacking. The result is that
the music can change dynamically, and that it sounds differently
over time, with different users and situations.
competence and experience
Similar to traditional music, a musician can use his
improvisational competences for making music by joining pre-
composed elements together.
Differently from traditional music is that a layman with less
musical competence can interact with the program and its
composition rules. The program interprets the interaction, delays
and changes the response in order to make it musically
satisfying, according to the composition rules.
The composition rules regulate the synchronisation of motifs to
the pulse after every user input. It also regulates tonal and
harmonic development so that they don’t contradict the genre
rules. Instead, the program waits for a rhythmically suitable
moment to play back motifs, and selects sound nodes that add
variation to the harmony and musical phrasing.
Unlike traditional music, laymen can communicate with each
other directly through the music. In ORFI laymen interact
actively and the program responds to individual, as well as
collective interactions.
An obstacle in traditional music is that it is hard to make music.
It is hard for a layman to keep the rhythm, pick the right notes
and create musically satisfying phrases.
In ORFI its different; The program and its composition rules are
tolerant, making it easier for laymen to synchronise to the pulse.
The composition rules tolerate deviations from what is
rhythmically correct and synchronise motifs to the harmony and
the pulse in other sound nodes. The result is avoidance of
technically difficult situations. And the laymen can instead focus
on communication and collaboration with others.
composition techniques in JAZZ
We have been inspired by traditional cool jazz and its modal
harmony, rhythmically improvised and laid back performing
style of such artists as Miles Davies and Ben Webster [15, 16].
In cool jazz a saxophonist can make it sound great letting the
instrument wander casually along a modal scale. He search his
way along the background of drums and falling fifths of a
walking base.
In cool jazz it is custom to use themes in modal scales with less
chords in order to make it easier to improvise freely, with focus
on rhythm and musical expression.
Similarly we use Dorian modality in ORFI, to motivate
improvisation and interaction.
As in cool jazz, interactive music composition, also use effects
like growling and dissonant saxophone with harsh timbre as a
musical rhetoric technique.
Unlike traditional jazz we use growling and dissonant
saxophone to express, stage and dramatize the conflict when
many users interact simultaneously. For instance when many
people play and tease each other, by interacting with many
ORFI modules at the same time. The result is that the program
creates many growling noises outside the Dorian scale, in
addition to the user created, soft and consonant tones.
As in traditional jazz we use soft consonant leading notes for
making musical ornaments. These motivate to improvisation,
such as call-and-response communication, and duets between
musicians.
Different is that the soft and consonant leading notes are used to
express pauses, motivating turn-taking between laymen. When
one or many users make pauses while in a sequence interact-
stop-interact-stop-interact, etc., the composition rules create soft
leading notes in Dorian scale. This in addition to the ones that
the user has chosen. The result is that the user becomes aware of
the silent pause between the interactions, and the relations
between his own actions and actions of others. This motivates
dialogue, imitations and play in call-and-response manner.
composition techniques in MINI
Similar to traditional minimalist music, the ORFI motifs borrow
polyrhythmic techniques from Gamelan, African and Middle
Ages music. Here, polyrhythmic and harmonic gaps in the
rhythmic patterns, make them fit into each other, creating
“hocket” patterns. This motivates improvisation and interaction.
A difference is that minimalist motifs are used to express
contrasting and varying responses that motivate laymen. The
composition rules then vary the pattern so that the hocket effects
disappear, in order to reappear when the rule for variation is
active again. A difference is that the music varies with the
number of users interacting, giving dub-delay effects to one user
and reverb to another. The result is a blurred and distorted
effect. The effect is used to separate between two individual
laymen motivating them to collaborate.
83
Narrative Structure
definition
We use the term narrative structure to describe structures for
connecting series of events in ORFI, creating experience and
expectations about future musical output.
role, action and expectations
The difference between linear music and ORFI`s music is that
the composer has to negotiate the narrative structure with
interacting users as well as passive listeners. Similar to linear
music, in ORFI there are often opposing or contradictory
expectations about what the narrative structures mean. For
example, a melodic structure driving the music forward creates
expectations of tension and a crescendo. In the same piece, a
rhythmic pulse in the ground, might create expectations of
bodily movements and dance to the pulse. Similar to linear music
it is possible for the users in ORFI to negotiate meaning,
following or denying expectations about the narrative structure.
Similar to traditional jazz, the narrative structure of our JAZZ
genre follows the development and tension in an 8 bar jazz blues
structure. Traditionally the blues structure is the ground for the
soloist’s improvisation over a repeated series of chords and a
pulse. Often the soloist creates expectations that follow the
convention, of playing as many rounds as he think suits him.
When he feels ready to hand over to the next soloists he gives a
sign, making cadences or finishing riffs. And the next person,
eager to make his interpretation and show off to the audience,
takes over. Building up to the moment just before the start of a
new round in bar 7-8, there exists a short period of 6-8 beats
where the tension is at top and the negotiation is strongest.
interpretation and negotiation
A difference in our JAZZ genre is that the system can analyse if
the user is synchronising his actions to the pulse in the
accompaniment. If he succeeds, ORFI answers with rewarding
horns riffs, stressing the harmonic, periodic and rhythmic
development in the blues.
Another difference is that the blues accompaniment with drums
and base in our JAZZ genre can be used to negotiate the
narrative structure. This is often made by users playing and
craving for more musical variations of a certain riff. We have
found that the accompaniment in addition can motivate two
users playing a game, dancing, or a person laying down resting
without focus on the music.
Another difference is that the accompaniment is divided into
three ground beats in different tempi creating possibilities for the
user to start, stop, change tempo, play together with horns, etc. It
increases the possibilities for the user to negotiate what and how
strong the narrative structure should be. When interacting, the
active users actions and the references to activities like dancing,
playing, creating music etc. produce, uphold and nurture a
narrative structure that potentially invites other users.
genres and experience
The traditional minimalist narrative structure follows the
development on a micro level, with fewer expectations on large
form structure development. It is almost a contradiction that
anything relevant should happen on the macro level in
minimalist music. Instead the expectations should be directed
towards the micro level and the tiny variations, we can hear if
we sharpen our senses.
A difference in our MINI genre is that the system organise the
synchronisation of the melodic patterns to the pulse. The
synchronisation is freeing the user from the responsibility to
keep track of the beat. Instead, we have found, it creates
possibilities for the user to focus on the communication and
improvisation with others.
The biggest difference, however is that it the user can choose to
negotiate what role to play, and if he wants ORFI to be a
tolerant minimalist sound carpet to sink down into, or a melodic
toy to throw and play with, or an improvising partner and active
actor, continuously inviting and motivating to communication.
same and different
We have tried to show how we have composed music for the
MINI and JAZZ genres in ORFI. We have found the comparison
between traditional popular music and its application to
interactive music to be very successful. Much of music’s
expressive qualities, variation and repetition techniques are the
same in interactive and linear music. A great deal of traditional
knowledge about analysis and composition of music can be
transferred to the interactive music.
Differences we found in the design of ORFI are primarily tied to
user expectations and structures in composition rules and
narrative structures to support those expectations. Our
experiences are that the interacting users need immediate
response to be able to orient and find their way, as well as more
complex response in order to get motivated to continue
interacting over time. We often found that musically complex
structures or processes found in traditional music could
strengthen other situations with laymen interacting and playing
alone and in collaboration with other people.
Add Perspectives
In our paper we have tried to show what we believe, are the
similarities and differences, between composing music for
interactive media compared to linear music. In our view, much
is the same, built on traditions that have been around for
centuries within music and composition. However, our main
conclusion about the new auditive medium is that we have to
broaden our perspectives. We have to put forward factors that
earlier was implicit in the musical and music making situations,
no matter if it was the concert hall, the church, or the club.
When composing interactive music we have to consider the
genre, the potential roles the listener might take, and the user
experience in different situations.
The consumer situation in interactive media is dynamically
changeable. Interactive music consumption can take place at
home, in the street, at school. It doesn’t need to be static, pre-
destined and hierarchical, with the professional and recognised
musician on stage and anonymous audience in darkness. In the
concert hall or the club the sound comes from a centrally placed
sound system. In interactive media, however, the sound can be
distributed and mobile, so that it moves and follows the persons
interacting.
The persons consuming the sound are not passive listeners
anymore, but active users, able to dynamically shift between
roles, by choosing position in space, relations and roles to other
people and the music. The user can take part in changing the
sound experience in real time, based on the rules the composer
has created as a potentiality in the software. This differs in a
significant way from the jazz improvisator or the professional
musician. The fact that the composer writes programming code
is an essential difference. Instead of writing one linear work, he
creates infinite numbers of potential music that reveal
themselves as answers to user interactions in many situations.
This might be like an instrument responding to a musical
gesture, or a competent and intelligent actor answering
musically in an improvisation session. But everything has to be
formulated in advance as rules in the software. The challenge is
to create music, through user interaction, that motivates to
further co-creation of the music and moving image narrative.
Everything has to be formulated in advance, based on genre and
music knowledge and competence in social behaviour. It’s all
about broaden the perspective to look wider, further and deeper.
84
Same but Different – Composing for Interactivity

Acknowledgements
Without Fredrik Olofsson’s unique artistic and technological
competence and knowledge in development of music, hardware
and software, ORFI would not have been possible to create. We
also thank Jens Lindgård, Petter Lindgård and Sven Andersson
for their work with music. We thank the Swedish
Inheritance Fund and Borgstena Textile AB for their
contributions. We thank Interactive Institute and K3 Malmö
University for being a source of inspiration to our work in the
group MusicalFieldsForever.

References
[1] Andersson, Anders-Petter, Cappelen, Birgitta, Olofsson,
Fredrik, ORFI, interactive installation, MusicalFieldsForever,
Art’s Birthday Party, Museum of Modern Art, Stockholm,
(2008)
[2] Cappelen, Birgitta & Andersson, Anders-Petter, From
Designing Objects to Designing Fields - From Control to
Freedom, Digital Creativity 14(2): 74-90, (2003)
[3] Fabbri, Franco, A theory of Popular Music Genres: Two
Applications, Popular Music Perspectives, Horn, D. & Tagg. P.
(ed.), Göteborg and Exeter: A. Wheaton, 52-81, (1982)
[4] Holt, Fabian, Genre in Popular Music, University of
Chicago, (2007)
[5] Tacchi, Jo, Radio Texture: between self and others, Material
Cultures, Why some things matter, (ed.) Miller D., London,
(1998)
[6] Weiser, Marc, The Computer for the Twenty-First Century,
Scientific American, 256(3): 94-104, (1991)
[7] Latour, Bruno, Pandora's Hope, Essays on the Reality of
Science Studies, Cambridge, MA; London, UK: Harvard
University Press, (1999)
[8] Godøy, Rolf Inge, Haga, Egil, Refsum Jensenius, Alexander,
Exploring Music-Related Gestures by Sound-Tracing. A
Preliminary Study, Congas, Leeds, (2006)
[9] Crawford, Chris, On game Design, US, (2003)
[10] Andersson, Anders-Petter & Cappelen, Birgitta, Ambiguity
- a User Quality, Collaborative Narrative in a Multimodal User
Interface, Proceedings AAAI, Smart Graphics, Stanford, (2000)
[11] Lindgård, Petter, Lindgård, Jens, Andersson, Sven (music),
Andersson, Anders-Petter (arr. & composition rules for
interactive installation), JAZZ genre, Do-Be-DJ/Mufi,
MusicalFieldsForever, (2000)
[12] Reich, Steve, Music for 18 Musicians, Recording, ECM,
(1978)
[13] SuperCollider, http://www.audiosynth.com, (2008
[14] Olofsson, Fredrik (music and composition rules for
interactive installation), MINI genre, ORFI/Mufi,
MusicalFieldsForever, (2007)
[15] ) Davies, Miles, Birth of the Cool, Recording, Capitol,
(1950)
[16] Webster, Ben (arr.), Arlen H., Koehler, T, I’ll Wind,
Recording, Soulville, Verve, (1957)

85
 The HarmonyPad
- A new creative tool for analyzing, generating and teaching tonal
music
G. Gatzsche, Fraunhofer IDMT, Ilmenau, Germany, gabriel.gatzsche@tu-ilmenau.de
M. Mehnert, Technische Universität, Ilmenau, Germany, markus.mehnert@tu-ilmenau.de
D. Gatzsche, Staatl. Berufsbildende Schule, "‘Janusz Korczak"’, Weimar, Germany, david.g@tzsche.de
K. Brandenburg, Technische Universität , Ilmenau, Germany, Karlheinz.brandenburg@tu-ilmenau.de

Abstract. Learning a classical musical instrument is a challenging task that requires long term practise, high
motor skills and intensive training. Within that relationship the following challenges exist: 1.) Students often
perceive pure score reading and reinterpretation as being boring. To teach musical improvisation additionally
would solve this problem. 2.) But to be able to improvise the student has to reach a certain technical level first.
3.) Pure score reading and reinterpretation does not automatically train the ability to improvise. Often very
good score players have difficulties to accompany a given melody. To overcome these problems a new musical
instrument is proposed that meets the following properties: The instrument is very easy to play. Its interface
is designed to reveal important structural properties of tonal music become geometrically: The relationships
between tones, intervals, chords and keys, functional aspects, aspects of consonance and dissonance and aspects
of tension and resolution. The instrument can be played without having extensive motor skills or prior music
theoretical knowledge. Through the usage of the proposed device the student implicitly acquires knowledge
about musical structure which again helps to compose, improvise, analyze musical pieces or to accompany a
given melody. Teachers can use the instrument to teach music theory.

1 Introduction To bypass or diminish the difficulties denoted be-

Many musicians are very good in score reading and
playing. But if they are required to improvise or to
compose difficulties arise. But particularly expressing
own feelings – mainly done through improvisation and
composition – is usually the heaven of every musician.
But why are there so many people that can play an
instrument but are not able to improvise and compose?
In our point of view there are three main reasons:
1. In the first years a musician is manly concentrated
to learn the required motor techniques and to be
able to read and play scores. But to have a good
grasp of this, much practice and a long play period
is required.
2. Additionally a musician has to learn music theory;
Beside playing the instrument much effort has to
be put into learning music theoretical terms and
basics of composition. This is often not or only
partially done because of lack of time, complexity
of the music theory and personal demotivation.
3. Without practicing technical motor skills and train-
ing music theory a musician is not able to learn the
highest level of music creation, that is the impro-
visation. The expression of own feelings during
play and musical articulation needs practice and of
course a set of different harmonic building blocks.
fore the following precondition have to be provided:
1.) Improvisation must not depend on the musician’s
personal motor skills. This means a musical beginner
should have the chance to improvise in very early sta-
ges of his musical career. 2.) A much better connection
of theoretical knowledge and its practical realization
has to be obtained. 3.) Musical elements like tones,
chords, keys, cadences, modulations etc. have to be
linked much stronger to visual associations. This would
help to maintain a conscious logical representation of
the learned things of musical elements beside the felt
and intuitive one. Furthermore the visual representa-
tion helps the student to build up and maintain a broad
musical vocabulary which can be used in improvisation.
In the following sections a creative tool is proposed
with the goal to realize the points denoted before. The
tool is called HarmonyPad, because it is mainly de-
signed for the creation of homophone harmonic song
and melody accompaniment. The HarmonyPad is very
easy to play: With only one finger it is possible to play
tones, intervals and complete chords, with a second fin-
ger the key can easily be changed. Therefore a musi-
cal beginner can easily learn to play chords. Further-
more the harmonic functions subdominant, dominant
and tonic are arranged in a similar way the human per-
ception handles music.

86
The HarmonyPad – A new creative tool for analyzing, generating and teaching tonal music
C e
a G
F h
d currently displayed.
IDMT001278
Figure 1: The HarmonyPad
1.1 Pitch space based musical instruments
The proposed creative tool is a so called pitch space
based musical instrument. Pitch space base musical
instruments consists of three main components: The
pitch space, the pitch selection and the user interface:
The pitch space defines a geometric arrangement of
pitches. Similar to color spaces the pitch space arranges
tones in a way such that semantic relationships between
the tones become geometrically apparent. Such aspects
are for example consonance or dissonance, chordal or
melodic grouping, cognitive similarities or simply the
pitch height. The better the tones are organized within
a pitch space the easier it is to generate a wanted sound
i.e. tone combination.
The selected area is a subregion of a given pitch space
which contains the tones to be played. If tones that
belong musically together are located in neighborhood
then it is possible to create meaningful tone combina-
tions through the definition of simple shaped selected
areas. By transforming the selected area (e.g. trans-
lation, scaling, rotating, inversion, ...) it is possible to
transform a selected chord into another one. Further-
more it is possible to assign alpha values to every point
1 Within this paper we focus on touch based techniques but it is
of the selected area which additionally assigns an indi-
vidual weight to every selected pitch.
The user interfaceis the mean by which the musician
controls the parameters of the pitch space (e.g. the
geometric arrangement of the pitches within the pitch
space), the selected area (e.g. the shape, translation or
rotation of the selected area. Furthermore the user inter-
face specifies the way the pitch space and the selected
area are presented to the user. The user interface can e.g.
be a touch screen, a game controller or a also a motion
capturing system.
Figure 1 shows the HarmonyPad as an example of
the things introduced to this point: In the inside of the
HarmonyPad shows the pitch space. The pitches are
represented by the small circles together with the note
names. The selected area is the grey circular segment
that marks the three tones C, e and G. Figure 2 shows
different controllers that can be used to move the se-
lected area to play chords or to change the pitch space
2 The user interface
The user interface of the HarmonyPad has to be some
kind of touch sensitive surface. This can be a normal
touch screen, a multi touch surface, a button matrix, a
pen display or also an innovative controller like the Re-
actable1 . The examples of the following explanations
have been implemented on the JazzMutant Lemur as
an multi touch controller and a Elo Touch Screen. The
advantage of the Elo touch screen is the possibility to
use the standard graphics driver to visualize the pitch
space, the selection function and also musical data. This
again allows a direct interaction with the visualized
pitch space. The player can touch and play the shown
tones or can transform the selected area. The drawback
of the Elo touch screen is that the screen it is not able to
process multiple touches simultaneously. This problem
is solved with JazzMutant Lemur which provides the
information of up to ten touch points. Another big ad-
vantage of the Lemur is the possibility to define different
user interface configurations very easily. The drawback
of the Lemur is that it is not possible to program and
show complex data visualizations and geometric mod-
els.
3 The pitch space
The HarmonyPad consists of two pitch spaces, a first
space to setup the key and a second pitch space to gen-
erate the actual chords. The pitch space that is used to
setup the key is the circle of fifth which is shortly denoted
also possible to use other controllers to control the pitch space
and the selected area (see Section 2) to get the music out of
the pitch space. So it would be possible to use a joystick like
controller like the 3DConnextion SpaceNavigator, a Theremin
based Controller or a innovative game controller like the Nin-
tendo WiiMote.
-2-
87
 The HarmonyPad – A new creative tool for analyzing, generating and teaching tonal music
IDMT001275
Figure 2: Possible controllers applied to con-
trol the HarmonyPad: From left to right: The
Nintendo WiiMote, The multi touch controller
JazzMutant Lemur, the 3dConnexion SpaceNav-
igator
with F.2 In Figure 1 F is represented by the twelve dots
shown in the outer ring. Every dot represents one of
the twelve major keys and its relative minor key in an
order according to the circle of fifths. Represented at a
touch screen the player can change the key by touching
the corresponding dot. The second pitch space is the
so called key related circle of third, shortly denoted with
TR.3 Key related means that the pitch space does not
contain all twelve chromatic pitch classes but only the
seven pitch classes of a selected key. In Figure 1 TR is
represented inside of F. The seven dots d, F, a, C, e, G, b
represent the seven pitch classes of the key C-Major. If
the player would select the key E-major by touching the
appropriate point at the outer circle, the TR of E-major
would be shown, i.e. the pitch classes f] , A, c] , E, g] , B,
d] . The TR was selected to be the basis of the proposed
musical instrument for the following reasons:
• The pitch space consists of only seven pitch classes.
This helps the musical beginner to be not confused.
If chords from other keys shall be played can use
the outer circle to change the key very easily.
• The pitch classes of the TR are in a order such
that three neighbored pitch classes generate a ma-
jor triad or a minor triad. This helps the player to
find chords very easily and to become conscious
about the pitch classes that form a certain chord.
• The pitch classes of the TR are symmetrically ar-
ranged around a so called symmetry axis. Tones
to the left of the symmetry axis form subdominant
chords (e.g. d-F-a, F-a-C), pitch classes to the right
2 The symbol F represents the term ”fifths”. More details and
semantic properties expressed by F can be found in [1] and [5].
3 The character T represents the term "thirds" and R stands for
"related", which means that the pitch space only contains pitch
classes of a certain key. The TR is part of the symmetry model,
a larger framework of circular pitch spaces which had been
introduced in [1] and [4]
of the symmetry axis form dominant chords (e-G-b,
G-b-D) and pitch classes centered around the sym-
metry axis form tonic chords (e.g. a-C-e, C-e-G).
This helps the student to become conscious about
functional relationships of chords.
As it can be seen the key related circle of thirds TR ex-
presses many music theoretical and cognitive relation-
ships by arranging only seven pitch classes. But pitch
classes alone are not enough to generate music because
every pitch class represents not an unique pitch but an
infinite number of pitches. An example is the pitch class
c, which represents the pitches c0 , c1 , c2 ,... cn . But what
of these pitches shall be played if the pitch class c is
selected? As an answer to this question we propose to
extend the key related circle of thirds TR and all other
pitch class based pitch spaces in the following way: The
pitch class of a pitch is represented at the first dimension
of a two dimensional space. We call this dimension the
pitch class axis. In the case of a polar coordinate sys-
tem this could be the angular axis. This is has already
been done in the case of the TR. The pitch height4 , i.e.
the different octaves of a pitch class are assigned to a
second dimension. We call this dimension pitch height
axis. In the case of a polar coordinate system this is
the radial axis. This is illustrated Figure 3: The pitch
classes d, F, a, C, e, G, b are assigned to the y-axis and
the pitch height (d0 , e0 , .. f 00 ) is assigned to the x-axis
of a two dimensional coordinate system. Therefore the
two dimensional coordinate of a pitch follows from the
pitch’s chroma and the pitch’s height. In Figure 3 the
resulting pitch positions are represented as black dots.
This solution has two big advantages which will become
obvious if the selected area (Figure 3, grey area) is in-
troduced: The shown selected area covers the a-minor
chord i.e. the tones e0 , a0 and c00 . Moving the selected
area towards the pitch class axis transforms the selected
a-minor chord into a C-major chord, but not by a sim-
ple shift (e0 , a0 , c00 → g0 , c00 , e00 ) which causes "forbidden"
fifths parallels. Through the proposed pitch arrangement
the second chord (dashed frame) is inversed automati-
cally such that a well sounding chord transition is gen-
erated (e0 , a0 , c00 → e0 , g0 , c00 ). The second advantage of
the proposed way to represent pitches in a two dimen-
sional pitch class/ pitch height space is the possibility to
generated very simply inversions of a given chord. In
Figure 3 the selected a-minor chord can simply be in-
verted by moving the selected area (grey area) towards
the pitch height axis (solid grey frame). In the example
4 The separation of pitch in the components pitch height and
pitch chroma is a fundamental of human auditory pitch per-
ception. A model that illustrates this is Roger Shepard’s spiral
helix [6]. Warren et al. [7] showed that pitch chroma resp.
pitch class and pitch height are processed in different regions
of the auditory cortex.
-3-
88
The HarmonyPad – A new creative tool for analyzing, generating and teaching tonal music

shown in Figure 3 the chord transition e0 , a0 , c00 → a0 , c00 , pressed button sound completely different then a whole
e00 is generated. tone shift? Through the geometric arrangement of tones
within the pitch space meaningful or often used tone
Pitch class
combinations are in geometric neighborhood and stand
in a simple geometric relationship. This again makes
D
it possible to define a simple formed shape, which cov-
ers neighbored pitch classes. Through a translation of
b the shape or a controlled change of the shape’s dimen-
G sions the desired sound can be formed. To make this
possible we have to think about a simple parameter set
e
to control the shape of the selected area. Such a set of
C parameters is presented now. The proposed parameter
set is explained using the TR but it can be applied to
a
every circular pitch space5 . In section 3 we proposed to
F represent pitches in a two dimensional coordinate sys-
tem. In the example of the TR this coordinate system is
d
a polar coordinate system whereas the first dimension
Pitch
is the angular dimension and the second dimension is the
height
d' e' f' g' a' b' c'' d'' e'' f'' ...gze@IDMT001270
radial dimension. Therefore the parameters start angle,
apex angle, start radius and apex radius are proposed to
control the shape of the selected area. This is illustrated
Figure 3: The pitch class/ pitch height space: The in Figure 4: The grey shape represents the selected area
first dimension of a two dimensional space repre- which is described by the four parameters named be-
sents the pitch height and the second dimension fore. We will discuss these parameters in more detail
the pitch class. The pitch space allows to generate now.
important chords and chord transitions by defin-
ing and moving rectangular areas.
C e

Start angle a
4 The selected area a a G
b Apex angle b
Principally it is possible to define arbitrary shaped se- r1 r2
lected areas to select and play tones. But the goal of Apex radius r2
this section is to propose an effective set of parameters Start radius r1
to describe the selected area. Through this parame-
terization it is possible to reduce the number of tasks
that have to be performed by the player to generate a F b

wished tone combination. In the example illustrated

d gze@IDMT000404
in Figure 5 the selected area is predefined such that it
covers major or minor triads. This results in the fact
that the musician needs only to touch a single point at Figure 4: A possibility to parameterize the se-
the instruments surface to play a whole chord. By mov- lected area is to use the four parameters start angle,
ing only one finger into a certain direction the original
apex angle, start radius and apex radius [2]
chord can be easily inversed or transformed into another
one. At the piano for example this task would be much
more complicated: The player has to define three points
i.e. he has to press three buttons. To transform the
played chord into another one he has to know which The start angle: The start angle defines the angular po-
of the three played tones have to be changed. There sition where the selected area starts. Figure 5 shows
are many possibilities to move the fingers in the wrong the selected area with three different start angles. The
way and the required finger movement does not stand
in a simple understandable relationship with the result- 5 Circular pitch spaces arrange pitch classes in a circular way [1].
ing perceptual effect: Why does a semi tone shift of a An alternative representation is the array representation [5].

-4-

89

The HarmonyPad – A new creative tool for analyzing, generating and teaching tonal music
apex angle of the selected area has been set in the way
that three neighbored tones are covered. Therefore it is
possible to play complete chord cadences by touching
the appropriate start angle at the touch surface. In Fig-
ure 5 the chord cadence F-major, C-major and G-major
is shown. The active tones have been marked red. Fur-
thermore it possible to crossfade a first chord into a
second by drawing the selected area from one point to
another.
The apex angle: The apex angle decides how many
tones are covered by the selected area. A small apex
angle like shown in the first example of Figure 6 covers
only a single pitch class. A continuous increase of the
apex angle leads to the effect that a single tone is cross-
faded into an interval and then into a triad and then into
a chord consisting of three tones (Figure 6). Through this
it is possible to also play single or two tone arpeggios
or also melodic lines. The example shown in Figure 6
crossfades the tone C into the interval C-e and then into
the major chord C-e-G.
The start radius: In section 3 we proposed to arrange
the different pitches of a certain pitch class along the
radius. This means that a movement of the selected
area along the radial dimension generates different in-
versions of the selected chord. In the current imple-
mentation we decided to assign lower tones to lower
radius and higher tones to higher radius. This means
that the height of a chord can be defined by locating
the selected area at the desired radius. This is shown in
Figure 7a: The figure shows a selected area that covers
the C-major chord in the lower octaves. The selected
area is moved along the radial dimension i.e. the pitch
height axis. Through this the selected C-major chord
is shifted towards higher octaves which means that it
sounds brighter and brighter. Analog to the crossfade
of different chords along the angular dimension it is
also possible to crossfade inversions of chords along the
radial dimension. Through the possibility to assign a
pitch height to complete chords the player can inter-
wove melodic elements into his or her chord play.
The apex radius: Analog to the increase of the apex
angle it is also possible to increase the apex radius of
the selected area. This means that tones from addi-
tional octaves are added to the chord currently selected.
Through the octave doubling of chord tones the chord
sounds brighter and fuller. Figure 7b illustrates how a
relative thin and dark C-major chord consisting of only
tones of one octave is transformed into a bright and full
chord consisting of tones from several octaves. This is
simply done by increasing the apex radius.
5 Playing non diatonic chords
In the following section it is discussed how chords can
be played that do not belong to the current set key. From
Änderungen bitte in IDMT001278 vornehmen und hier rein kopieren!
C e C e C e
a G a G a G
h h h
F F F
d d d
IDMT001271
Figure 5: Through changing the start angle of the
selected areaÄnderungen
it is possible to play tone and chord
bitte in IDMT001278 vornehmen und hier rein kopieren!
sequences.
C e C e C e
a G a G a G
F h F h F h
d d d
IDMT001272
Figure 6: Increasing the apex angle makes it pos-
sible to transform a single tone into an interval, a
triad or a four tone combination.
section 3 we know that the key related circle of thirds
TR only represents the pitch classes of a one key. While
for beginners this is a enormous simplification it will
become a limitation the more someone’s gets familiar
with the instrument. Therefore we have to look for ways
how to play chords from other keys or to play chords
that fit not into an unique key (e.g. the augmented C-
major chord c-e-g] ). All in all there are two possibilities,
to generate such chords: 1.) A change of the current
key. 2.) Rising or lowering single pitch classes of the
underlying pitch space.
5.1 Changing the key
In standard portable keyboards beginners can change
the key by simply transposing all tones by a number
of semitones. This has the following disadvantages:
The number of semitones that the current key has to be
transposed do not stand in a relationship with the result-
ing perceptual effect: A key change by a half semitone
would transpose the chord C-major into a the chord C] -
major. But such a chord combination is very seldom
needed. Other key changes that are more often used,
for example to change the chord G-major into the chord
D − major would require seven five changes. Another
drawback of the transposition of the whole pitch system
-5-
90
 Änderungen bitte in IDMT001278 vornehmen und hier rein kopieren!
The HarmonyPad – A new creative tool for analyzing, generating and teaching tonal music
C e C e
a G a G
F h F h
d d
IDMT001274
Figure 7: a) Increasing the start radius crossfades
a chord through different inversions. b) Increasing
the apex radius adds tones from other octaves to
the chord
is the generation of fifths parallels which are forbidden
in classical composing. But the problems shown before
can be solved with the following two steps: 1.) Instead
of using a semitone based transposition we propose to
perform the key changes based on the circle of fifths.
The circle of fifths represents all diatonic keys in an or-
der, that keys with more common tones are located close
together, keys with less common tones are located more
far away. For example the keys C-major and G-major
have six of seven tones in common, that are the tones c,
d, e, g, a and b. In the circle of fifths C-major and G-major
are in neighborhood. Another advantage of using the
circle of fifths for key changes is that the relative minor
key and the parallel major key of a given key are local-
ized in a simple geometric ratio with the current key. If
the current key is the key C-major then the parallel key
c-minor can be found by selecting the key −90◦ of the
current one6 . The last advantage of using the circle of
fifths is that it reveals symmetries between different key
changes: To find the parallel major key of a minor key
is done in nearly the same way like finding the parallel
minor key of a major key. The difference is that not the
key -90◦ but the key +90◦ of the current one has to be se-
lected. Revealing this symmetries helps music students
to recognize structural redundancies and to internalize
music theoretical knowledge much effective.
The realization of the proposed way to change keys
can be seen in Figure 1. The outer ring consists of twelve
grey points which all represent a key. The assignment
of keys to the grey points is not absolute: The point
at the circle’s top represents the key currently selected.
To the left and to the right of the current key the other
keys in an order according to the circle of fifths follow.
If the musician wants to change into the parallel minor
6 We do not use the standard mathematical system where the
angle 0◦ is located at the x-axis and angles increase counter
clockwise. We use the musical coordinate system. Here the
angle 0◦ is located at the positive y-axis i.e. the circle’s top.
Furthermore the angle increases clockwise
key for example, he/she has to touch the point located
at -90◦ . All key changes become immediately visible
and audible. Therefore it is possible to play the cadence
C-major, F-major, G-major and C-major by selecting the
chord C-major (like done in Figure 1) and touching the
appropriate key points at the circle of fifths.
With the help of the circle of fifths we could reach a
more intuitive way to change the key and to play key
from other chords. But the problems of fifths parallels
still exists: If we select the key C-major and change
to the key G-major then the played chord C-major is
transposed by seven semitones and a forbidden fifths
parallel occurs. To solve this problem we have to go
back to the pitch class/ pitch height space shown in
Figure 3: The space is not simply transposed but shifted
along the pitch class axis. As described in section 3 this
leads to the automatic generation of a well sounding
chord transition.
5.2 Rise or lower single pitch classes
The second way to play chords from other keys is to rise
or lower single pitch classes. This means for example to
transform the chord C-major into a chord C-minor the
pitch class e can be lowered by a half semitone to become
the pitch class e[ . Like denoted above this alternative
allows it also to generate non diatonic or other dissonant
chords. A possibility to implement such a feature is to
assign an appropriate user interface element to every of
the pitch classes. With such an interface element it is
possible to rise the appropriate pitch class by a certain
number of semitone.
6 Pedagogical application
Combined with a substantiated pedagogical concept the
HarmonyPad becomes a helpful tool in early music edu-
cation e.g. in kindergartens, in primary schools but also
in schools, music schools. Music students should use
the tool to improve their music theoretical knowledge
and their ability to compose. Through the simplicity of
play older people can use the HarmonyPad to learn a
musical instrument still in advanced years. The sub-
sequent paragraphs summarize music theoretical resp.
tonal relationships that can be taught using the Har-
monyPad:
1. The student learns the tones that build the most
often used major and minor chords. By selecting
a narrow region at the surface of the HarmonyPad
the student can listen to single tones. Enlarging the
selected region the single tones can be crossfaded
into major and minor chords.
2. The student learns functional relationships: By se-
lecting an area on the left side of the HarmonyPad
-6-
91
 The HarmonyPad – A new creative tool for analyzing, generating and teaching tonal music
the subdominant chords S; Sp , by selecting an area
on the right side the dominant chords D; Dp and
by selecting an area in the upper center the tonic
chords T; Tp can be played7 .
3. The HarmonyPad links visual geometric posi-
tions, musical gestures, musimathematical struc-
tures and of course the resulting sound effect. This
again trains to think in harmonies and to remember
musical elements.
4. The circular arrangement of pitch classes in the
HarmonyPad allows to visualize chord progres-
sions as geometric tracks. Chord progressions can
be linked with a geometric shape. For example the
chord progression T,S, D, T can be visualized as a
triangle at the HarmonyPad’s surface.
5. By playing the HarmonyPad students learn to as-
sign minor and minor chords to its relative minor
and major chords. This is possible because major
and its relative minor chords occupy neighbored
regions8 the HarmonyPad.
6. The student learns automatically common tones of
third related chords [3, S. 142]: For example the
chords a-minor and C-major have an overlapping
region which contains the common tones C and e.
7. With the HarmonyPad it is very easy to learn,
which tones can be used to accompany a given
tone in a given key. To find the right tone the se-
lected area has to be setup such that it contains
three (or four) tones. After that all chords that can
be covered by the previously defined selected area
and that contain the tone to be accompanied can be
used to accompany the denoted tone9 .
8. Last but not least the student learns that there are
different inversions of every chord. This inversions
can be easily created by moving the selected area
towards the pitch class axis.
7 Summary and conclusion
The state of the HarmonyPad described to this point is
a base around which many improvements like a pitch
height dependent change of the apex angle to prevent
dissonant tone combinations in the lower frequency re-
gions, the integration of multi touch controllers like the
JazzMutant Lemur or other geometric representations
like a cartesian one have been implemented. The out-
come of these further developments is that it is now
7 To find out more about the symbols S, Sp , T, ... look at [3]
8 For example a − C − e and C − e − G
9 At this point it becomes clear that the material for the harmonic
accompaniment are won from the own scale’s tones. [3, S.11]
required to optimize the HarmonyPad for a given tar-
get group, this can be infants, children, music pupils,
music students but also older people. All in all it can
be said that the HarmonyPad complements the piano
keyboard in a very good way: While the piano key-
board organizes the tones along melodic relationships
the HarmonyPad does the same for harmonic relation-
ships. For this beside the piano the HarmonyPad should
become a central part of every scholar and private music
education.
References
[1] G, Gabriel ; M, Markus ; B-
, Karlheinz ; A, Daniel: Circular Pitch
Space based Musical Tonality Analysis. In: 124th
AES Convention (2008)
[2] G, Gabriel ; M, Markus ;
S̈, Christian: Interaction with tonal
pitch spaces. In: Proceedings of the 8th International
Conference on New Interfaces for Musical Expression
NIME08, 2008
[3] K̈, Thomas: Harmonielehre im Selbststudium.
Neuausg. Wiesbaden u.a. : Breitkopf und Härtel,
2006. – ISBN 9783765102615
[4] M, Markus ; G, Gabriel ; B-
, Karlheinz ; A, Daniel: Circular Pitch
Space based Harmonic Change Detection. In: 124th
AES Convention (2008)
[5] M, Markus ; G, Gabriel ; G,
David ; B, Karlheinz: The analysis of
tonal symmetries in musical audio signals. In: Inter-
national Symposium on Musical Acoustics ISMA 2007,
2007
[6] S, Roger: Geometrical approximations to the
structure of musical pitch. In: Psychological Reveview
89(4) (1982), Jul, S. 305–333
[7] W, J. D.: Seperating pitch chroma and pitch
height in the human brain. In: Proceedings of the
National Academy of Sciences of the United States of
America, 2003, S. 10038–10042
-7-
92
 Sonic interactions with hand clap sounds
Antti Jylhä and Cumhur Erkut
Dept. Signal Processing and Acoustics
Helsinki University of Technology
antti.jylha@tkk.fi

Abstract. In this paper, we present a control interface, which applies the hand claps of the user as control input.
With this system, we aim at providing more realistic and engaging control over the output. We present three
exemplary use cases: controlling a synthetic audience, controlling the tempo of a musical piece, and controlling
a simple sampler. Qualitative evaluation shows that the system performs well in the use cases. The control
interface has potential in other types of human-computer interaction as well.

1 Introduction Hanahara, Tada, and Muroi [4] have recently pro-

posed a novel method for human-robot and robot-robot
Analysis and synthesis of everyday sounds is gaining
interaction based on a hand clap language. Instead of
more and more attention in the research field and
two separate communication systems, their proposal
among practitioners. Sound designers of computer
implies a uniform communication system to enable
games and movies, for example, could make use of
more genuine social interaction among humans and
novel ways of adding synthetic, yet realistic and en-
robots. They have defined a set of hand clap patterns
gaging sounds to their products. Furthermore, sonic
as syllables, which can be concatenated to form words
interaction design1 (SID), a new branch of interaction
and meaningful constructs for both humans and robots
design [12], has increased the demand and discussion
to understand.
of using everyday sounds as conveyors of information.
In any context, to extract meaningful control infor-
As a young research area, SID still lacks tools for sys-
mation from hand clap sounds, the claps must be iden-
tematic development and evaluation of sonic interac-
tified. Considering the onset detection in audio signals
tion. The same goes for sound design. In this study,
and the tempo tracking of a musical piece, numerous
we explore the possibility of applying hand clapping, a
techniques exist. For onset detection, a detailed review
simple sonic gesture, as the control signal of an inter-
of techniques has been presented by Bello et al. [1]. A
active system. By sonic gesture we refer to a human
real-time onset detection algorithm and its implemen-
action, which is intended to result in a specific sound.
tation for the GPL-licensed Pure Data (PD)3 software
The analysis and synthesis of hand clap sounds has
[9] has been depicted by Puckette et al. in [10]. For a
been previously studied for example by Peltola et al.
review of tempo tracking and beat tracking techniques,
[8], who describe two different software implementa-
see e.g. [3] and [11]. A tempo tracker based on the lat-
tions for synthesis and parametric control of hand
ter work has also been implemented for PD under the
clapping sounds. Although the implementations dif-
name rhythm estimator4 .
fer, both systems aim at the synthesis of realistic clap
sounds by physically-based modeling. An alternative Audio Gestures5 is a real-time software implemen-
applause synthesis implementation has been presented tation for using general non-speech audio gestures in
by Farnell2 . human-computer interaction. Audio Gestures provides
means to train the computer discrete commands based
Lesser and Ellis [7] have studied hand claps in the
on the user’s non-speech audio gestures. The imple-
context of rhythm therapy. They propose a technique
to detect and evaluate the claps of a student, who has 3 Available at http://puredata.info/downloads.
been instructed to repeat a given rhythmic pattern by 4 Available at ftp://iem.kug.ac.at/pd/Externals/
hand claps. Their aim is a multi-student system, which RHYTHM/
5 Audio Gestures was implemented as a course project in
could be simultaneously applied by all the students in
a classroom. Princeton University on the course Computer Science 436
Human-Computer Interface Technology, see http://www.
1 http://www.cost-sid.org cs.princeton.edu/courses/archive/fall07/cos436/ and
2 http://obiwannabe.co.uk/tutorials/html/ http://www.cs.princeton.edu/courses/archive/fall07/
tutorial_applause.html cos436/FinalProj07.html.

93
[Sonic interactions with hand clap sounds]

mentation has been built on sndpeek6 , a package for possible for the user to perform some other music on
real-time audio visualization. the beat.
In this study, we aim at providing a prototype of It is noteworthy that the three-agent system of the
a control interface, which applies hand claps as con- user, a piece of music, and the synthetic audience, is
trol input. In the current implementation, the control not restricted to this form of interaction. In another
interface can be applied to control a synthetic crowd possible scheme, it could be the audience controlling
of clappers, the tempo of a musical piece, or a simple the user to clap, and the music would follow. Alter-
sampler, by hand clap gestures in a continuous fashion. natively, the user could become the conductor of both
This implementation should be considered an early ex- the audience and the music simultaneously.
ploration of the possibilities of such a system, and we
aim at extending the framework to other sound syn-
thesis and human-computer interaction applications as
well. Examples of intended future implementations in-
clude interactive games, novel HCI schemes, and a tool
for sound and interaction designers for prototyping and
evaluation.

2 Applications
The current implementation of the system can perform
in three different functional modes. In the audience
mode, the user aims at synchronizing a synthetic au-
dience with her clapping tempo and possibly with a
musical piece by clapping to the beat of the music. In
the music tempo control mode, the user claps her hands
to control the tempo of music. In the sampler mode,
the user claps her hands to control a simple table-read
sampler. Technical details of the implementation will
be discussed in Sec. 3.

2.1 Synchronizing the audience to the beat

Combining sound synthesis of a virtual audience with
tempo tracking of hand claps, we can provide a means
for controlling a crowd of clappers to the beat of music.
This use case is illustrated in Fig. 1. Given a piece
of music, the user (The Patron) claps her hands to
the beat of the music. This way, the user becomes
the music information retrieval system, and the tempo
of the user’s clapping hands is applied to control the
audience. Once the user claps to the beat, the audience
follows, and so the user becomes the conductor of the
audience in this case, being also part of the clapping
crowd. As a result, the audience becomes synchronized
with the music. BPM (beats per minute) of the music
is presented to the user based on the clap tempo. This
way, the user can as a by-product of the system also
determine the BPM of a musical piece, assuming that
she claps to the correct beat.
The user interface can directly be applied to control
the virtual audience of clappers also without any ref-
erence music. This way, it is possible for example to
train the crowd to clap to a desired beat, making it
6 http://soundlab.cs.princeton.edu/software/
sndpeek/
Figure 1: The audience synchronization system.
2.2 Controlling the tempo of music
The clapping interface can be applied to control the
tempo of a musical piece. For this, there are two op-
tions. First, the music can be a pre-recorded piece,
which is then time-stretched in real-time according to
the tempo of the clapping. In a second scheme, the clap
tempo can be applied to control a sequencer in PD.
Given for example a pre-programmed musical tune or
a random-walk sequencer, it is straightforward to map
the clapping tempo to the tempo of the music. The
strength of the claps can be mapped to the volume, for
example.
In this study, we implemented the first option.
When the user claps her hands to control the tempo,
auditory feedback on the claps is also presented to the
user to indicate the detected claps.

-2-

94
[Sonic interactions with hand clap sounds]
2.3 Hand-clap-driven sampler
Another simple application for the interface in the mu-
sical domain is a sampler controlled by the clapping
hands. The user can select an audio sample (.wav-file)
and then control the playback of the sample with the
claps. This application can also function in two ways.
The sample can be automatically looped so that the
tempo of the claps can control the looping rate and
the rate of reading the sample from a table. Alterna-
tively, the sample can be played every time a clap is
detected.
A clap-driven sampler could be applied for exam-
ple to control the tempo of a drum loop in a musical
performance, by mapping the clap tempo to the loop-
playing rate. Another example could be the sound de-
sign of movies, when discrete auditory events are to be
placed in the movie soundtrack in a rhythmic way or in
instantaneous locations. The sound designer could for
example mimic the rhythm of a monster’s footsteps
with her hand claps to glue the samples into desired
place.
3 System architecture and technique
The architecture of our system consists of a computer
running PD, a microphone, and the user. The user is
an integral part of the system, as the user’s gestures
are needed to control the system.
Capturing the hand clap sounds is easy to do with
any conventional microphone. To extract control infor-
mation from hand claps in our system, the microphone
does not have to be of high quality. This makes the
system widely applicable, because consumer-oriented
computer microphones are not expensive, and many
computers even have a sufficient built-in microphone.
PD is a graphical programming language that was
originally developed for audio signal processing [9]. PD
programs consist of graphical patches, which may con-
tain different objects (functional elements), messages
(parameters for the objects), arrays, and other pro-
gramming elements. These elements are connected to
each other from their inlets and outlets by drawing a
line between them, i.e., routing the data through pro-
gramming commands. For example, a summation ele-
ment contains two inlets and one outlet to accept two
numbers to sum and to give their sum as output. A
PD patch may contain many nested parts, which may
be either subpatches (saved along with the main file),
abstractions (graphically programmed individual .pd
files), or externals (PD objects compiled from C code).
Control data, e.g., numbers and strings, is processed
without real-time requirements, while signals are pro-
cessed in blocks of data at a different rate.
For the virtual audience application of Sec. 2.1, PD
runs a modified version of the hand clap synthesis en-
gine ClaPD [8], with a new control interface and addi-
tional functionality. In its previous versions, the con-
trol parameters of ClaPD have only been adjustable by
conventional HCI techniques, i.e., the mouse and the
keyboard. Here we propose a technique for using hand
claps as input to extract control parameters for the sys-
tem. The hand clap audio data is processed with PD to
yield parametric control data for the synthesis engine.
ClaPD and the control interface are described in detail
in the remainder of this section. Also the techniques
for implementing the music tempo control and sampler
applications are discussed.
3.1 Extracting control parameters
From the user’s hand claps, we extract three types of
control parameters: onsets, tempo, and strength. For
onset detection, we have experimented with different
envelope-based and band-based methods. In this pro-
totype, we chose to apply a readily implemented PD
object known as bonk∼, which is an object designed for
detecting and classifying percussive sounds [10]. The
algorithm is based on analysis of the incoming signal in
11 frequency bands in overlapping time windows. The
overall change in the power of the bands is applied
for detecting an attack. The bonk∼ object can also
be trained to classify percussive sounds by template
matching [10].
The output of bonk∼ is the power of each frequency
band, the tag of the class if classification task is rele-
vant, and the summed-up power of the subbands. We
use the output to determine the onset to onset interval
(OOI) between subsequent hand claps with a simple
deterministic tempo tracker. Every time bonk∼ gives
an output, an onset has occurred, and we can use this
onset information for further rhythm estimation. From
bonk∼, we also directly obtain the power of the attack
as an estimate for the strength of the current clap. Al-
though the clap strength remains unused in this pro-
totype, its potential is acknowledged for future work.
The onset information obtained from bonk∼ is
further processed by the rhythm estimator object5 ,
which is an online tempo tracking object for onset in-
formation, i.e., bang messages in PD. The rhythm es-
timator algorithm was originally developed for tatum
grid analysis of musical signals [11], but it serves well to
provide an estimate of the intended clapping tempo in
our research, too. Tatum is the smallest metrical level
in musical rhythm, which in the case of quasi-periodical
hand clapping translates to the average duration be-
tween successive claps. The algorithm is described in
more detail in [11]. It is based on analyzing the OOI’s
by storing the OOI values into a time-varying OOI his-
togram (inter-onset interval (IOI) histogram in [11]).
-3-
95
[Sonic interactions with hand clap sounds]

This analysis is performed continuously not only for

each consecutive onset pair, but for each detected on-
set within a specified time frame. The histogram is
updated for every new onset as a leaky integrator, and
a remainder error function is calculated for the his-
togram information as a function of the tatum period.
The minimum of the error function is searched for by
parametric thresholding, and if the threshold is met,
the point is chosen as the estimate of the tatum pe-
riod.
The rhythm estimator object’s right outlet gives
the estimate of the average OOI as intended by the
performer of the rhythmic sequence. The value is in
milliseconds, so it can be converted to BPM as
60
BP M = 1000 ∗ . (1)
OOI
While experimenting with rhythm estimator, we
found out that the OOI histogram array in PD was
of fixed size. In practice, this means that the his-
togram bin width (i.e., resolution) is proportional to
the maximum OOI allowed by the histogram. This
leads to a tradeoff between BPM resolution and lower
bound. With default settings (OOI resolution = 5 ms),
the maximum OOI detected by the system is 395 ms,
yielding the lowest possible BPM of 151.9 according to
Eq. 1, which is very high for most musical styles. Al-
though it is shown in the rhythm estimator example
Figure 2: The clap tracker PD abstraction. The
patches how the histogram could be made longer by two inlets accept audio data (left) and bang messages
PD messaging, this feature seemed not to work with (right). The object outputs estimated clap strength as
the test platforms. However, changing the histogram the velocity output of the bonk∼ object and the OOI
resolution did work, so we changed the resolution to and BPM values calculated by the rhythm estimator
object. The ooi2bpm object converts OOI values to
13 ms in order to obtain maximum OOI of more than BPM. For bonk∼, a threshold value is provided to
one second. This would be required to detect BPM’s make the abstraction more robust against environmen-
of 60, which we assumed as a reasonable lower bound tal noise. The spigot tests if clapping was selected as
for this prototype. input type before announcing the detection of a real
By combining bonk∼ and rhythm estimator, we cre- clap.
ated a clap tracker PD abstraction to detect the
claps of the user and to estimate OOI, BPM, and the
strength of the clap. The abstraction is depicted in
which are fine-tuned by hand-clapping statistics. It
Fig. clap tracker. To make it possible to compare the
can produce expressive, human-like synthetic applause
gestural control interface to a more conventional input,
of a single clapper with adjustable hand configuration,
the clap tracker also accepts bang messages into its
or asynchronous or synchronous applause of a clapper
other inlet. In our implementation, these messages can
population (audience).
be generated by mouse clicking in the graphical control
The three layers in ClaPD form a sound synthesis
interface depicted in Sect. 3.6.
and control hierarchy: the event layer determines the
clapping mode (synchronous/asynchronous), the con-
3.2 Synthesis of hand claps with ClaPD
trol layer determines the tempi and the hand configura-
ClaPD7 is a PD library for hand clapping synthesis and tion of the individual clappers, and the synthesis layer
control [8]. It contains low-level synthesis and higher- carries out the actual audio calculation. The possibility
level control blocks, and primitive event generators, of interacting with the audience via a master clapper,
7 ClaPD has been released as a free software under the that is, the master clock each clapper listens to in the
GNU Public License (GPL) and can be downloaded from control layer, was indicated in [8], but left unexplored.
http://www.acoustics.hut.fi/software/clapd After the initial release of the ClaPD (rel-0.1), its au-

-4-

96
[Sonic interactions with hand clap sounds]
dience mode has been significantly extended [2]. Cur-
rently the audience can be dynamically generated and
multiple sound generators can be hosted in ClaPD. The
last property allows us to implement a special Patron
object, which by listening to tapping or clapping of the
user can drive the audience to synchronize with itself
within the limits of the virtual audience. These lim-
its are imposed by the cosc (coupled oscillator) object
that performs the entrainment of each virtual clapper.
In the synchronized mode, each clapper aims to clap
around the same rate (frequency-locking) and absolute
time (phase-locking) with the Patron, and calculates
its phase difference (measured in milliseconds) with the
Patron. Since the Patron has a quasi-stationary OOIP ,
the phase difference can be considered a uniform distri-
bution U (0, OOIP ) with the mean of OOIP /2 ms. If a
clapper is trailing behind of the Patron, then the phase
difference is smaller than OOIP /2 ms and its clapping
rate is accelerated. Similarly, if the clapper is ahead
of the Patron (phase difference greater than OOIP /2)
ms its clapping rate is slowed down. The exact expres-
sions for the acceleration and deceleration are given in
[8] for the constant Patron OOI of 440 ms.
This scenario can be extended to generate syn-
chronous virtual applause along an external rhythmic
piece of music, as explicated in Sec. 2.1. In our exper-
iments, we found synchronizing the ClaPD clappers to
be easy, when clapping around the built-in preferred
clapping rate of the virtual clappers (440 ms), but clap-
ping much faster or slower did not lead to synchro-
nization. Therefore, we have made the Patron rate
variable. Admittedly, this approach makes the origi-
nal model behave in a less natural way, but gives good
results in synchronizing the audience.
3.3 Music tempo control
To provide control over the tempo of recorded mu-
sic without pitch artifacts in the resulting sounds, a
time-stretching technique is required. As a prototype
time-stretching method, we use a phase vocoder pro-
vided with the PD distribution3 in the help patch
I07.phase.vocoder.pd, with added functionality re-
sponding to external user control. The phase vocoder
applies short-time Fourier transform (STFT) to de-
compose the incoming signal into frequency bands, and
resynthesizes the signal based on the analysis of the de-
composed signals [6]. In the resynthesis, it is possible
to scale the time of the output, which we apply for ad-
justing the tempo while maintaining the original pitch.
Basically, the estimated BPM of the clapper is
mapped to the BPM of the music. This is done by
adjusting the precession speed of the phase vocoder
synthesis stage according to the user input. The phase
vocoder accepts the precession speed in hundredths of
a second. In order for the mapping to work realisti-
cally, that is for the clapper’s claps to match the beats
of the music, the reference BPM of the original song
needs to be known. If the reference is not known be-
forehand, it can be determined by the user by clapping
to the beat of the song in the virtual audience mode.
The mapping of the clap BPM to the phase vocoder
precession speed is linear, defined as
BP Mclap
v = 100 ∗ , (2)
BP Mref
where v is the precession speed, BP Mclap is the esti-
mated clap BPM, and BP Mref is the reference BPM.
If a wrong reference BPM value is applied, the claps of
the user will still control the tempo, but they will not
match the beats of the music.
When the user claps to control the music, sonic feed-
back is provided to the user to indicate the detected
claps. This feedback is a hand clap sound sample8 ,
which has been written to a table and is played back
every time a clap has been detected. While it would
also be possible to route the clapper’s own clapping
sounds back to the audio output, this would result in
feedback problems if loudspeakers are used to repro-
duce the sounds.
The major downside of this time stretching tech-
nique is that the applied phase vocoder implementa-
tion does not readily work with streaming audio. There
are also some audible artifacts in the processed sound,
such as ”phasiness” and ”loss of presence”, which are
characteristic to phase vocoders [6]. However, for pro-
totyping purposes, the simple phase vocoder proved to
be quite sufficient.
3.4 Sampler
Our implementation for the sampler follows a simple
interpolating table-read approach. The user can select
the sample she wishes, which is then written to a table
in PD. Depending on the user’s preferences, the sample
can be read from the table either every time a clap is
detected, or sequentially according to the tempo of the
user’s claps.
In the non-looping mode, the user’s claps act as a
trigger for the samples. For each detected clap the sam-
ple is simply read from the table it has been written to,
and played. In the looping mode, the estimated OOI of
the user’s claps is mapped directly to the looping rate,
i.e., the sample is repeated sequentially according to
the tempo of the claps. Also the pitch of the sample is
allowed to change by reading the sample from the ta-
ble with speed inversely proportional to the estimated
8 The sample can be freely obtained from http://
bigsamples.free.fr/d_kit/clap/handclap.wav.
-5-
97
[Sonic interactions with hand clap sounds]
OOI. The table reading speed σ (samples per second)
is
L
σ= , (3)
OOI
where L is the length of the sample in samples. Thus,
slower clapping results in a sequence of slowly repeated
low-frequency sounds, while rapid clapping leads to a
rapid looping of high-pitched instances of the sample.
3.5 Latency
In the applications, there is an amount of latency in-
troduced by the system between the excitation and
response. The latency as a whole is a sum of many
software and hardware-related latencies, including the
soundcard latency, PD audio buffering, and the com-
putational latency of the PD program. The overall
system latency is strongly dependent on the operat-
ing system and its audio drivers, and the way these
are configured. While it is not straightforward to mea-
sure all the different latencies, the latency within the
PD program can be estimated by PD’s own realtime
object, which calculates the elapsed time between two
program events.
As a result of the latency, the user’s claps and the
claps of the virtual audience are not trivially simulta-
neous. Measuring the latency and taking into account
the cyclic nature of clapping, a simple remedy for the
problem is delaying the system response to match the
time of the user’s next predicted clap. This prediction
can be calculated as the difference of the estimated
time between claps and the measured latency.
Naturally, latency appears also in the other func-
tionalities, where it cannot be compensated for with
the simple trick used in the audience mode. On the
other hand, informal evaluation of the system indicated
that while the latency is noticeable, it is not necessarily
disturbing.
3.6 Graphical control interface
The graphical control interface is presented in Fig. 3.
Although the actual interaction with the system is per-
formed with hand claps, a conventional interface for
selecting the functionality and other relevant controls
is required.
The control interface consists of a selector for the
functional mode, a selector for the input type, a BPM
visualizer, output gain and audio on/off settings, and
GO! and STOP! buttons for performing mode-specific
start and stop commands. In the virtual audience
mode, the GO! button starts the synthetic applause,
while the STOP! button ends it. In music tempo con-
trol mode, the user should first adjust the song BPM
to the normal BPM of the piece. Then loading a song,
hitting GO! and starting to clap will start playback of
Figure 3: The PD control interface of the example ap-
plications. The user can select the functional mode and
adjust the other settings as required. The BPM esti-
mated from the user’s claps is presented in all modes.
the music at reference BPM, and STOP! will end the
playback. In sampler mode, GO! and STOP! will start
and stop a loop, if looping has been selected. There are
also buttons for loading the music or the sample, i.e.,
opening up an ”Open file” window, so that the user
can select the music she wants to control or the sample
to use in the sampler. In this prototype, these must be
.wav files.
In the virtual audience mode, a visualization of the
clapping crowd is presented as a blinkenlights9 can-
vas, i.e., a grid of flashing pixels indicating individual
clappers. The middlemost pixel in the grid visualizes
the claps of the user.
4 Informal evaluation of the system
To evaluate the system, we requested two persons to
try out the user interface. The emphasis was on the
virtual audience mode and the tempo control mode.
The persons were first instructed to clap their hands
and try to get the virtual audience synchronized with
them. Both test subjects reported that the synchro-
nization feels both realistic and engaging. However,
an amount of latency is noticeable, especially during
accelerandos and ritardandos.
The idea of controlling the tempo of music with hand
claps was appealing to the test subjects. It turned
out, however, that controlling the tempo of an unfa-
miliar musical piece required some practice. The origi-
nal tempo of the music seemed to control the clapping
tempo of the user at first, and it required conscious
9 http://ydegoyon.free.fr/software.html
-6-
98
[Sonic interactions with hand clap sounds]
effort to break the cycle. After a few tryouts, the test
subject was able to control the tempo. Although the
test subjects did not report it, according to the au-
thors’ own experiments a similar phenomenon seems
to occur in the audience mode. The tempo of the clap-
ping crowd affects the tempo of the user’s clapping, and
it requires concentration to start clapping in a different
tempo than the crowd.
Latency was noticed by both test subjects in the vir-
tual audience application, but it was not reported too
distracting. With real clapping the latency is more se-
vere than with mouse clicks due to buffering for the
incoming audio and also in part due to the bonk~ ob-
ject, which is naturally computationally intensive when
compared to simply sending a bang message.
5 Conclusions
We introduced a prototype of a gestural control inter-
face for sound synthesis driven by hand clap sounds,
and simple applications to demonstrate the potential
of such an interface. Hand claps, being a universally
understood sonic gesture, make a natural control sig-
nal which is easy to learn. The most important steps
in the future are to elaborate the system with more
efficient and novel algorithms, and to come up with
innovative applications for the control interface. Al-
though the possibilities of using hand claps as control
signals may seem limited, these possibilities are well
worth exploring due to the potential of hand claps as
an easily understood means of control.
A drawback of the current system is that it incorpo-
rates an amount of latency between the excitation and
response. An important objective in the future is to
optimize the computational efficiency of the program
in order to reduce this latency to the minimum. Also,
the rhythm estimator histogram resolution issue dis-
cussed in Section 3.1 needs to be solved in a way that
doesn’t affect the resolution of the estimated BPM as
much. Alternatively, another algorithm could be de-
veloped for rhythm estimation.
During this study, we did not formally compare the
two control mechanisms, i.e., hand claps and mouse
clicks. To make a final justification for applying a hand
clap interface in human-computer interaction, such an
experiment should be undertaken. Also the distrac-
tiveness of the latency could be subject to subjective
testing.
To provide another modality to the interface and
more variety in the control signals, the trajectories of
the hands can be extracted from the actions of the
clapper. In the running of this research, we already
begun experimenting with optical hand tracking. The
concept will be studied in more detail for the next pro-
totype of the interface.
The current system makes no effort to distinguish
between hand claps and other impulsive excitation sig-
nals, such as tapping the microphone. The system
would be more robust against impulsive noises if such
a distinction could be made. Another important ad-
dition to the system is to exploit the identification of
different hand clap types. Using this discriminative in-
formation, it is possible to construct more diverse con-
trol mappings and sophisticated human-computer in-
teraction with simple and natural gestures. Although
bonk∼ would serve as a testbed for such an extension,
too, we aim at providing another classification scheme
for the purpose. An algorithm and first results of of-
fline identification of eight different hand clap types
have already been presented [5], and a real-time im-
plementation of the algorithm as well as its further
development has been indicated as future work.
6 Acknowledgements
This work is supported by the Academy of Finland
(project 120583 Schema-SID). Our thanks to Leevi Pel-
tola for the initial release of the ClaPD software, and to
Jussi Pekonen and Hannu Pulakka for their comments
and for volunteering as test subjects in the evaluation
of the system.
References
[1] J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury,
M. Davies, and M. B. Sandler. A tutorial on onset
detection in music signals. IEEE Trans. Speech
and Audio Processing, 13(5):1035–1047, 2005.
[2] C. Erkut and K. Tahiroglu. ClaPD: A testbed
for control of multiple sound sources in inter-
active and participatory contexts. In PureData
Convention 2007, Montreal, Canada, August
2007. Online proceedings at http://artengine.
ca/~catalogue-pd.
[3] F. Gouyon and S. Dixon. A Review of Automatic
Rhythm Description Systems. Comp. Music J.,
29(1):34–54, 2005.
[4] K. Hanahara, Y. Tada, and T. Muroi. Human-
robot communication by means of hand-clapping
(preliminary experiment with hand-clapping lan-
guage). In Proc. IEEE Intl. Conf. Systems,
Man and Cybernetics, pages 2995–3000, Montreal,
Canada, October 2007.
[5] A. Jylhä and C. Erkut. Inferring the hand con-
figuration from hand clapping sounds. In Proc.
11th Intl. Conf. Digital Audio Effects (DAFx-08),
pages 301–304, Espoo, Finland, September 2008.
-7-
99
[Sonic interactions with hand clap sounds]

[6] J. Laroche and M. Dolson. Improved phase

vocoder time-scale modification of audio. IEEE
Trans. Speech and Audio Processing, 7(3):323–
332, 1999.
[7] N. Lesser and DPW Ellis. Clap detection and dis-
crimination for rhythm therapy. In Proc. IEEE
Intl. Conf. Acoustics, Speech, and Signal Process-
ing, volume 3, pages 37–40, Philadelphia, PA,
USA, March 2005.
[8] L. Peltola, C. Erkut, P.R. Cook, and V. Välimäki.
Synthesis of hand clapping sounds. IEEE
Trans. Audio, Speech and Language Processing,
15(3):1021–1029, 2007.
[9] M. Puckette. Pure data: another integrated com-
puter music environment. In Proc. Second Inter-
college Computer Music Concerts, pages 37–41,
Tachikawa, Japan, 1996.
[10] M. Puckette, T. Apel, and D. Zicarelli. Real-time
audio analysis tools for Pd and MSP. In Proc.
Intl. Computer Music Conference, pages 109–112,
Ann Arbor, MI, USA, October 1998.
[11] J. Seppänen. Computational models of musical
meter recognition. Master’s thesis, Tampere Uni-
versity of Technology, 2001.
[12] H. Sharp, Y. Rogers, and J. Preece. Interaction
Design: Beyond Human Computer Interaction.
Wiley, Indianapolis, IN, USA, March 2007.

-8-

100
Toward a Salience Model for Interactive Audiovisual Applications of
Moderate Complexity
Ulrich Reiter, Q2S - NTNU, Trondheim, Norway, reiter@q2s.ntnu.no

Abstract. To provide users of interactive audiovisual application systems with subjectively high presentation
quality of the content (Quality of Experience, QoE), it is usually not effective to increase the simulation depth
of the rendering process alone. Instead, by focusing on salient parts of the content, perceived overall quality can
be increased without causing additional computational costs. This paper provides the basis for a novel salience
model for interactive audiovisual applications of moderate complexity that is based on influence factors which
have been identified in a coordinated series of experimental studies.

1 Introduction cusses briefly the main results. References to the full

The question of saliency of objects in audiovisual appli-
cations is only recently becoming an issue of examina-
tion [1]. Until now, many application systems mainly
rely on visual display and feedback to the user, with
some kind of “support” in the auditory domain. As
the computing power available in home application and
consumer electronics systems is constantly increasing,
we see a tendency toward integrating more modalities
which until now have only been available in specialized
Virtual Reality (VR) systems. With this development,
users can expect an increased degree of immersion.
This is interesting in many aspects, one of them being
that applications with higher immersion are generally
considered more user-friendly because they provide a
feeling of personalization. Additionally, these systems
better represent real life and the complexity of real-life
experiences by offering information multimodally.
The general problem with this approach is that re-
sources in consumer-oriented application systems are
always limited. It is not feasible to perform a fully
grown, detailed simulation of multimodal impressions
in real-time. Furthermore, the time and investment
necessary to develop completely accurate auditory and
visual models is as much of a limiting factor for how
much detail will be rendered, as is the computational
power alone. It is therefore reasonable to focus only on
the most important stimuli and leave out those that
would go unnoticed in a real world situation. In or-
der to do so, it is necessary to estimate what the most
important stimuli or objects in the overall audiovisual
percept are.
After giving a definition of saliency in the audiovi-
sual context in section 2, section 3 describes the role of
interactivity and presence in the perception of audiovi-
sual quality. Section 4 introduces the salience model it-
self. Section 5 summarizes the experiments performed
to verify the factors contained in the model. It also dis-
papers are given in each subsection. Finally, section 6
gives a short summary and outlook.
2 Saliency of Stimuli
In the absence of information about the history of an
interactive process, an object can be considered salient
when it attracts the user’s visual attention more than
the other objects [2]. This definition of salience origi-
nally valid for the visual domain can easily be extended
to what might be called “multimodal salience”, mean-
ing that
• certain properties of an object attract the user’s
general attention more than the other properties
of that object
• certain objects attract the user’s attention more
than other objects in that scene.
Of course, a salience model requires a user model of
perception, as well as it needs a task model. The user
model describes familiarity of the user with the ob-
jects’ properties, as attention on the properties of an
object varies with the user’s background. This corre-
lates to the concept of schemata described in Neisser’s
“Perceptual Cycle”, see fig. 1. In Neisser’s model,
schemata represent the knowledge about our environ-
ment [3]. They are based on previous experiences
and are located in the long term memory. Neisser
attributes them to generate certain expectations and
emotions that steer our attention in the further ex-
ploration of our environment. The exploratory pro-
cess consists, according to Neisser, in the transfer of
sensory information (the stimulus) into the short-term
memory. In the exploratory process, the entirety of
stimuli (the stimulus environment) is compared to the
schemata already known. Recognized stimuli are given
a meaning, whereas unrecognized stimuli will modify

101
Reiter - Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity
Stimulus
Environment
modifies samples
Perceptual
Schemata
Exploration
directs
Figure 1: Neisser’s Perceptual Cycle, after [3],
modified.
the schemata, which will then in turn direct the ex-
ploratory process.
Whereas a picture of a human being or a human
speech utterance can be considered more or less equally
salient to all users, because its significance to humans
is embedded genetically, an acoustically trained per-
son might focus more on the reverberation in a virtual
room than a visually oriented person. The task model
describes the fact that salience depends on intention-
ality, so that depending on the task the user is given,
his focus will shift accordingly.
Salience also depends on the physical characteristics
of the objects themselves. Following the Gestalt theory
introduced by Wertheimer [4], the most salient visual
form is the one requiring the minimum of sensory in-
formation to be treated. In the auditory domain it is
known that certain noises which can be characterized
with properties like ’sharpness’ or ’roughness’ call the
attention more than others [5], often by skirting mask-
ing effects in the time or frequency domain due to their
spectral or temporal characteristics. Adding to this,
salience can be due to spatial or temporal disposition
of the objects. Thus a classification of the properties
that can make an object salient in a particular context,
the so-called influence factors, have to be established
and verified in order to draw any useful conclusions
from the “multimodal salience” approach.
3 Influence of Interactivity and Presence
The concept of interactivity has been defined by Lee
et al. based on three major viewpoints: technology ori-
ented, communication-setting oriented, and individual
oriented views [6, 7]. Here, the technology-oriented
view of interactivity is adopted. The “technology-
oriented view of interactivity defines interactivity as a
characteristic of new technologies that makes an indi-
vidual’s participation in a communication setting pos-
sible and efficient” [7]. It is this individual’s quality
experience that we are interested in.
Steuer holds that interactivity is a stimulus-driven
variable which is determined by the technological struc-
ture of the medium [8]. According to Steuer, interac-
tivity is “the extent to which users can participate in
modifying the form and content of a mediated envi-
ronment in real time” - in other words, the degree to
which users can influence the target environment. He
identifies three factors that contribute to interactivity:
• speed (the rate at which input can be assimilated
into the mediated environment)
• range (the number of possibilities for action at
any given time)
• mapping (the ability of a system to map its con-
trols to changes in the mediated environment in a
natural and predictable manner)
These factors are related to technological constraints
that come into place when an application is supposed
to provide interactivity to the user. Adding to these
factors, the perceived quality of the system’s feedback
to the user - the quality of the audiovisual stimuli
generated as a reaction to the user’s input - plays an
equally important role.
Closely related to interactivity is presence. Pres-
ence in interactive audiovisual application systems or
Virtual Environments is often described as the feeling
of “being there” [9] that generates involvement of the
user. Lombard and Ditton define presence in a broader
sense as the “perceptual illusion of nonmediation” [10].
According to Steuer, the level of interactivity (de-
gree to which users can influence the target environ-
ment) has been found to be one of the key factors for
the degree of involvement of a user [8]. Steuer has
found vividness (ability to technologically display sen-
sory rich environments) to be the second fundamental
component of presence. Along the same lines, Sheridan
assumed the quality and extent of sensory information
that is fed back to the user as well as exploration and
manipulation capabilities to be crucial for the subjec-
tive feeling of presence [11].
Interactivity requires attention of the user. Only
when both - the system and the user - react to each
other, true interactivity is in place. Therefore, inter-
activity can be regarded as one way of controlling the
focus of attention of a user. This is important, as ob-
jects in the user’s focus will naturally be more salient
than others.
4 Salience Model
A salience model would thus mainly contain (and ide-
ally quantify) the influence factors that control the
-2-
102
Reiter - Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity

INFLUENCE INFLUENCE INFLUENCE QUALITY

FACTORS 1: FACTORS 2: FACTORS 3: ATTRIBUTES:
reproduction setup physiology subject (experience, synchrony (temporal,
setup-dependent (acuity,masking,…) gender, social, spatial)
content expectations,…) balance of modalities
task (interactivity, qualitative attributes
application,…) (motion, mood,
space, action)

experimental
SENSORY COGNITIVE QUALITY evaluation
STIMULUS
PERCEPTION PROCESSING IMPRESSION

REACTION

Figure 2: The suggested salience model for interactive audiovisual applications of moderate complexity.

level of saliency of each perceived object. It is easily
seen that a generalized salience model is too complex
and the influence factors too manifold to cope with at
the current state of knowledge. Therefore it is nec-
essary to get away from a generalized salience model.
Instead, it is reasonable to focus on a salience model
valid for interactive audiovisual applications of moder-
ate complexity.
Fig. 2 shows how such a salience model may be struc-
tured. The basis of human perception are the stimuli.
For interactive applications these are generated by the
application system itself, so they will depend on a num-
ber of factors: the influence factors of level 1. These
factors comprise the audiovisual reproduction setup,
e.g. (multichannel) loudspeaker setup, headphones, fre-
quency range, panning laws and -algorithms applied,
size and resolution of screen, brightness and color dis-
tribution, etc. Note that the actual weight of these fac-
tors may also depend on the audiovisual content itself:
a static, acoustically “dry” sound source in a frontal
position will not be critical to different panning laws
or number of loudspeaker channels in the back. In-
fluence factors of level 1 also comprise technical input
devices for user feedback to the system. As an ex-
ample, navigation in a 3D scene can be controlled via
computer mouse, keyboard, joystick, accelerometer or
other advanced technical devices that offer differing de-
grees of freedom (DOF) and thus differing amount and
precision of control. This in turn influences how pre-
cisely the system can react by appropriately producing
/ modifying the stimuli. To summarize, influence fac-
tors of level 1 are those related to the generation of
stimuli.
The core elements of human perception have been
identified to be sensory perception on the one hand and
cognitive processing on the other hand. Sensory per-
ception can be affected by a number of influence factors
of level 2. These involve the physiology of the user
(acuity of hearing and vision, masking effects caused
by limited resolution of the human sensors, etc.) as
well as all other factors directly related to the physical
perception of stimuli.
Cognitive processing produces a response by the
user. This response can be obvious, like an immedi-
ate reaction to a stimulus, or it can be an internal re-
sponse like re-distributing attention, shifting focus or
just entering another turn of the Perceptual Cycle (see
[3]). Obviously, the response is governed by another set
of influence factors of level 3. These span the widest
range of factors, and also the most difficult to quantify:
experience, expectations, and background of the user;
difficulty and aim of task (if any); degree of interactiv-
ity; type of application; etc. Influence factors of level 3
are related to the processing and interpretation of the
perceived stimuli.
Cognitive processing will eventually lead to a certain
quality impression (Quality of Experience, QoE) that
is a function of all influence factors of levels 1-3. This
quality impression cannot be directly quantified by hu-
mans. It needs additional processing to be uttered in
the form of (quantitative) ratings on a quality scale, as
(qualitative) semantic identifiers, and so on.
The common way of assessing the overall quality im-
pression is to evaluate single or combined quality at-
tributes. The scientific community has developed a
number of attributes that are believed to be relevant
for an overall audiovisual quality impression. Among
these are audiovisual synchrony (both temporal and
spatial), the localization of events, sound as well as
video quality by themselves (which, nevertheless, influ-

-3-

103
Reiter - Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity
ence each other), responsiveness to interaction (when
applicable), and many more.
Woszczyk et al. have tried to arrange these into a
4×4 matrix of “perceptual dimensions” (Action, Mood,
Motion, Space) vs. attributes (Quality, Magnitude, In-
volvement, Balance) within these dimensions [12]. But
again, a quantification of their impact is hardly possi-
ble as of now. This is because their weight not only
depends on the audiovisual content (the stimulus) un-
der assessment, but also on the experimental evalua-
tion (the test methodology) itself. An attribute that is
explicitly asked for will probably be assumed to be of
higher importance by the test subject (we know from
our experience that only important things are asked
for in any kind of test). The subject’s attention will be
directed toward the attribute currently under assess-
ment, an act that distorts unbiased perception of the
overall multimodal stimulus. Therefore, the subject’s
reaction might be influenced as well.
5 Experimental Evaluation
A number of subjective assessments have been per-
formed to verify the influence factors that were pre-
viously identified for a typical prototype reproduction
setup of interactive audiovisual content. This exem-
plary setup makes use of a large projecting screen, a
multichannel loudspeaker setup, and real-time room-
acoustic simulation rendered on a standard PC. Fig.
3 adumbrates the top view of the reproduction setup
with the test subject located in the center. A descrip-
tion of the audio rendering engine can be found in [13].
The results of these experiments reveal a number of
interesting points:
5.1 Multimodality vs. Unimodality
The first two assessments focused on a possible reduc-
tion of algorithmic complexity for the bimodal (audio-
visual) case compared to the unimodal (audio only)
case. The simplifications assessed were directly related
to the computational load that the real-time rendering
of audio imposes on the processor.
The first experiment [14] evaluated the number of
loudspeakers necessary in interactive audiovisual appli-
cation systems of moderate complexity using a Vector
Base Amplitude Panning (VBAP) approach to posi-
tion the sound sources. The room acoustic simulation
method applied was an image source model simulat-
ing the early reflections, combined with uncorrelated
diffuse reverberation created separately for each loud-
speaker channel. For these situations, the complexity
of the algorithm (and thus the computational load) is
significantly growing with the number of loudspeakers
involved. Thus, it is desirable to keep the number of
-15° 15°
2.72 m
-45° 45°
2.8 m
3.4 m
-105° 105°
-165° 165°
Figure 3: Overview of the test setup: large pro-
jecting screen in the front and eight channel loud-
speaker setup unevenly distributed between front
and back. The higher number of loudspeakers in
the screen area (front) increases audiovisual coher-
ence.
loudspeaker channels as low as possible without intro-
ducing deteriorations in localization quality.
It has been shown that the necessary number of
loudspeaker channels mainly depends on the content
itself. Among all tested factors, the different motion
paths through the scene that were presented in the
assessment (and therefore the directions of incidence
of the sound source) had the greatest impact on the
perceived subjective quality of the loudspeaker setup
used. As a rule of thumb and generalized result, the
well-known five-channel setup defined in ITU-R BS.775
[15] is suitable for such systems.
The second experiment [16] evaluated the number
of internal workchannels for the MPEG-4 Audio Per-
ceptual Approach reverberation algorithm [17]. The
MPEG-4 Audio Perceptual Approach is also known
as the SPAT reverberation algorithm for Cycling’74’s
Max/MSP. As opposed to the image source method, it
generates reverberation generically, i.e. without a phys-
ical/geometrical reference of the room to be simulated.
The amount of so-called internal workchannels involved
determines the complexity/density of the reverberation
pattern. Increasing the number of workchannels there-
fore generates a higher density reverberation pattern.
The density of the reverberation pattern is usually at-
tributed to be the foremost quality criteria of artificial
reverberation.
-4-
104
Reiter - Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity
The assessment compared three versions of the Per-
ceptual Approach algorithm to each other. The test
results make clear that - for the audiovisual case - sub-
jects were not able to identify the three versions of the
algorithm under assessment. Increasing the density of
the diffuse reverberation part remains without perceiv-
able improvement of the quality in bimodal (audiovi-
sual) perceptual situations. Therefore the Perceptual
Approach algorithm as specified in MPEG-4 Scene De-
scription can be simplified to use only four internal
workchannels without degrading the overall perceived
quality in the audiovisual context.
5.2 User Interaction
The next three assessments [18, 21, 22] focused on the
effect that user interaction with the audiovisual ap-
plication might have on the perceived overall quality.
Here the general assumption was that by offering an at-
tractive interactive content or by assigning the user a
challenging task, the user would become more involved
and thus experience a subjectively higher overall qual-
ity.
The first experiment in this series compared the per-
ceived overall quality of audiovisual scenes under dif-
ferent degrees of interaction [18]. The actual amount
of interaction was determined by three different tasks
that the test subjects had to fulfill during the assess-
ment. These were:
1. Listen and watch task: Test subjects were pre-
sented with an automated movement through the
virtual scene. No activity on their side was re-
quired. The automated movement lasted around
30s, selected from two different predefined motion
paths.
2. Listen and press a button task: Again, test sub-
jects were asked to experience an automated
movement through the virtual scene. This time,
an object automatically appeared within the field
of view. It was subsequently approached and
(again automatically) collected. Then, a new ob-
ject would appear, and so on. Test subjects were
asked to immediately press a button whenever the
object appeared.
3. Listen and collect an object task: Test subjects
were using the computer mouse to navigate freely
inside the virtual scene. Their task was to col-
lect the object that was positioned somewhere on
the floor. When they had approached the object
closely enough, it was collected and re-appeared
in another location. The new location was either
within the field of view, or the subjects had to
turn around to see it again. They were asked to
collect as many objects as possible within a given
time limit of 30s.
Interestingly, and contrasting with results obtained
by others (e.g. Zielinski, Kassier, Rumsey, Bech et al.
in [19] and [20], both based on smaller sample sizes),
the different tasks that test subjects had to perform did
not have an effect on the quality evaluation (Friedman:
χ2 = 3.3, df = 2, p > 0.05, p = 0.190, ns).
Two possible explanations exist:
• The subjects’ task of navigating through the scene
was not demanding enough, making the differ-
ences in quality too obvious.
• Whereas user interaction was related to visual and
haptic modalities, the quality rating was based on
audiovisual percepts. The distraction generated
by interaction was not high enough to be signifi-
cant across modalities.
These possible explanations were examined in the
next two experiments [21, 22]. On the one hand, user
interaction, rating process and tasks aimed at sharing
the same modality. On the other hand, test subjects
were confronted with a mentally complex, yet easily
scalable task: the n-back working memory paradigm.
In this, the subject typically is required to monitor a
series of stimuli and to indicate whether or not the
stimulus currently presented is the same as the one
presented n steps before.
Here, a sequence of spoken numbers was presented
and subjects had to compare the numbers. At the same
time, the reverberation time was varied and subjects
were subsequently asked to correctly rate the length
of reverberation in comparison to a reference rever-
beration time1 . Unlike in previously published exper-
iments, both the attribute to be rated and the dis-
tracting task were located in the same modality. An
analysis of the collected data indicates that the pre-
cision with which auditory parameters can be rated /
discriminated by humans is dependent on the degree of
distraction present in the same modality. A highly sig-
nificant difference in rating accuracy was shown for the
“navigation only” condition vs. the “navigation with 2-
back task” condition using Wilcoxon’s T test (matched
pairs signed ranks, T = 20, p ≤ 0.01).
This result further confirms and specifies the find-
ings of [18, 19, 20]: Whereas cross-modal division of
attention only renders a small significant effect and
- apart from being listener-specific - depends on the
experimental conditions, with inner-modal distraction
test subjects would predictably commit errors in their
1 In [18], an additional semi-structured interview had re-
vealed that reverberation time was regarded as one of the
most important attributes for the given type of interactive
audiovisual content by all test subjects.
-5-
105
Reiter - Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity
ratings. Apparently, inner-modal influence is signifi-
cantly greater than cross-modal influence. This is also
supported by some of the theories of capacity limits in
human attention [23].
5.3 Cross-Modal Interaction
Finally, the last assessment [24] in this series investi-
gated the possibility of cross-modal influence of inter-
action upon perceived quality. Whereas in the previous
two assessments the influence of interaction within the
same modality was investigated, here the influence of
a visual (-motion) task upon the perceived audio qual-
ity was evaluated. This experiment is borrowing from
what Zielinski et al. [19] and Kassier et al. [20] have de-
scribed, but the test panel was significantly larger (31
test subjects opposed to 6 and 7, respectively), thus
allowing a profound statistical analysis.
For this experiment, a computer game was designed
to assess the effect of divided attention in the evalu-
ation of audio quality during involvement in a visual
task. Subjects had to collect selected flying objects
(donuts) by running into them and avoid the collision
with other objects (snowballs). For the navigation, test
subjects used the left and right arrow keys of a com-
puter keyboard. Movement was only possible to the
sides, at a fixed distance from the source of the flying
objects.
A game score was recorded for each subject to verify
subjects’ involvement in the game and to prod the sub-
jects to actively play the game. By collecting the right
object (donut) the score was increased by one point,
whereas a collision with a snowball decreased the score
by one point.
For the experiment, each subject carried out a pas-
sive and an active session. The active session consisted
in playing the computer game and evaluating the audio
quality. This session was designed to cause a division
of attention between the process of rating the audio
quality and the involvement in the computer game. In
the passive session, subjects were asked to evaluate the
audio quality while only watching a game demo. The
audio quality degradations were realized by modifying
the tonal quality. The original music signal (16kHz)
was low-pass filtered using three different cut-off fre-
quencies fc = 11kHz, 12kHz and 13kHz. Addition-
ally, an anchor with a low-pass filtering at the cut-off
frequency fc = 4kHz was created.
The Wilcoxon T test showed that the quality ratings
of the active session varied significantly from the rat-
ings of the passive session for cut-off frequencies up
to 12kHz. A significant decrease in rating correct-
ness was shown for the Game condition in compar-
ison to the No Game condition for the anchor item
(T = 37, p ≤ 0.01), the cut-off frequency fc = 11kHz
(T = 452.50, p ≤ 0.01), and the cut-off frequency
fc = 12kHz (T = 812, p ≤ 0.01). The low-pass filter-
ing in the active session (Game condition) was rated
as being generally less perceptible.
This assessment showed that a cross-modal influence
of interaction is possible when stimuli and interaction
are carefully balanced. Interaction performed in one
modality (e.g. visual-haptic) can dominate the percep-
tion of stimuli in another modality (here: auditive).
Yet, at this time it is not possible to determine or quan-
tify that balance a priori.
5.4 Conclusions
The experiments have clearly identified a number of
factors that influence the perceived quality of audiovi-
sual content. These are of technical nature, i.e. depend-
ing on the reproduction setup and simulation algorithm
used, but also of contextual and subjective nature, i.e.
depending on user task, on degree and modality of in-
teraction offered, and on individual attention capacity
limits.
6 Summary and Outlook
The model introduced here identifies and classifies the
most important influence factors that determine the
saliency of objects in a multimodal perceptual situa-
tion. It has been specifically developed to describe the
perception of audiovisual content in interactive appli-
cation systems of moderate complexity, yet it can be
extended to include true multimodality. It is based on
the experimental evaluation of perceived overall quality
(Quality of Experience) tested in a coordinated series
of subjective assessments.
The model needs further refinement to be put to
use in real-world applications. One of the tasks that
remain is the context-dependent quantification of the
influence factors: in its current state of development,
the model is a purely qualitative one that does not yet
allow a priori statements (quantified estimations) on
the weight of individual factors.
References
[1] Reiter, Ulrich. On the Need for a Salience Model
for Bimodal Perception in Interactive Applica-
tions. IEEE/ISCE’03, International Symposium
on Consumer Electronics. Sydney, Australia, De-
cember 3-5, 2003.
[2] Landragin, Frederic; Bellalem, Nadia; Romary,
Laurent. Visual Salience and Perceptual Grouping
in Multimodal Interactivity. Proc. International
Workshop on Information Presentation and Nat-
ural Multimodal Dialogue IPNMD. Verona, Italy,
December 14-15, 2001.
-6-
106
Reiter - Toward a Salience Model for Interactive Audiovisual Applications of Moderate Complexity
[3] Farris, J. Shawn. The Human Interaction Cycle:
A Proposed and Tested Framework of Perception,
Cognition, and Action on the Web. PhD Thesis.
Kansas State University, USA, 2003.
[4] Wertheimer, Max. Untersuchungen zur Lehre von
der Gestalt II. Psychologische Forschung. 4, 1923,
pp 301-350.
[5] Zwicker, Eberhard; Fastl, Hugo. Psychoacoustics -
Facts and Models. 2nd updt. ed., Springer Verlag.
Berlin, 1999, ISBN 3-540-65063-6.
[6] Lee, Kwan Min; Jin, S. A.; Park, N.; Kang, S. Ef-
fects of narrative on feelings of presence in com-
puter/video games, Annual Conference of the In-
ternat. Communication Association (ICA), New
York, NY, USA, May 2005.
[7] Lee, Kwan Min; Jeong, Eui Jun; Park, Namkee;
Ryu, Seoungho. Effects of Networked Interactiv-
ity in Educational Games: Mediating Effects of
Social Presence, PRESENCE2007, 10th Annual
International Workshop on Presence, Barcelona,
Spain, Oct. 25-27, 2007, pp 179-186.
[8] Steuer, Jonathan. Defining Virtual Reality: Di-
mensions Determining Telepresence. Journal of
Communication. 42/4, 1992, pp 73-93.
[9] Larsson, Pontus; Västfjäll, Daniel; Kleiner,
Mendel. On the Quality of Experience: A Multi-
Modal Approach to Perceptual Ego-Motion and
Sensed Presence in Virtual Environments. Pro-
ceedings First ISCA ITRW on Auditory Quality of
Systems AQS-2003. Akademie Mont-Cenis, Ger-
many, April 23-25, 2003, pp 97-100.
[10] Lombard, Matthew; Ditton, Theresa. At the Heart
of it All: The Concept of Presence. Journal of
Computer-Mediated Communication, 3, 1997.
[11] Sheridan, Thomas B. Further Musings on the Psy-
chophysics of Presence. Presence, 5/1994, pp 241-
246.
[12] Woszczyk, Wieslaw; Bech, Soren; Hansen, Villy.
Interactions Between Audio-Visual Factors in a
Home Theater System: Definition of Subjective
Attributes. AES 99th Convention, New York,
USA, 1995, Preprint 4133.
[13] Reiter, Ulrich. TANGA - an Interactive Object-
Based Real Time Audio Engine. Audio Mostly
2007, 2nd Conference on Interaction with Sound,
Ilmenau, Germany, September 27-28, 2007.
[14] Reiter, Ulrich. Subjective Assessment of the Opti-
mum Number of Loudspeaker Channels in Audio-
Visual Applications Using Large Screens. Proc.
AES 28th Internat. Conf., Pitea, Sweden, June
30 - July 2, 2006, pp 102-109.
[15] Recommendation ITU-R BS.775-1. Multichannel
stereophonic sound system with and without ac-
companying picture. International Telecommuni-
cation Union, Geneva, Switzerland, 1994.
[16] Reiter, Ulrich; Partzsch, Andreas; Weitzel,
Mandy. Modifications of the MPEG-4 AAB-
IFS Perceptual Approach: Assessed for the Use
with Interactive Audio-Visual Application Sys-
tems. Proc. AES 28th Internat. Conf., Pitea, Swe-
den, June 30 - July 2, 2006, pp 110-117.
[17] Int. Std. (IS) ISO/IEC 14496-11:2004. Informa-
tion technology - Coding of audio-visual objects -
Part 11: Scene description and Application en-
gine. Geneva, Switzerland, 2004.
[18] Reiter, Ulrich; Jumisko-Pyykkö, Satu. Watch,
Press and Catch - Impact of Divided Attention
on Requirements of Audiovisual Quality. 12th In-
ternat. Conf. on Human-Computer Interaction,
HCI2007, Beijing, PR China, July 22-27, 2007.
[19] Zielinski, Slawomir; Rumsey, Francis; Bech,
Soren; de Bruyn, Bart; Kassier, Rafael. Computer
Games and Multichannel Audio Quality - the Ef-
fect of Division of Attention Between Auditory
and Visual Modalities. Proc. AES 24th Interna-
tional Conference on Multichannel Audio, Banff,
Alberta, Canada, June 2003.
[20] Kassier, Rafael; Zielinski, Slawomir; Rumsey,
Francis. Computer Games and Multichannel Au-
dio Quality Part 2 - Evaluation of Time-Variant
Audio Degradation under Divided and Undivided
Attention. AES 115th Convention, New York,
USA, October 2003, Preprint 5856.
[21] Reiter, Ulrich; Weitzel, Mandy; Cao, Shi. Influ-
ence of Interaction on Perceived Quality in Au-
dio Visual Applications: Subjective Assessment
with n-Back Working Memory Task, Proc. AES
30th International Conference, Saariselkä, Fin-
land, March 15-17, 2007.
[22] Reiter, Ulrich; Weitzel, Mandy. Influence of In-
teraction on Perceived Quality in Audio Visual
Applications: Subjective Assessment with n-Back
Working Memory Task, II. AES 122nd Conven-
tion, Vienna, Austria, May 5-8, 2007.
[23] Pashler, Harold. The Psychology of Attention. 1st
paperback edition, The MIT Press, Cambridge,
MA, USA, 1999, ISBN 0-262-66156-X.
[24] Reiter, Ulrich; Weitzel, Mandy. Influence of In-
teraction on Perceived Quality in Audiovisual Ap-
plications: Evaluation of Cross-Modal Influence.
Proc. 13th International Conference on Auditory
Displays (ICAD), Montreal, Canada, June 26-29,
2007.
-7-
107
 An Embedded Audio-Based Vehicle Classification
Based on Two-level F-ratio

Jiqing Han, Tao Jiang

School of Computer Science and Technology, Harbin Institute of Technology, Harbin, P.R.China, 150001
jqhan@hit.edu.cn

Abstract. The human auditory system can be regarded as a complex signal and information processing machine. With the rapid
progress of information technology, it is reasonable to use computer to simulate the ability of human auditory system. One of this
kind of researches is audio sense technology, which uses computer to analyze the environment sounds and identify the sounder by
analysis of audio signals. It can extend the audio sense ability of people. One application of audio sense technology is vehicle
classification, which means using the audio features of different vehicles to design a classifier for identifying the vehicles. For
embedded applications, generally a linear weighted classifier is used for the vehicles classification, and the F-ratio, which denotes the
contributions of different audio feature dimensions, is adopted as the weights. In this paper, we find just using F-ratio as the weight
there are confusion errors of classification, thus an approach of using two-level F-ratio as the weights is proposed to overcome the
problem. We first use F-ratio of total patterns as the weight to select the candidate set of confusion patterns, and then in the candidate
set, we use the F-ratio of the confusion patterns as the weight to give the final classification results, and a real time accuracy of
82.1% is gotten in an embedded platform based on a MSP430F149 microcontroller.

1. Introduction
The human auditory system can be regarded as a complex signal
and information processing machine. With the rapid progress of
information technology, it is reasonable to use computer to
simulate the ability of human auditory system. One of this kind
of researches is audio sense technology, which is a main branch
of human computer interface, and it is important for the
automatism and intelligence of computer. This technology can
use computer to analyze the environment sounds automatically
and identify the sounder by analysis of audio signals, which can
make computer has the auditory intelligence. Audio sense
technology will be widely used, because it can extend the audio
sense ability of people by using computer, which will be a
primary assistant to human being in information acquisition
domain.
The motivation of this paper is to discuss the approaches of
using audio sense technology to identify different kinds of
vehicles. It is an important signal processing task that has been
found widespread applications such as intelligent transportation
systems, sensor network application [eg. 1-3]. Many researches
had been explored for this kind of works. In [1], the
classification of moving vehicles in a distributed, wireless
sensor network was investigated, where a local pattern classifier
at each sensor node first made a local decision on what type of
the vehicle was based on its own feature vector. The probability
of correct classification can also be estimated. The local
decision, together with the estimated probability of being a
correct decision then can be encoded and transmitted efficiently
via the wireless channel to a local fusion center ready for
decision fusion. It was found that data fusion and decision
fusion enhanced the performance of vehicles classification. In
[2], an adaptive threshold algorithm were proposed for real-time
vehicle detection applications. It is a time-domain energy-
distribution-based alogrithm. Which first computes the time-
domain energy distribution curve and then slices the energy
distribution curve using a threshold updated adaptively by some
decision states. Finally, the decision results from threshold
*
This work is supported by National Natural Science Foundation of China
under Grant No. 60672163.
slicing are passed to a finite state machine, which makes the
vehicle detection decision. In [3], a traffic monitoring detector
for the vehicle classification was developed to aid traffic
management systems, where a Time Delay Neural Network
(TDNN) was chosen to classify individual traveling vehicle
based on their speed-independent acoustic signature, and the
Linear predictive coding (LPC) preprocessing and feature
extraction technique were applied to the work. In [4], a fusion
framework using both video and acoustic sensors for vehicle
detection and tracking was proposed. In the detection phase, a
rough estimate of target direction-of-arrival was first obtained
using acoustic data through beam-forming techniques. This
initial estimate designates approximate target location in video.
Given the initial target position, the method was refined by
moving target detection using the video data. And Markov
Chain Monte Carlo techniques were then used for joint audio-
visual tracking.
Considering there are many embedded applications which are
based on a processor of limited ability of computation, the
approaches of simple computation and small storage are needed
to develop. Generally, the simple feature extraction and liner
classification algorithm are adopted for reducing the
computation complexity. In order to represent the contributions
of different audio feature dimensions, a method of audio feature
weighted Euclidean distance is used for pattern matching in the
linear classifier, and the F-ratios of different audio feature
dimensions are selected as the weights. In this paper, we find
that using F-ratios of total patterns as the weights, there are also
errors of classification. We analyze the confusion patterns of
error classification and find their features with larger values of
F-ratios are near similar, and the features with smaller ones are
very different. Consequently, in order to improve the
performance, a two-level F-ratio weighted algorithm is proposed.
In which we first use F-ratio of total patterns as the weight to
select the candidate set of confusion patterns, and then in the
candidate set, we use the F-ratio of confusion patterns as the
weight to give the final classification results. In the paper, audio
signal gathering technology, audio signal feature extraction
technology and embedded system technology are also carefully
analyzed. The schematic diagram of our vehicle classification
system is shown in Figure 1. Furthermore, an embedded real-

108
time audio sense processing system for identifying different Generally, the feature analysis of acoustic signal is based in
kinds of vehicles based on a Texas Instruments MSP430F149 three domains: time, frequency, and time-frequency domains.
microcontroller is implemented. Acoustic signal processing in time domain is a natural approach,
but not an optimal one due to the complexity of the
environment, i.e., the time domain signatures of acoustic signals
Preprocessing Feature Two-level F-ratio can be hampered by noise from other moving vehicles. The
extraction method
features of the frequency and time-frequency domains are useful
to get a good performance of classification, but they are not
suitable for real-time vehicle classification since they tend to
Sound of vehicle Vehicle classification require intensive computation and samples from a long period of
time. Considering the requirement of real-time processing and
Figure 1. Schematic diagram for acoustic signature the limited computation ability of MSP430F149, the time
processing for vehicle classification domain features are adopted in this paper, and six features in
time domain are used, which include four general ones, such as
The paper is organized as follows: in section 2, the pre- short time energy(STE), short time zero-cross-rate(ZCR), short
processing and feature extraction of vehicles audio signal are time band-pass zero-cross-rate(BZCR) and the number of short
first introduced, which includes preemphasis, windowing and time peaks. And also two new time features are used, i.e., the
feature computation, and four traditional features and also two average value of short time peaks and the variance of short time
new features are gotten, next the proposed two level F-ratio peaks. The above features are defined as follows,
based method is discussed. And then in section 3, the Let ~yl (n) (n=0,1, …, N-1) denote the acoustic signal samples in
experiments and discussions are given, which includes the
a frame, the short time energy is defined as,
corpus of our experiments, the audio signal preprocessing and N −1
feature analysis, the experimental results of vehicles STE (m) = ∑ ~
yl ( n) (4)
n =0
classification based on audio signal and also some discussions.
Finally, section 4 gives the summary of our work and the ZCR is another basic acoustic features that can be computed
conclusions. easily. It is equal to the number of zero-crossing of the
waveform within a given frame,
2. Two-Level F-ratio Method ZCR (l ) = {
1 N −1
∑ | sgn[ ~
2 n =0
yl (n)] − sgn[ ~
yl (n − 1)] |} (5)
2.1. Pre-processing Where sgn(x) is the sign function,
The pre-processing is a most important part of acoustic signal ⎧1 if x > 0
processing, which converts the sound waveform to some type of ⎪
parametric representation (generally at a considerably lower in sgn( x) = ⎨0 if x = 0 (6)
formation rate) for further analysis and processing. It includes ⎪− 1 if x < 0
the following steps [5],
⎩
(1) Preemphasis, in which the digitized acoustics signal is put ZCR has the following characteristics: (1) In general, ZCR of
through a low-order digital system to spectrally flatten the both unvoiced sounds and environment noise are larger than
signal and to make it less susceptible to finite precision voiced sounds. (2) It is hard to distinguish unvoiced sounds from
effects later in the signal processing. The digital system environment noise by using ZCR alone since they have similar
used in the preemphasizer is either fixed or slowly adaptive. ZCR values. (3) ZCR is often used in conjunction with the
The most widely used preemphasis network is the fixed volume for end-point detection. In particular, ZCR is used for
first-order system, detecting the start and end positings of unvoiced sounds.
However, the ZCR parameter is easily affected by lower
H ( z ) = 1 − a~z −1 0.9 ≤ a~ ≤ 1.0 (1) frequency noise, thus the short time band-pass ZCR is used as
(2) Frame Blocking, the acoustic signal is a slowly time following,
varying signal in the sense that, when examined over a 1 N −1
sufficiently short period of time (between 5 and 100 msec), BZCR(l ) = {∑ | sgn[ y%l ( n) − T ] − sgn[ y%l ( n − 1) − T ] | +
its characteristics are fairly stationary, so it is reasonable to 2 n =0 (7)
cut the continuous acoustic signal into short time parts, and + | sgn[ y%l (n) + T ] − sgn[ y%l (n − 1) + T ] |}
each part is called a frame. In this step the preemphasized Where T is a threshold.
acoustic signal is blocked into frames of N samples, with
adjacent frames being separated by R samples. Let pl (n) represent the related peaks in the frame, which is
(3) Windowing, the next step in the processing is to window defined as,
each individual frame so as to minimize the signal ⎧1, ~
y l ( n) > ~
yl (n − 1) & & ~
y l ( n) > ~
yl (n + 1)
discontinuities at the beginning and end of each frame. If p l ( n) = ⎨ (8)
⎩0 , others
we define the window as w(n),0 ≤ n ≤ N − 1 , then the
Let N l denote the number of peaks (PN) in the frame,
result of windowing is the signal,
~ ∞
yl (n) = yl (n) w(n), 0 ≤ n ≤ N − 1 (2) N l = ∑ pl ( n ) (9)
n = −∞
Where yl (n) and ~
yl (n) are the lth frames of signal before The average value of short time peaks (AP) is defined as,
windowing and after windowing respectively, and a typical ∞

window is the Hamming window, which has the form, ∑ | ~yl (n) pl (n) |
2πn Pl = n = −∞
(10)
w( n) = 0.54 − 0.46 cos( ), 0 ≤ n ≤ N − 1 (3) Nl
N −1 The variance of short time peaks (VP) is defined as,

2.2. Feature Extraction

109
 ∞
∑ [ y% (n) p (n) − M ]
n =−∞
l l l
2
vl = (11)
Nl
Where M l is the average value of amplitudes of short time
signal in a frame.
The above six parameters have been constructed as a feature
vector for the frame, and the total forty frames are used as an
analysis window, then the features of a window are used to
represent the information of a kind of vehicle.
2.3. Two Level F-ratio Based Method
For the vehicle classification, it is a pattern recognition approach,
and has four steps [5],
(1) Feature measurement, in which a sequence of
measurements is made on the input signal to define the
“test pattern”.
(2) Pattern training, in which one or more test patterns
corresponding to sounds of the same class are used to
create a pattern representation of the features of that class.
The resulting pattern, generally called a reference pattern,
can be a template, derived from some type of averaging
technique, or it can be a model that characterizes the
statistics of the features of the reference pattern. Because of
the non-stationary nature of acoustic signal, the short-time
feature extraction as section 2.2 are performed sequentially
over time, producing a sequence of feature vectors as
pattern.
(3) Pattern classification, in which the unknown test pattern is
compared with each class reference pattern and a measure
of similarity (distance) between the test pattern and each
reference pattern is computed.
(4) Decision logic, in which the reference pattern similarity
scores are used to decide which reference pattern (or
possibly which sequence of reference patterns) best
matches the unknown test pattern.
The factors that distinguish different pattern recognition
approaches are the types of feature measurement, the choice of
templates or models for reference patterns, and the method used
to create reference patterns and classify unknown test patterns.
In the paper, the linear classification algorithm is designed and
an approach of feature weighted Euclidean distance metric is
adopted. In which the relevant weight is the F-ratio of different
feature dimensions.
F-ratio is a statistic which is often used in speaker recognition
[6], it is proportional to the ratio of the variance of the means of
each speaker’s feature distribution to the average value of the
variance of each distribution. The farther apart the individual
distributions are with respect to their average spread, the higher
the F-ratio. Thus the F-ratio is an indication of a feature
effectiveness. In vehicles classification, the feature parameters
are evaluated in terms of their ability to discriminate vehicles
and their dependence on other parameters. For the former
purpose the F-ratio of the analysis of variance is used. For a
given parameter, the values obtained from the repetitions by
each vehicle may be regarded as samples from a probability
distribution associated with that vehicle. For vehicles
classification, a good parameter is one for which these
individual vehicle distributions are as narrow and as widely
separated as possible.
The F-ratio is given by [6],
n m 1 n m
F= ∑ (μ j − μ ) / ∑ ∑ ( xij − μ j ) (12)
2 2
m − 1 j =1 m(n − 1) i =1 j =1
Where xij is the parameter value on the ith repetition by the jth
vehicle, i = 1, ... , n , j = 1,..., m ;
μ j is the estimated mean for the jth vehicle,
1 n
μj = ∑ xij (13)
n i=1
And μ is the estimated over-all mean,
1 m
μ= ∑μj (14)
m j =1
Using the F-ratio weight we got a basic performance for five
kinds of vehicles classification. However, there are also errors of
classification. We analyze the confusion patterns of error
classification and find their features with larger values of F-
ratios are near similar, and the features with smaller ones are
very different. Consequently, in order to improve the
performance, a two-level F-ratio weighted algorithm is proposed.
In the algorithm, we first use F-ratio of total patterns as the
weight to select the candidate set of confusion patterns, and then
in the candidate set, we use the F-ratio of confusion patterns as
the weight to give the final classification results. The algorithm
is as follows,
Step 1. calculate the F-ratio of over-all patterns (Fall) and
the F-ratio of the confusion patterns (Fconfu) ;
Step 2. for input acoustic signal of the vehicle, get the
feature vectors of the analysis window;
Step 3. use Fall as the weight, calculate the weighted
Euclidean distances between the input feature vectors and
different patterns;
Step 4. select the two best patterns as the candidates;
Step 5. use the Fconfu of candidate patterns as the new
weight, recalculate the weighted Euclidean distances
between the input feature vectors and candidate patterns;
Step 6. get the best pattern as the final result.
3. Experiments
The experiment audio data of the corpus were selected from
Internet [7]. It includes the acoustic signal of five kinds of
vehicles, i.e., airplane, helicopter, truck, tank, jeep. The
MSP430F149 is used to construct the embedded real-time
processing platform, which is a Texas Instruments 16-bit Ultra-
low-power microcontroller, it has a powerful 16-bit RISC CPU,
a fast 12 bit ADC, and hardware multiplier. It is useful for the
typical applications include sensor systems that capture analog
signals, convert them to digital values, and process and transmit
the data to a host system.
In the experiments, the audio signal is first parameterized using
a feature analysis. Each analysis frame is processed with a
Hamming window, and the frame length is 25 milliseconds and
frame overlapping length is 12.5 milliseconds, the preemphasis
factor a~ is selected as 0.97. Each testing file is cut into pieces,
and the length of each piece is about 2 seconds. There is only
one kind of vehicle in a piece and the system will give one result
for a piece. The six features are calculated for each frame, they
are STE, ZCR, ZBCR, VP, AP, PN. A typical F-ratio of six
features for a vehicle is shown in Figure 2. It is noted that the F-
ratios of two new features, VP and AP, have larger values, thus
it is reasonable to think that they have much contribution for the
classification results.
Using the F-ratio as the weight of Euclidean distance between
the test vehicle and the different patterns, the classification
accuracy of 77.3% is gotten. This result is not very good, there
are confusion classification errors.
110
An Embedded Audio-Based Vehicle Classification Based on Two-level F-ratio

compare performances of different lengths, and the results of

30 using different lengths of the analysis window are shown in
25 Figure 4.
20
From Figure 4, it is noted when 20 frames is used as the length
of an analysis window, the accuracy is lower, it is just more than
15
F-ratio 65%. When the analysis window is increased to 40 frames, the
10 accuracy increases to 82.1%. After that the accuracy increase
5 slowly, but the processing time increase, which is not suitable
0 for the real time processing. Considering the performance and
ZBCR VP AP STE ZCR PN
processing time, thus 40 frames is selected as the length of an
analysis window in our system.
Figure 2. A typical F-ratios of six time domain features for a vehicle
4. Conclusion
We analyzed the confusion patterns, and Figure 3 shows the
normalized feature parameters of two confusion patterns. It is It is very useful to use audio sense technology to classify the
noted when the F-ratios of the two patterns are larger, there are different kind of vehicles. In this paper, we have presented a
little differences between the two patterns for the BZCR, AP and two-level F-ratio based method for vehicle classification, which
VP; and when the F-ratios of the two patterns are little, there are first uses F-ratio of total patterns as the weight to select the
larger differences between the two patterns for the STE, ZCR candidate set of confusion patterns, then in the candidate set, the
and PN. Using the two-level F-ratio can just overcome the F-ratio of confusion patterns are used as the weight to give the
problem, since the F-ratio represents the over-all character of final classification results. The performance is evaluated on an
patterns, and two-level F-ratio represents individual character embedded platform based on MSP430F149. It can be seen that
between confusion patterns. Then we did the experiments of using two-level F-ratio based method can get a better result than
using two-level F-ratio weighted method. that of using F-ratio based method, and near a 5% performance
improvement can be gotten.
95
90 Although the proposed method has a better performance, there
85 are lots of works should be researched in depth. We should
80 pattern 1
75 enlarge the corpus to evaluate the performance of the proposed
pattern 2
70 method for a large test data, and also explore the approaches of
65
60 classification for the continuous acoustic signal of different kind
ZBCR VP AP STE ZCR PN
of vehicles. All the above researches are our future works.
Figure 3. Normalized features of two confusion patterns

The average performances of using F-ratio based method and

two-level F-ratio based method are given in Table 1. References
[1] Marco F. Duarte and Yu Hen Hu, Vehicle Classification in
Table 1. The performances of using F-ratio and two level F-ratio Distributed Sensor Networks, Journal of parallel and distributed
based method computing, 64: 826-838,(2004)
F-ratio based Two level F-ratio Based [2] Jiagen (Jason) Ding, Sing-Yiu Cheung, Chin-Woo Tan and
accuracy 77.3% 82.1% Pravin Varaiya, Signal Processing of Sensor Node Data for
Vehicle Detection, ITSC2004,(2004)
It is note that the two-level F-ratio based method improve the [3] A. Y. Nooralahiyan, H. R. Kirby, Vehicle Classification by
accuracy about 5% comparing with the F-ratio based method, Acoustic Signature, Mathl. Comput. Modelling Vol. 27, 205-
and a precision rate of 82.1% is gotten. It is shown that the two- 214,(1998)
level F-ratio based method is useful to improve the classification [4] Rama Chellappa, Gang Qian and Qinfen Zheng, Vehicle
performance of vehicles. Detection and Tracking Using Acoustic and Video Sensors,
ICASSP2004, III793-796,(2004)
90 [5] Lawreence Rabiner, Biing-Hwang Juang, Fundamentals of
Speech Recognition, Prentice Hall, (1993)
85 [6] Jared J. Wolf, Efficient Acoustic Parameters for Speaker
Recognition, Journal of Acoustics Society of American,
80
51:2044-2056,(1972)
75 accuracy [7] http://spib.rice.edu/spib/data/signals/noise
(%) [8] Jean-Luc Rouas, Jérôme Louradour, Sébastien Ambellouis,
70 Audio Events Detection in Public Transport Vehicle,
ITSC06,(2006)
65 [9] Daniel P. W. Ellis, Prediction-Driven Computational
Auditory Scene Analysis, Ph.d thesis, Massachusetts Institute of
es

es

es

s
me
am

am

am

Technology, (1996)
ra
fr

fr

fr

0f

[10] Darryl Godsmark, Guy J. Brown, A Blackboard

20

40

80

12

Architecture for Computational Auditory Scene Analysis,

Figure 4. Classification performances Speech Communication, 27:351-366, (1999)
with different lengthes of an analysis window

In the proposed method, there is a changeable factor, the length

of analysis window, in order to evaluate whether it affects the
classification performance or not, we did experiments to

111
 dots: an Audio Entertainment Installation using Visual and Spatial-
based Interaction

Nikolaos Grigoriou, Nikolaos Moustakas, Andreas Floros and Nikolaos Kanellopoulos

Dept. of Audiovisual Arts, Ionian University, 49 100 Corfu, Greece

{av200432, av200406, floros, kane}@ionio.gr

Abstract. “dots” is an interactive sound installation that takes into account the spatial position of an arbitrary number of participants
in order to algorithmically synthesize an audio stream in real-time. The installation core is a software application developed during
this work, which employs advanced video and audio processing techniques in order to detect the exact participants’ positions and to
weight-mix short audio granules. Audio mixing is performed using a virtual spatial gridding of the installation space in two
dimensions. The synthesized audio stream reproduction is combined with a number of appropriately designed visual effects, which
aim to enhance the participants’ comprehension and render the “dots” installation a high-quality interactive audiovisual platform.

1. Introduction The “dots” interactive platform presentation is here mainly

The continuous growth and evolution of digital audiovisual
technologies is providing the scientific framework to design and
develop/demonstrate new ways of artistic expression [1]. The
recently established high-definition video and multichannel
audio formats and standards are becoming widely accepted by
both the media-producers’ and the end-users’ market.
Additionally, new terms and ideas originating from the general
concept of interaction are nowadays frequently used to provide
novel means of audio and visual production, where the audience
is actively participating in the production process [2], [3]. As a
consequence, a continuously growing number of new media
artists have now the option to express their thoughts and feelings
by developing and demonstrating their installations, using high
quality audio/visual synthesis and playback techniques and
equipment [4].
The above interactive installations can be also considered as
advanced tools for creating and synthesizing in real-time high-
quality audio/visual content. For example, in a typical case, the
visual information obtained or processed from an interactive
installation can provide feedback to a sound synthesis system for
producing novel sound streams in real-time. The opposite
approach can be alternatively employed for synthesizing
complex visual content and environments using sound recorded
signals as input. In any case, the user interaction is usually
realized using an interaction algorithm that defines the rules
under which selected synthesis parameters are varied. The
variation amount is usually obtained from the installation
environment, typically using a wide range of wired or wireless
sensors, image capturing devices and video and/or audio
recording equipment [5].
In this article we present the “dots” interactive platform, which
aims to produce complex audio streams taking into account the
movement and the instantaneous spatial positions of the
participants. In order to provide a motion motivation to the
participants, an additional visual component is produced and
concurrently projected in real-time, which also depends on the
participants positions and motion. Hence, a complex spatial,
audio and visual interaction effect is achieved which apart from
dynamically creating audio content can be also considered as an
interactive game with colours and sounds. The visual content
consists of a controlled number of dots with varying colours,
providing the name “dots” to the overall installation platform.
performed in technological terms, typically focusing on the
platform architecture and some elementary design / algorithmic
issues and concepts. Additionally, within the framework of this
work, we investigated and evaluated the interaction functionality
and the overall installation performance under real-world
conditions during an audiovisual festival / exhibition. This
evaluation process also allowed the collection and interpretation
of some observations related to the behaviour and the means of
interaction of the participating audience.
The rest of the paper is organized as following: In Section 2, the
“dots’ installation architecture is described in detail, focusing on
both the core application and the installation room requirements
and design. Next, a demonstration of the installation interactive
features is presented in Section 3, followed by a brief analysis of
the functional and behavioural observations made during an
installation exhibition. Finally, Section 4 concludes this work
and accents further interaction and audio/visual enhancements
that may be integrated in the “dots” platform in the future.
2. Analytic installation description
As mentioned previously, the “dots” platform developed during
the present work is an interactive audio/video installation, an
interactive game, combining space, time and audio/visual
content. From the design point of view, it consists of (a) the core
application, which is responsible for handling the basic input /
output video and audio signals and for realizing all the signal
processing necessary for recognizing the spatial placement of
the participants as well as for appropriately mixing and finally
producing the audio (and video) signal and (b) the installation
space, where the participants are moving and interacting with
the core application, producing the final, complex audiovisual
output. Both core application and installation space subsystems
are described in detail in the following two Sections.
2.1. The “dots” core application
The application was developed and programmed with the open
source tool “Processing” [6], which represents a powerful
software sketchbook and professional production tool used in
many fields of signal processing, science/technology and arts. A
number of particular Processing libraries were additionally used
for implementing the “dots” installation application (see Figure
1). For example, advanced image and signal processing

-1-
112
dots: an Audio Entertainment Installation using Visual and Spatial-based Interaction
techniques were employed to identify blobs from video real-
time captures and extract all the spatial information needed.
More specifically, the libraries that were used are the following:
i) the video capture library (which is included in the basic
Processing installation package) ii) the minim library [7] and iii)
the “blob detection” library [8]. With the order that they were
listed above, the first code library is responsible for the input of
the video image into the core application. The second employs
the well-known JavaSound API to provide an easy-to-use audio
library and it is here used for the reproduction/playback of the
synthesized audio component. Finally, the blob detection library
is employed for detecting blobs in sequential video frames in
real-time (i.e. for the recognition of the outlines from the
footage of the video signal).
Figure 1: Core application architecture
For the accurate position recognition of the people moving into
the installation space and in order to further process the
incoming spatial information and to convert the extracted spatial
information to an audio stream, a virtual grid was created. This
grid was designed based on the coordinates of the specific
installation space (and more particularly after testing where a
participating person could stand or move inside the installation’s
room, in order to avoid non-active grid areas). Finally, for the
purposes of this work, provided that the selected installation
room has simple geometry with parallel surfaces only, the grid
was designed to have nine discrete rectangle areas with equal
surface areas. It is also significant to mention that apart from the
above discrete rectangle areas, some audio-fading grid territories
were also defined within a specific distance of the grid borders.
This distance was selected to be equal to 10% of a grid area
width/length. The process of audio-fading within the grid
borders is analytically described later in this paragraph.
For synthesizing the visual component of the installation, the
initially defined goal was to design very simple geometrical and
coloured shapes that do not require detailed visual analysis and
attention by the participants. Hence, after an extensive sequence
of design experiments, the final decision was to employ three
circles (dots) with a set of colours of blue, yellow and red. As it
is shown in Figure 2, these three dot types are also included into
the “dots” installation logo, giving the name to the overall
project.
The appearance of these dots on the projection area of the
installation is algorithmically defined by the instantaneous
positions and the total number of the participants within a
specific grid area. More specifically, a dot appears when one of
-2-
the grid areas noted above gets triggered by the presence of a
participant. On the other hand, the dot colour is defined by the
number (N) of the detected participants within the i-th specific
grid area, using the following equation:
, 1
  , 2 (1)
, 3
that is, the blue dot appears when only one person is traced, the
yellow when two people are traced and the red when there are
three or more people in the specific active grid area.
Figure 2: The dots colour shapes designed and the “dots” logo
Focusing on the audio output synthesis module, a number
(typically five) of short sound granules were statically attached
to each grid area. These granules are original, exclusively
produced for the purposes of this work by the authors, with the
use of an external synthesizer and a typical, open source sound
processing software tool. Alternatively, some of the audio
granules were produced using a real-time sound synthesis
engine (typically based on additive or FM synthesis) with a
predefined set of synthesis parameters.
The audio synthesis engine is activated similarly to the visual
component, that is, when a participant is detected to be within a
specific rectangle grid area. In this case, an audio granule
attached to the specific grid area is randomly selected and mixed
to form the final audio output. When the playback of this
granule stops, another one is selected using the same selection
algorithm, provided that the participant is still within the
specific grid area borders. The random granule selection process
is always performed using a normal probability distribution.
When more than one participants are detected in the same grid
area, no audio granules are selected. Instead, the playback
volume for the corresponding audio granule is increased by 1dB
for each additional participant, that is
  (2)
where i denotes the specific grid area, V is a playback gain
constant (in dB) that obviously on the standard playback gain
imposed by the sound-card and the active loudspeaker initial
setup and N[i] represent the number of the participants traced
within the current grid area. Clearly, when all the audio granules
are linearly mixed, the number of participants within a grid area
defines the audio mixing-weights.
The playback of an audio granule immediately stops when no
participants are detected within a specific grid area.
Additionally, as mentioned previously, some audio-fading
territories are defined closely to the grid border. When a
participant is moving towards a 10% grid border line, an audio
fade-out process is activated, by linearly re-adapting the overall
audio granule gain as a function of the distance to the grid
border.
A fade-in gain is correspondingly applied when a participant is
moving away from the 10% grid border (but he is still within the
fade territory). In that case, the final playback gain for the
specific sound granule is progressively decreased as a function
of the participants distance within the 10% grid border area.
113
dots: an Audio Entertainment Installation using Visual and Spatial-based Interaction
2.2. The “dots” installation space
The technical equipment that is needed for the realizing the
complete installation is a personal computer (typically with a
core2duo processor, 2GB of basic memory and ideally a high
performance video processing expansion card), a High
Definition (HD) video camera for creating the input video
signal, a video projector with a minimum analysis of 1024x768
pixels and two monitor active loudspeakers. As it shown in
Figure 3, the camera is located on the ceiling so that it can track
and trace the movement of people that exist and move into the
installation room. In the same Figure, the 3x3 virtual grid
mentioned in the previous Section is also illustrated.
The computer required for executing the core application is
located inside the room, and preferably towards the rear of it.
Based on the minimum requirements that the computer system
must have, it would be good to take into consideration that the
technical equipment must be silent so that there will not be any
audience noise disturbance. Hence the computer was placed
inside a special sound absorptive construction. The “dots” core
application is running on the computer receiving the data from
the camera and projecting the final result on the opposite wall
with the help of the projector. As it is understandable, the
projector must be installed at the back of the room projecting the
final visual effect on the opposite front wall.
The stereo loudspeakers employed are located next to the front
two corners of the room and give feedback to the user with
sounds that are produced each time with a different
combination. In a future enhanced version of the installation,
there will be reproduction surround sound from four or more
loudspeakers in the room.
During this work, a number of important aspects related to the
installation effective realization where considered, such as the
existence of light (and more preferably the presence of ambient
light) which is shed equally in the whole room, and the selection
of a light coloured floor or a white carpet so that the blob
detection accuracy is significantly increased. Moreover the
colour used for painting the walls is an important installation
parameter, as it should be the same for all room surfaces and
preferably light (for example light blue or white).
Figure 3: Typical installation space layout
Finally, a small CRT (or LCD) monitor is installed outside the
installation room, in which the same reproduction of the
-3-
projector inside the room is made. This means, that one visitor
outside the installation room can see the exact position of each
person together with the number of them that are in a specific
part of the grid inside the room.
3. “dots” installation demonstration
The complete “dots” platform was realized for the purposes of
the second annual festival of the department of audiovisual arts,
held in Corfu, Greece on May 2008. The installation room
dimensions were 4 m (width) x 4m (length) x 3.4m (height). As
mentioned previously, the camera was placed on the ceiling.
Under these dimensions, the effective detection surface covered
a satisfactory part of the installation room.
The colour of the room was plain and bright so that the contrast
between the human and the room would be high. This condition
is very significant for the blob detection algorithm, as it clearly
defines the human edges/borders. Accordingly, a white carpet
was placed on the floor in order to additionally increase the final
video signal contrast. Additionally, the illumination inside the
room had to be uniform and ambient; otherwise it would cause
problems to the blob detection process, while the sensitivity of
the detection was adjusted according to the specific illumination
during the initial installation setup.
The equipment described in Section 2.2 had to be hidden in
order to be transparent to the people participating into the
installation and not to attract their attention. The camera that
used was connected via a typical firewire interface, allowing fast
and robust digital video signal transmission to the core
application.
Finally, the gain of the different sound sources was
appropriately and independently adjusted, depending on the
acoustic properties of the room and the disturbance it would
cause to neighbour installations.
An additional, important point of interest was to design an
appropriate audiovisual sub-system to attract the attention of the
people passing outside of the installation, as it was difficult to
notice the installation functionality from outside. An ambient
sound together with the small monitor that was installed outside
of the installation room (as mentioned previously) showing the
dots’ patterns created by the participants’ interaction finally (and
efficiently) solved this problem (see Figure 4). The ambient
sound excited the visitors’ curiosity, so that most of them finally
decided to enter into the installation room.
During a series of observations organized under real usage and
interactions conditions with participants that didn’t know
anything about the experiments (see Figures 5(a) – (d)) it was
found that the combination of both sound/audio and visual
effects were the basic installation characteristics which gave
“energy” to the audience movement. The visual content creates
a strange feeling of following up, which in a small number of
cases makes somebody moving and watching the projector
without necessarily thinking about the existence and production
of sound. On the other hand, some other participants were
mainly interested in audio signal composition without paying
much attention on the visual component. It was interesting that
these users were making strange movements following the
rhythm of the sound composition. However, the majority of the
participants focused on discovering the relation between the
audio and visual components by creating complex motion traces
and using various spatial body movements. The discovery
approach was audio-based (that is the participants were trying to
relate the sound produced with the dots visual effect) or visual-
based (the visitors were associating the visual effect with the
produced audio stream). In most cases, the interaction algorithm
was described as a complex audio/visual interaction, without a
114
dots: an Audio Entertainment Installation using Visual and Spatial-based Interaction

specific algorithmic structure. Specific movements towards the

room borders were also frequently observed, aiming to
determine the spatial interaction limits.

(d)

Figure 5: Demonstrating the “dots” installation

Finally, it should be noted that during the interaction sessions,

the complete installation was considered by many participants to
be a complex game of sound and colours. Hence, in many cases,
two or more participants decided to comprehensively play.
Driven by both sound and projected colours they were able to
produce / synthesize interesting audio and visual effects.

4. Conclusions and ongoing work

In this work, an interactive sound installation prototype called

“dots” is presented which aims to synthesize and produce high-
Figure 4: The external monitor subsystem quality audio streams in real-time, using a novel interaction
algorithm that employs spatial and motion characteristics of the
installation participants. More specifically within the installation
room, the participants are motivated to move using a
dynamically controlled visual content (i.e. a number of coloured
dots, the number and colour of which depends on the number
and the relative position of the participants within a virtually
designed, two-dimensional grid). Accordingly, the human
movement dynamically controls the sound synthesis engine,
which performs weight-based mixing of short synthesized
granules, taking into account the instantaneous participants’
positions within the virtual grid. Hence, a complex spatial, audio
and visual interaction effect is achieved, which produces
continuous audio variations and non-repetitive and time-variant
sound patterns.
Apart from the analytic technical description of the proposed
interactive sound synthesis platform and the discussion of a
number design issues and concepts, during this work we
(a) (b) additionally demonstrated the “dots” installation in the context
of an audiovisual festival recently organized by the department
of Audiovisual Arts, Ionian University. A number of tests
sessions and audience observations organized during the real-
world installation demonstration have shown that the proposed
interactive platform can be indeed considered as both a visual-
based or audio-based synthesis platform, as well as an audio-
visual game with complex synthesis rules. In all test cases, after
a relatively small acquaintance time interval (typically in the
range of 10 seconds up to half minute) the majority of the
participants were able to interact with the system, aiming to
control the interaction parameters and produce specific sound
and visual effects. It should be also noted that depending on
their specific artistic interest, some of the participants
considered the visual effects as the basis of sound production
(c) and vice versa, confirming the dual synthesis nature of the
proposed installation.
Taking advantage of this approach, future enhancements of the
proposed platform will be considered, mainly in terms of spatial
audio reproduction through modern multichannel audio systems
(such as 5.1, binaural [9], wavefield synthesis or ambisonics),

-4-
115
dots: an Audio Entertainment Installation using Visual and Spatial-based Interaction

panoramic, three-dimensional visual content projection using

multiple video projectors and complicated, dynamically adjusted
motion and interaction scenarios.

5. Acknowledgments

The authors wish to thank Mr. Loukas Ziaras for providing the
photographic material included in this work.

References
[1] D. Birchfield, K. Phillips, A. Kidané and D. Lorig,
“Interactive Public Sound Art: a case study”. In Proc. 2006
International Conference on New Interfaces for Musical
Expression (NIME06), Paris, France (2006)
[2] D. Birchfield, D. Lorig, and K. Phillips, "Network Dynamics
in Sustainable: a robotic sound installation", Organised Sound,
10 (2005), pp. 267-274
[3] S. Boxer, "Art That Puts You in the Picture, Like It or Not",
New York Times, April 27, 2005.
[4] N. A. Streitz, C. Rocker, Th. Prante, R. Stenzel, & D. Van
Alphen, “Situated Interaction with Ambient Information:
Facilitating Awareness and Communication in Ubiquitous Work
Environments”. In Proc. HCI International, Crete, Greece (2003)
[5] I. Zannos and J.P Hebert, "Multi-Platform Development of
Audiovisual and Kinetic Installations" in Proc. of the 8th
International Conference New Interfaces for Musical Expression
(NIME 2008), Genova, June 5-7 (2008)
[6] http://www.processing.org (last visited May 29th, 2008)
[7] http://code.compartmental.net/tools/minim/ (last visited May
29th, 2008)
[8] http://www.v3ga.net/processing/BlobDetection (last visited
May 29th, 2008)
[9] Ch. Tsakostas and A. Floros, “Real-time Spatial
Representation of Moving Sound Sources”, Audio Eng. Soc.
123rd Convention, New York, October 2007 (preprint 7279)

-5-
116
AllThatSounds: Associative Semantic Indexing of Audio
Data
Authors: Hannes Raffaseder, Matthias Husinsky, Julian Rubisch
University of Applied Sciences St.Pölten, Austria

Abstract
Motivation: Demand and supply of digitally stored sound files increased rapidly during the
last years and reached an unmanageable amount. For media producers the search for suitable
sounds is an essential and time-consuming part of their work.
The research project AllThatSounds tries to improve the search procedure by indexing the
files in an associative, semantic way. A method for the systematic categorization of sounds is
introduced to simplify the annotation of audio files with metadata. Furthermore additional
data is collected by the evaluation of user profiles and by analyzing the sounds with signal
processing methods.
The project's result is a tool for structuring sound databases with an efficient search
component, which means to guide users to suitable sounds for their sound track of media
productions.

Introduction
Supply and demand for digitally stored audio increased rapidly in the recent years. Number
and diversity, as well as quality of available sound files reached an unmanageable amount.
Efficient approaches in the search for audio data play an important role in the process of
media production. Most of the available search tools nowadays require the user to know
important features of the sound before the search can be carried out. A search request using
semantic features, that are closer to human perception rather than technical parameters, is
hardly possible. In a broader perspective the search procedure is also harder because of the
volatility of the medium sound, which makes it difficult to uptake acoustic events. This results
in a disability to verbally describe sounds properly. Usually, not the sounds themselves, much
more the preceding events that caused the sound are described.
The research project AllThatSounds aimed on simplifying the process of finding suitable
sounds for media productions. For this purpose many different possibilities to categorize and
describe sonic events were analyzed, evaluated and linked. Apart from the applicatory use of
the tool, the research questions raised by the work on these topics trigger a discussion process
about perception, meaning, function and effect of the sound track of a media product.

The meaning of the sound track in media production

The immense influence of the sound track on the artistic and commercial success of an audio-
visual product is undisputed in contemporary media theory. Examples like “2001 – A Space
Odyssey” by Stanley Kubrick or “The Birds” by Alfred Hitchcock prove this impressively. At
the same time it is astounding that their directors found the music of György Ligeti and the
sound of the Trautonium of Oscar Sala respectively, as both composers were very much
unknown at the time of the production.
These examples show that many times these insider tips, these seldom played and by a broad
audience unheard pieces, are the bits that make the works special. Apparently, an enormous
special knowledge, far greater than that of the director or the producer, is needed together
with an excellent overview over the available music and sounds, to find the best sounds that

117
fit optimally to the mood and image of the product. Since only a few persons have a sufficient
knowledge in this area, the search for audio is performed more or less arbitrary, at best guided
by intuition and the own taste. Many interesting pieces of music and audio files will not be
taken into account.

Description and categorisation of audio files

The main goal of the project AllThatSounds is the description and categorization of digitally
stored audio files. In particular, four different approaches are considered:
• Descriptive Analysis
• Listener’s Analysis
• Machine analysis
• Semantic analysis

Descriptive Analysis
The descriptive analysis enables for the uniform description of acoustical events from the
sound designers perspective. The aim is that already at the time the upload of a sound into the
database takes place, it can be sufficiently and distinctly registered.
Based on the insights of Murray Schafer (3), David Sonnenschein (4), Darry Truax (5), Theo
von Leeuwen (5) and earlier thoughts of the author (2), a general classification of sounds was
developed, which enables for a differentiated description in the following categories: Type of
excitation, sound source, timbre, pitch, dynamics, room, familiarness and possible use.
Very soon the difficulties when trying to describe the sonic events in an adequate and
universal way, become apparent. On one hand a detailed and satisfactory description of the
sound requires high complexity and accuracy, but on the other hand the process of describing
the sound must not take too long in practical use. To keep time short when describing the
sound at upload, full description cannot be reached.

Listener’s Analysis
The context, personal mood of the listener, the field of application or the intention of the
production have a big influence on the description of an acoustical event. There is a semantic
gap between human perception and objectively measurable signal parameters. Sometimes,
two technically very similar signals have a totally different effect.(2)(6)
Therefore only one description of the acoustical event might lead to an unwanted result in
another context. To disarm this problem, AllThatSounds uses collaborative interfaces and
description methods which became popular under the buzzword Web 2.0.

Machine Analysis
In the recent years many research institutions started working on methods to give machines
similar perceptual possibilities as humans. In the research field of “Music Information
Retrieval” fundamental work was done by using the results of signal processing methods on
audio data for getting an idea about its musical content or its semantic content. Classification
is often done using artificial intelligence models. Basic examples are automatic beat and
tempo tracking, pitch estimation and melody or instrument extraction.
For AllThatSounds the application of sound similarity models promised to be very
meaningful. In many audiovisual productions sounds, which have no relation at all to the
objects shown on the screen, but still appear to be authentic, are used. Therefore a similarity

118
model based on Mel Frequency Cepstrum Coefficients (MFCCs) is used to compare each
sound with each other in the database. This model proved to be very useful in comparing and
classifying sounds and music (1). The 20 most related sounds are stored with each sound and
enables that way for a new kind of browsing for viable sounds.
Furthermore, for each sound descriptors, corresponding the MPEG 7 standard and some other
psychoacoustic parameters like roughness and sharpness, are calculated and can be used in
search requests.

Semantic Analysis
In media production the semantic content in acoustic events is utterly important to achieve a
certain perception at the consumer. Until today it is almost impossible to automatically extract
semantic meaning from audio signals automatically, since not only signal parameters, but to a
wide extent also cultural and social experience of the listener play a role. How these
denotations to specific aural events happen, how they manifest and are described, is in many
respects still not investigated. Theo van Leeuwen shows (6) that different intermodal
interplays play an important role, as well as the social, cultural or historic environment.
AllThatSounds aims on combining the annotated metadata from the users and the calculated
features to explore possibilities to automatically extract semantic denotations. Conclusions
can be done, when the database has been used for a longer time by numerous users.
Also, to investigate this topic further, a library of short film and video clips, that stand out
because of their interesting or typical sound design, was created. These clips were tagged and
indicated in categories like “event”, “symbol” and “acoustic material”. That way it is easy to
search for clips with certain events like “murder” (event), “grief” (symbol) or “orchestra
music” (acoustic material). A detailed evaluation of this material in the clip library has yet to
be done.

Prototype and Perspective

The first prototype of the AllThatSounds Soundlibrary with an improved search tool was
released to the public on May 30th 2008 as a web based application under
www.allthatsounds.net . At the time point of the release the database contained about 3000
sounds, recorded by the students of the University of Applied Sciences St.Pölten’s Media
Engineering course. The sounds can be downloaded in uncompressed wave-format and be
used under the Creative Commons Sampling Plus license. Everyone is welcome to participate
in the library. The release of a standalone client using AllThatSounds technology is planned.
Even though the prototype application proved to be extremely useful in the search for the
right sound in many respects it revealed a lot of new research questions. In future research
projects a deeper examination of the semantic content of acoustic events is absolutely
necessary. Nonetheless, AllThatSounds offers a great basis to build on.

Acknowledgement
Starting in October 2005 a research team at the University of Applied Sciences St.Pölten
worked together under the project leader Hannes Raffaseder with the Vienna based companies
Audite / Interactive Media Solutions, Team Teichenberg and, since April 2007 also the
University of Applied Sciences Vorarlberg. The works were supported by the Österreichische
Forschungsförderungsgesellschaft (FFG) in their FHplus funding programme until May 2008.

119
References
[1] Feng, D., Siu, W.C., Zhang, H.J. (2003): Multimedia Information Retrieval and Management, Springer-Verlag, Berlin.
[2] Raffaseder, H. (2002): Audiodesign. Hanser-Fachbuchverlag, Leipzig.
[3] Schafer, Murray R. (1994): The Soundscape – Our Sonic Environment and the Tuning of the World. Destiny Books,
Rochester.
[4] Sonnenschein, D. (2001): Sound Design – The Expressive Power of Music, Voice, and Sound Effects in Cinema.
Michael Wiese Productions, Studio City.
[5] Truax, B. (2001): Acoustic Communication (2nd ed.), Ablex Publishing, Westport.
[6] van Leeuwen, T. (1999): Speech, Music, Sound. MacMillan Press Ltd., London.

120
 IMPROVe: The Mobile Phone as a Medium for Heightened Sonic
Perception
Richard Widerberg
Dånk! Collective
Karl Johansgatan 47 G
SE-41455 Göteborg, Sweden
rwiderberg@g mail.com
+46708977452

Zeenath Hasan
School of Art s and Communication
Malmö University
SE-20506 Malmö, Sweden
zeenath.hasan@gmail.com

Abstract. In this paper, we describe the design and research phase of a project that aims to create conditions for heightened sonic
perception through a mobile phone based software application. The initial design concept is that of an aural architecture for sonic
socio-cultural exchange where sonic realities of the everyday are improvised live in a non-linear mode. The design approach adopted
is collaborative. The project is a work in progress.

1. Introduction
Our experience of our everyday lives is mediated through a
‘multitude of mechanically produced sounds’ [1]. The everyday
sounds that we experience are produced outside of our own
volition. The capability to capture sounds and play it back has
made it possible to listen to sounds outs ide of its original context
[2]. Sound recording technology can be used as an extension of
listening and enhance our aural awareness[3]. The mobile phone
is also a medium through which sounds are heard outside of
their original context. However, the normative definition of the
mobile phone as a medium for communication has restricted its
potential as a medium for sounds that exist outside of the
immediate tele-communication. This design and research project
explores the potential of the mobile phone as a medium of
communication beyond its currently dominant role as a
transmitter of sounds. The design space for exploration is the
mobile phone as a digital networked medium that is
appropriated by social networks to communicate across
boundaries of time, space and context [4]. The project thus
proposes the design of the mobile phone as a medium for the
exchange of everyday sounds within communities and across
socio-cultural contexts by mobilizing the potential of the mobile
phone as a tool for the production of everyday sounds.
2. Approach
The project adopts a collaborative design approach by gathering
a community of interest [5] consisting of members from
different communities of practice [6] who conduct their
activities at various levels of involvement. The design process
gathers participants around an initial des ign concept that is used
as a boundary object [7] or as a common point of reference. The
design concept that emerges in the interactions with the gathered
participants is regarded as an artifact [8] because perceiving it as
such reveals the direct and incidental connections between the
different aspects that come together in its creation.
3. Initial Design Concept
3.1. Background
A community of practice with a history and tradition of working
closely with found sounds through the means of electronic and
digital tools is that of electro-acoustic musicians. When viewed
on a scale of involvement from active to passive, the members
of this community of practice involves not only those who
actively engage in the creation and reproduction of their own
tools or instruments for mixing sounds, but also those who
passively listen to the sounds that are produced. Lastly, one
mode in which electro-acoustic musicians might work to
compose sounds in a group is spontaneously through live
improvisation.
3.2. Improvisation
Professional musicians have practiced improvisation to create
compositions spontaneously. Melodies, harmonies and rhythms
are combined within the traditional structures of mus ic that the
professional musician has been trained in. Musical instruments
have been known to tear away from their established histories to
accommodate and challenge each other. When the mobile device
is used as a musical instrument in an improvisation, what
musical structures, if any, emerge? Improvisation is a collective
activity. Professional musicians practice it to scope the
boundaries of the musical form. What pursuits will the untrained
improviser indulge in when involved in sonic improvisation?

-1 -
121
4. Working Prototype
A working prototype was developed for use as a common point
of reference for discussions on the design concept with gathered
participants and other stakeholders.
4.1. Scenarios
Scenarios were constructed as a way to unfold the initial design
concept at work and also to have a shared understanding within
the project team. Provided below are two scenarios in brief.
4.1.1. Scenario 1
A group of friends, untrained in music, record soundscapes from
their daily life. The group meets at a local pub, where there is a
sound system for playing the gathered sounds. They perform a
live-remix of the sounds on their mobile devices. The aural
exchange affects individual and group understanding at the
cultural and social level through a sharing of the everyday
soundscape.
4.1.2. Scenario 2
Trained mus ic practitioners, like cellists, record sound objects
through a mobile phone. The group meets in a concert hall. They
perform a group improvisation with the collected sounds
through their respective mobile devices. The exchange is an
exploration of the formal aspect of aural compos ition that builds
on traditional music structures and creates new forms of music.
4.2. System Functionality
Sounds are collected via a mobile phone and sent to a location
where they can be played back into a sound system. The same
mobile phone controls the playback of the collected sounds in
the soundsystem. Playback control occurs in the physical
location of the soundsystem. The sounds that are played back
are process ed live via interaction through the mobile phone. The
output of the processed sound can be heard directly through the
soundsystem.
4.3. Prototype Application
Python programming language was used for rapid prototyping
on Nokia Series 60 devices [9]. The phone microphone is used
for the recording interaction. As current audio processing
capabilities on the phone requires some amount of work, mixing
and playback functions are processed on an external computer
using Pure Data [10]. The processing is then controlled live by
the mobile phone via bluetooth connection.
4.4. Graphica l User Interface
For the recording of sounds, a simple interface allows recording,
listening to the recording and then uploading of the recording to
a server. The recording-interface cons is ts of three buttons for
these three functions. For the improvisation interface, the four-
way directional button was the only key activated for interaction
with the GUI. The ‘Play’ command selects recorded sounds at
random. Three options are provided for live mixing of the
played back sounds, allowing the participant to control the
volume, speed and loop length. The ‘Stop’ command stops the
playback.
5. Fi el d Acti vity
The working prototype has been used in various venues and
with different settings. For example at a private coffee lounge
consisting of 15 people, at a dinner for four in a res taurant and
in two separate weeklong workshops resulting in a live
performance by 20-25 people per workshop at two different
public venues. Video documentation from different locations,
-2 -
which also clarifies the concept, can be viewed on the project's
web site [11]. The initial design concept was also introduced
separately to eight participants who were selected based on their
active to passive involvement with found sounds. Participants
were given a mobile phone to carry for a month’s period after
which separate discussions were held with each of them.
6. Findings
When the mobile phone was used as an instrument for recording
everyday sounds by participants who do not pursue musical
performance as a profession, they reported emotional, nostalgic,
anecdotal and politically analytical associations with the sounds
they chose to record. Participants who pursued music production
as a career mainly recorded aesthetically pleas ing sounds for use
in their next performance. The possibility of recording sounds
and then processing and playing them back into a soundsystem
makes the mobile device a musical instrument among
professional musicians. Although the above two groups have
been presented as a dichotomy, it is not a strict division becaus e
both groups reported associations with sounds other than that of
the aesthetic.
7. Conclusi on
The project began with the objective to conceive of the mobile
phone as a medium for heightened sonic awareness. It has
achieved proof of concept on people’s reception to the existence
and use of such an application on the mobile phone. The next
threshold is to use this application for heightened sonic
awareness in different pedagogical contexts.
References
[1] Bull, M., Back, L. (Eds) The Auditory Culture Reader. Berg
Publishers, Oxford, New York, (2003)
[2] Kahn, D.. Noise, Water, Meat: a history of sound in the arts.
The MIT Press, London, (2001)
[3] Truax, B. Acoustic Communication. 2nd edition. Ablex
publishing, Westport, (2001), 219
[4] Rheingold, H. Smart Mobs. Perseus Publishing, Cambridge,
Masachusetts, (2002)
[5] Fischer, G. Beyond ‘Couch Potatoes’: In Consumers to
Designers and Active Contributors First Monday, volume 7,
number 12. (2002)
[6] Wenger, E. Communities of Practice: Learning, Meaning and
Identity. Cambridge University Press, USA, (1998)
[7] Marick, B. Boundary Objects
http://www.visibleworkings.com
[8] Diaz-Kommonen, L. Art, Fact and Artifact Production:
Design Research and Multidisciplinary Collaboration.
University of Art and Design Helsinki, Finland, (2002)
[9] PyS60. http://sourceforge.net/projects/pys60/
[10] Pure Data real-time graphical programming environment
for audio, video, and graphical processing.
http://www.puredata.org/
[11] IMPROVe project web site. http://www.riwid.net/improve
122
 Audio Interface as a Device for Physical Computing

Kazuhiro Jo, RCAST, University of Tokyo / Culture Lab, Newcastle University, jo@jp.org

Abstract. In this paper, we would like to describe the employment of audio interface as a device for physical computing. We
compare the audio interface with other devices and describe its characteristics. We also present examples of the employment with
three different art works, Monalisa "shadow of the sound", The SINE WAVE ORCHESTRA stay amplified, and AEO. We explain
the implementation of each work with different physical components. Finally, we discuss some of the potential of the employment of
audio interface for future implementation.

1. Introduction Table 1. Comparison of devices

An audio interface is a device that allows a computer to (specifications are taken from the website)
receive/send audio signal from/to the outside. It converts a
stream of bits (digital) into a time-varying voltage (analog) and
vice versa. Most of computers currently have a built-in audio
interface. The interface generally provides two (stereo) built-in
input/output channels with 44.1KHz sampling rate and 16-bit
sampling resolution. People employ such audio interface for
listening or recording audio in the field of music, film, and/or
gaming. Some of external audio interface serves more
input/output channels, higher sampling rate, and higher
sampling resolution (e.g. 56-ch, 192KHz, 24-bit).
Attachment:
Physical computing is an approach to sensing and controlling
Most of computers currently have a built-in audio interface with
the physical world with computers [8]. People create a
two (stereo) input/output channels. It enables people to connect
conversation between the physical world and the virtual world
with several components (e.g. headphone, microphone, speaker)
of the computer by employing not only standard computer
without additional input/output equipments. External audio
interface (e.g. keyboard, mouse, display, speaker) but also
interface offers extra input/output channels.
several kinds of physical components (e.g. sensor, switch,
actuator). The conversation comes into sight as a form of
Sampling resolution:
musical controller, gaming console, interactive installation etc.
Sampling resolution is a measure of how preciously the device
treats the quantity of object. The audio interface basically offers
To realize such conversation, people have been developed
16-bit sampling resolution, while the other devices provide 10 or
several input/output devices, such as Infusion I-CubeX
12-bit resolution. Some external audio interface has higher
(http://infusionsystems.com/), Arduino (http://www.arduino.cc/),
resolution such as 20-bit or 24-bit.
and MAKE Controller Kit
(http://www.makezine.com/controller/). These devices allow a
Sampling rate:
computer to communicate with the physical world by employing
Sampling rate is a measure of how frequently the device looks at
physical components. The devices convert the signals from/to
the input from the outside. Most of the audio interfaces support
the components with several protocols. I-CubeX employs MIDI,
44.1KHz or more sampling rates because of the sense of hearing
arduino employs serial, and MAKE Controller Kit employs OSC
(about 20Hz to 20KHz). Owing to the limitation of the protocol
(Open Sound Control). The programming environment such as
or the hardware, other devices offer lower sampling rate than the
Cycling74 MaxMSP (http://www.cycling74.com/), Adobe Flash
audio interface.
(http://www.adobe.com/), and Processing
(http://www.processing.com/) manages multiple input/output
2. Implementation
signals from/to the physical world and maps them into other
We have employed the audio interface as a device for physical
types of representation (e.g. graphic, text, sound).
computing with three different art works, Monalisa "shadow of
In this paper, we would like to propose the employment of audio the sound", The SINE WAVE ORCHESTRA stay amplified,
and AEO. In each work, we employed different physical
interface as a device for physical computing. It allows a
components with the audio interface: a push button for Monalisa
computer to communicate with the physical world by audio. The
"shadow of the sound", a fader controller and a foot switch for
signals from/to the components in the physical world are treated
The SINE WAVE ORCHESTRA stay amplified, and two
as audio and processed on the programming environments in the
accelerometers, a set of distance sensor, and two custom built
computer.
converter boxes for AEO. In following sections, we explain the
implementation of each work in both hardware and software
Audio interface is unique to compare with the other devices,
settings.
I-CubeX, Arduino, and MAKE Controller Kit, at least in three
points: attachment, sampling resolution, and sampling rate
2.1. Monalisa "shadow of the sound"
(Table 1).
Monalisa "shadow of the sound" is an installation created with
Norihisa Nagano. It is a part of software platform Monalisa
which enables to "see the sound, hear the image" by treating all
the image and the sound as the sequence of numbers represented

-1-
123
[Audio Interface as a Device for Physical Computing]
by binary codes [7]. The work was premiered at Open Space at
NTT InterCommunication Center from 9th June 2006 to 11th
March 2007. The work consists of a set of computers with
custom version of Monalisa applications, projector, camera,
microphone, push button, and speaker situated in a room (Figure
1). We briefly describe the procedure of the work below.
Figure 1: Monalisa "shadow of the sound"
When entering the room, each participant saw his/her image,
caught with the camera, projected on the screen. When he/she
pushed the push button, the image was captured as a static
bitmap image data. The image data was transformed to the
application and automatically played as a stream of sound
through the speaker. The microphone simultaneously captured
the stream of sound and transformed it to the application. The
application sequentially re-projected the incoming sound data as
an image on the screen from top left to down right of the screen.
Because of the acoustics of the room, the re-projected image
was blurred like a shadow with reverberations of the sound.
2.1.1. Push button
We employed a push button and a set of input/output channel of
a built-in audio interface to receive the action from participants.
The push button has two terminals. The terminals conduct with
the push of the button. To detect the conduction with the audio
interface, we cut a standard audio into two fragments. Each
fragment of the cable is attached to each terminal of the button.
Each other side of the fragments connected to the input/output
channel of the audio interface (Figure 2).
Figure 2: Push button
We detected the conduction of the button as audio signal on
MaxMSP. We assigned a sine wave for the output channel. We
set up a gate for the input channel. When the terminals conduct
with a push of a participant, the gate identifies an incoming
audio signal. With the identification, the application captures the
image of the participant.
2.2. The SINE WAVE ORCHESTRA stay amplified
The SINE WAVE ORCHESTRA stay amplified is a work
which was premiered at WAVES exhibition at the Exhibition
Hall ARSENALS of the Latvian National Museum of Art from
25th August to 17th September 2006 [3]. It is a work of a
participatory sound performance project The SINE WAVE
ORCHESTRA (http://swo.jp). The author serves as the one of
four core organizers of the project. In the project, under the
basic concept that each participant plays a sine wave, people are
invited to create a sea of sine waves as a collective sound
representation. Each of the work in the project employed the
different instrument with different temporal, physical,
environmental, and procedural settings [6]. This work consists
-2-
of a set of control computer, sound synthesis computer, light,
foot switch, fader controller, rotational controller, and 16
speakers circularly situated in a room (Figure 3). We briefly
describe the procedure of the work below.
Figure 3: The SINE WAVE ORCHESTRA stay amplified
When entering the room, each participant was exposed to a
collective sound representation. The sound representation
consisted of sine waves that previous participants had produced.
16 speakers that horizontally encircled the participant produced
the sounds. When a participant stood on the foot switch in front
of the controllers, a new sine wave started to play. As he/she
moved the fader controller, the frequency of the sine wave was
changed. As he/she rotated the rotational controller, the sound
source position (i.e. speaker) of the sine wave was changed.
He/she selected the frequency and the sound source position of
the sine wave. By leaving from the food switch, he/she left
his/her own sine wave as a part of collective sound
representation. The volume of the sine gradually attenuated and
disappeared after the exhibition period. As more participants left
their sine waves, more sounds were accumulated. During the
exhibition period, the collective sound representation was
changing from a phase where each sine wave was discriminable
to a cluster that consisted of mutually interfering sine waves like
a white noise that contains all frequencies.
2.2.1. Foot switch, Fader controller
We employed a foot switch, a fader controller, a rotational
controller, and two sets (stereo) of input/output channels of a
built-in audio interface to receive the action from participants.
The foot switch and the fader controller are connected with a
control computer with the audio interface and the rotational
controller is directly connected with the control computer
through USB. The foot switch has two terminals. The terminals
conduct with the push of the foot switch. The fader controller
also has two terminals. It behaves as a variable resistor. The
move of the fader changes the value of the resister. The foot
switch and the fader controller are separately connected with
two sets of input/output channels of a built-in audio interface in
the same manner of the push button of Monalisa "shadow of
sound" (Figure 4).
Figure 4: Foot switch (left) and Fader controller (right)
We detected the conduction of the foot switch and the resistance
of the fader controller as audio signals on MaxMSP. We
assigned a sine wave for each output channel. For the input
channel of the foot switch, we set up a gate to identify the
124
[Audio Interface as a Device for Physical Computing]

conduction in the same manner of the push button of Monalisa
"shadow of sound". For the input channel of the fader controller,
we set up a volume calculator. It reports the change of the
resistance of the fader as the change of volume of incoming
audio signal. With the change of the volume, the control
computer sends a message for the synthesis computer engine to
change the frequency of a sine wave. The sampling resolution
for the volume is 16-bit (i.e. 65535 level). Therefore it is
sufficient to accomplish subtle changes of the frequency of the
sine wave.
audio input, three audio outputs, one extra audio input/output, a
power socket, and three ring modulation circuits (Figure 6).
Every audio inputs/outputs of the converter boxes are directly
connected to the outputs/inputs channel of a 56-ch, 192KHz,
24-bit external audio interface, RME Fireface 800
(http://www.rme-audio.com/).

2.3. AEO
AEO is a sound performance project consisting of three
members: Eye, Taeji Sawai and the author. AEO has performed
at several international festivals (e.g. Dutch Electronic Art Figure 6: Converter boxes (Black for the sphere with the
Festival 2004, Radar 7 at 24 Festival de MEXICO 2008). In the transmitter, White for the sphere with the receiver)
project, each member takes one of three roles: performance
(Eye), sound design (Sawai) and instrument design (the author). The amplified box has two audio output, one volume, two audio
During AEO performances, the performer holds the instrument inputs, a power socket, and two channels amplifier (Figure 7).
in each hand and shakes, sways, or swings them. These
movements by the performer produce patterns of sound and light
through devices and a computer. The instrument has undergone
a transition in function and forms over six iterations [5].

In the latest iteration for the performance at Fujirock Festival

2008 (http://www.fujirockfestival.com/) on 27th July 2008, the
instruments consist of two sets of hand-held plastic spheres.
Each sphere contains a three-axis accelerometer, a part of
distance sensor, a small light bulb, and a switch. One sphere has
a transmitter of the distance sensor, while another has a receiver.
The accelerometer converts the inclination and acceleration of
each axis (i.e. x, y, z) into changes in analog voltage. The
distance sensor measures the distance between two spheres by
employing ultrasonic sine wave (40KHz) with the transmitter
and the receiver (Figure 5). The small light bulb changes its
brightness by the change of analog voltage. The switch changes
the represented patterns of sound.
The device for the instrument was changed in each iteration. We
employed I-CubeX for the first and second iteration, and
MakingThings Teleo (http://www.makingthings.com/teleo/) for
the third and the fourth iteration. In the fifth iteration, we
developed two sets of converter boxes for the accelerometer and
the distance sensor. We also employed MAKE Controller Kit to
change the brightness of the bulb with PWM (Pulse With
Modulation). In the latest iteration, in addition to the converter
boxes, we developed an amplified box for the small light bulb.
The accelerometer, the distance sensor, and the switch are
connected to the convert box with a multi cable, while the small
light bulb is connected to the amplified box with other cable.
Figure 7: Amplified box
The audio outputs are connected to the sphere and the audio
inputs are connected to the output channel of RME Fireface 800.
We treat all the audio signals on MaxMSP.
Accelerometer:
For each accelerometer, we employ one audio input, three ring
modulation circuits, and three audio outputs of the converter box.
We assign an audio signal to the audio input. The audio signal is
distributed to the three ring modulation circuits. The amplitude
of the audio signal carries the change in analog voltage from
each axis of the accelerometer (i.e. x, y, z) by employing ring
modulation circuit with two transformers (Sansui ST-75) and
four germanium diodes (1K60) (Figure 8). Each audio output
produces the change of acceleration in each axis as modulated
audio signal.

Figure 8: Ring modulation circuit

Figure 5: AEO instruments
2.3.1. Accelerometer, Distance sensor, Small light bulb
We developed two sets of converter boxes for the accelerometer
and the distance sensor, and an amplified box for the small light
bulb. Each converter box has a connector to the sphere, one
We detected the inclination and acceleration of each axis of the
accelerometer as a separate audio signal. For each signal, we
setup a volume calculator. It reports the change in analog
voltage from each axis as the change of volume of incoming
each audio signal. We employ the reported value to control the
parameter of represented sound of the performance.

-3-
125
[Audio Interface as a Device for Physical Computing]
Distance sensor:
For distance sensor, we employ one extra audio input and one
extra audio output of the converter boxes. The extra audio input
is connected to the transmitter and the extra audio output is
connected to the receiver. We assign an ultrasonic sine wave
(40KHz) to the extra audio input. The audio interface enables
the utilization of such high frequency with its high sampling rate
than the built-in audio interface. The external audio output
produces the audio signal from the receiver. We detected the
distance between two spheres as an audio signal. For the signal,
we setup a volume calculator. It reports the change of the
distance as the change of volume of the audio signal. We
employ the reported value to control the parameter of
represented sound of the performance.
Small light bulb:
For each small light bulb, we employ one audio output, one
audio input, and one channel amplifier of the amplified box. By
feeding audio signals in appropriate amplifier, it is possible to
bring the coil of the bulb in an excited state. We assign the
represented sound of the performance to the audio input. We
amplify the sound and connect it to the small light bulb. In the
performance, the brightness of the bulb reflects the change of
the represented sound changes. The volume of the amplified box
adjusts the threshold of the brightness.
3. Related works
There are some precursors who also tried to employ audio
signals as means to communicate with the physical world.
Allison and Place [1] developed the SensorBox. The device
accepted six sensor inputs and two audio inputs. The data from
each sensor was carried as the amplitude of a sine wave, which
was located in the 18KHz to 20KHz, and mixed back on the two
audio inputs. They did not provide its technical detail well,
however their approach was quite same as ours of the converter
box of AEO. Canadian artist artificiel also explored the light
bulb as sound source in their electro-acoustic installation
"beyond6281 [2]". They feed processed audio signals in
powerful amplifiers in the similar way to the small light bulb of
AEO. PingPongPlus [4] employed audio signals for position
tracking. It detects the position of ping-pong ball with a sets of
microphones mounted under the ping-pong table. They detect
the time of hit with each microphone and calculate the position
with the time difference. Ms. Pinky [9] by Scott Wardle consists
of a set of vinyl records and software running on MaxMSP. The
vinyl contains special signals and from the signal, the software
decodes the velocity, direction, and physical position of the
needle on the surface of the vinyl in real time.
4. Discussions
We explored the characteristics of the audio interface as a
device for physical computing. The audio interface
accomplished higher sampling resolution and sampling rate than
the other devices. In the installation Monalisa "shadow of the
sound", we simply employ the push button with the built-in
audio interface. In The SINE WAVE ORCHESTRA stay
amplified, we accomplish subtle change of the frequency of the
sine wave with 16-bit sampling resolution. With the latest
instrument of AEO, the sampling resolution and sampling rate
of the audio interface enables to represent subtle movements of
the performer as patterns of sound and light than the other
devices. During the latest implementation of AEO instruments,
we have conducted informal comparison with the audio
interface and the MAKE Controller Kit. We assigned the change
of the acceleration into the change of the volume of white noise
sound. With the audio interface, when the performer shakes the
sphere, the resulted sound could be heard as a kind of maracas
-4-
percussion with its fine time precision. However, with the
MAKE Controller Kit, the resulted change of the sound did not
reflect the subtle movement of the performer. As Wessel and
Wright pointed out [10], the low latency between gesture and
gesture controlled audio output is essential for live computer
music performance. We believe that our approach suggests a
sensitive way to communicate physical world with its
preciousness of measurements and time.
Wessel and Wright argued the possibility to apply existing audio
DSP (Digital Signal Processing) modules (e.g. Filters, Fast
Fourier Transforms, Linear Predictors) to process the signals
from/to the physical world [10]. We have not investigated such
possibilities well. However we consider handling multiple
sensor signals with one channel of the audio interface by
employing band-pass filters.
As a future work, we consider to publish our developments for
public. We plan to provide various instructions and examples of
the employment of audio signals to communicate with the
physical world. We hope to encourage people to stimulate each
other to discover the potential of the audio interface as a device
for physical computing.
Acknowledgements
We would like to thank for the other member of Monalisa, The
SINE WAVE ORCHESTRA, and AEO. Monalisa "shadow of
the sound" was developed under the support of NTT
InterCommunication Center and FY2005 IPA
(Information-Technology Promotion Agency) Exploratory
Software Project (Project Manager: KITANO Hiroaki).
References
[1] Allison, J., Place, T., SensorBox: practical audio interface
for gestural performance, In Proceedings of the 2003
Conference on New interfaces For Musical Expression,
pp.208-210 (2003)
[2] artificiel, beyond6281, Art + Communication 2006: WAVES,
RIXC, the Center for New Media Culture, Riga, Latvia,
pp.38-39 (2006)
[3] Furudate, K., Jo, K., Ishida, D., Noguchi, M., The SINE
WAVE ORCHESTRA stay amplified, Art + Communication
2006: WAVES, RIXC, the Center for New Media Culture, Riga,
Latvia, pp.104-105 (2006)
[4] Ishii, H., Wisneski, C., Orbanes, J., Chun, B., Paradiso, J.,
PingPongPlus: design of an athletic-tangible interface for
computer-supported cooperative play, In Proceedings of CHI
'99. ACM, New York, NY, pp.394-401 (1999)
[5] Jo, K., Transition of an Instrument: The AEO Sound
Performance Project, Leonardo Music Journal No.17, pp.46-48,
(2007)
[6] Jo, K., Furudate, K., Ishida, D., Noguchi, M., Transition of
instruments on The SINE WAVE ORCHESTRA, ACM
Computers in Entertainment, on October (2008) (to be
published)
[7] Jo, K., Nagano, N., Monalisa: "see the sound, hear the
image", In Proceedings of the 8th International Conference New
Interface for Musical Expression, pp.315-318 (2008)
[8] O'Sullivan, D., Igoe, T., Physical Computing, Boston, USA:
Thomson Course Technology (2004)
[9] Wardle, S., Ms Pinky, http://www.mspinky.com/
[10] Wessel, D. and Wright, M., Problems and Prospects for
Intimate Musical Control of Computers, Computer Music
Journal. 26, 3, pp.11-22 (2002)
126
 Automatic genre and artist classification by analyzing improvised
solo parts from musical recordings
Jakob Abesser, Christian Dittmar, Holger Grossmann ({abesjb,dmr,grn}@idmt.fraunhofer.de)
Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany

Abstract. This paper introduces a set of high-level features to describe instrumental solo-parts. The set consists
of 148 single- and multidimensional features related to the melodic, harmonic, rhythmic and structural properties
of four instrumental domains. A simple yet common instrumentation model has been applied to describe both
the soloing and the accompanying instruments as well as rhythmic and melodic interaction between them. To
evaluate the features’ discriminative power related to different musical styles, an evaluation for content-based
genre and artist classification has been performed each with two different test sets consisting of symbolic and
real audio data. Two different classifier approaches have been utilized, one commonly used support vector
machine (SVM) classifier with preliminary discriminant analysis (LDA) and one novel approach based on the
Rhythmical Structure Profile which allows a tempo-adaptive representation of the rhythmic context provided by
the accompanying instruments. For both classification scenarios, ensemble decisions based on single instrument-
related classifiers led to the highest scores of 84.0% for genre and 58.8% for artist classification.

1 Introduction commonly derived from the progression of pitches and

intervals. Basic statistical attributes like mean, stan-
Within all musical cultures, music-pieces are never per-
dard deviation, entropy as well as complexity-based
formed exactly the way they are transcribed. Each
descriptors are therefore applied such as in [13], [12],
musician has got an individual conception and under-
[5] and [11]. Genre classification systems are commonly
standing of aspects like timing, dynamics and articu-
based on low- and mid-level features. A combination
lation which directly affects his rendition of the piece.
of high- and low-level features for this purpose is de-
These nuances enhance our perception of music to be
scribed in [11]. In [13], a set of 109 musical high-level
lively, exciting and rich of variety. The majority of
features are applied for genre-classification for three
publications within the Music Information Retrieval
root-genres each with three sub-genres achieving very
(MIR) community aim to characterize musical pieces
good classification results. To the current knowledge of
by means of features like tempo, bar measure or key.
the authors, there are no works dealing with genre clas-
Our goal is to analyze the semantics of the performance
sification solely based on high-level features extracted
of these pieces by investigating the individual style of
from the solo part of a song. An overview of publi-
the participating musicians. We focus on solo parts be-
cations concerned with performance analysis and the
cause they offer the most freedom of individual musical
investigation of improvisation and interaction can be
expression to the soloist in spite of a mostly predefined
found in [3] and [15]. In this context, different appli-
rhythmical and harmonical composition. Since many
cation scenarios were introduced within the literature
contemporary musicians are usually active within one
such as the analysis of improvisations performed in
or only a few music genres, the assumption can be made
clinical music therapy [7] or artist classification based
that there exists typical playing styles within songs of
on typical sequences derived from the progression of
a certain music genre besides general rhythmic, har-
tempo and dynamic within piano performances [14].
monic and melodic characteristics.

1.1 Related work 2 Exposition

The extraction of high-level features is covered in sev-
eral publications in the MIR literature. Different ap-
proaches to extract rhythmic features as for instance
derived from typical rhythmical deviations [8], from
the percussion-related instrumentation [9] of a music
piece or from different statistical spectrum descriptors
based on periodic rhythm patters [11] have been re-
ported. Melodic and harmonic high-level features are
To investigate solo parts from songs of different music
genres, a simple yet prevalent instrumentation model
consisting of four different instrument categories has
been used. It contains the melody instrument (MEL)
played by the soloist, the harmony (HAR), bass (BAS)
and percussion instrument (DRU) as musical accompa-
niment providing the rhythmic and harmonic context
within the solo part. All songs within the assembled

127
Automatic genre and artist classification by analyzing improvised solo parts from musical recordings
test sets (described later on in 2.3) fit into this model.
2.1 System overview
2.1.1 Transcription and pre-processing
The implemented system allows the processing of both
symbolic and real audio data (MIDI and Audio files).
Our experiments are all based upon excerpts from the
analyzed solo parts of 20 to 40 seconds length. To ex-
tract the score parameters from symbolic audio files,
the MIDI Toolbox for MATLAB [6] has been used
for data conversion. It allows to derive a list of all
notes containing the parameters note onset and dura-
tion (both in seconds and bars), velocity, MIDI pitch
and MIDI channel. To process real audio data, the
Transcription Toolbox [4] (developed at the Fraunhofer
Institute for Digital Media Technology) has been uti-
lized. It is a software toolbox that encapsulate four
different algorithms to perform a separate transcrip-
tion of the melody, harmony, bass and drum track of
a music-piece. It furthermore offers the user manifold
ways to correct the transcription results e.g. by choos-
ing a temporal quantization grid or a pitch correction
causing all notes to fit to the manually selectable key
of the analyzed excerpt of the song. The Transcrip-
tion Toolbox also extracts the beat grid of the song
which enables a subsequent projection of all detected
note onsets from their absolute values in seconds to
certain multiples of the bar lengths and thus allows a
tempo-independent onset representation.
2.1.2 Quantization and harmonic analysis
For some of the extracted rhythmic high-level features,
the note onsets and durations have additionally been
quantized to a 64th-note beat grid. Furthermore, a
simplified harmony analysis has been applied to the
harmony track. The goal was to determine the root
note of each played chord. The system is able to de-
tect the most common 2-, 3- and 4-note chords in all
possible inversions by using chord interval templates.
In case the chord was unknown, the lowest note was
supposed to be the root note. For internal representa-
tion, all played chord notes are artificially elongated to
allow a detection of the harmonic context for each note
played by the soloist. By mapping the interval between
each note of the solo melody towards the detected root
note of the simultaneously sounding chord, a represen-
tation called functional pitch was defined. Here, only
the type of the interval (third, fifth, etc.) is projected
to the corresponding integer value (3, 5, etc.), the size
(e.g. major or minor third) is not taken into account
to increase the independence from the key-type (major
or minor).
2.2 Feature extraction
To describe the soloists’ way of playing within a solo
part, three main questions have been investigated.
Which notes are played within the given harmonic and
rhythmic context? How is the solo part structured?
To what extent does the soloist interact with the ac-
companying instruments? Timbral characteristics of
the instrument, the precise instrumentation of a solo
(e.g. whether the soloist plays a electric guitar or a
saxophone) as well as applied playing styles of the
soloist (like glissando or vibrato) have explicitly not
been taken into account here. A total of 148 high-level
features both single- and multi-dimensional have been
implemented and a certain sub-set of them can be ex-
tracted for each one of the four instrumental tracks.
In the following four sections, a selection of the imple-
mented features will be explained in detail.
2.2.1 Melodic and harmonic features
Three different representations of the melodic progres-
sion have been examined to derive melodic and har-
monic high-level features. Besides the absolute and
relative pitch (intervals between adjacent notes in half-
tone steps mapped to one octave), the functional pitch
(see 2.1.2) of each note within the solo is determined
based on the aforementioned harmony analysis. A wide
range of different features characterizing the melody
have been extracted. These are e.g. the pitch range
in halftones, a measure of chord-tone ratio (derived by
analyzing simultaneously played chord notes of the har-
mony instrument) as well as the temporal ratio of poly-
phonic parts, chromatic note sequences (with consec-
utive intervals of a half-tone step) and note sequences
with constant pitch. Additionally, the progression of
the relative pitch was also converted into the corre-
sponding functional pitch values to derive a key- and
scale-independent representation of the applied inter-
vals. All single probabilities (e.g. of a fifth downwards
or a third upwards) as well as some other basic statis-
tical features like zero- and first order entropy and the
D’Agostino measure [2] have been furthermore com-
puted as melodic features. The temporal ratio of frag-
ments with a constant melodic direction is mapped to
a measure of balanced direction, furthermore the dom-
inant direction (ascending or descending) is thereby
determined as additional feature.
2.2.2 Rhythmic features
For the computation of rhythmic high-level features,
the note onsets, durations and inter-onset intervals
have been analyzed. To characterize the perceived
rhythmical precision of a track related to different beat
grids (4th, 8th, 16th and 32th-note grid), the quantiza-
-2-
128
Automatic genre and artist classification by analyzing improvised solo parts from musical recordings
tion cost was calculated as an inverse measure of rhyth-
mic precision within the particular beat grid. Further-
more, a swing ratio was also calculated for the beat
grids mentioned above using a similar approach as de-
scribed in [8]. To derive a rhythmical representation of
all notes of an instrumental track that is independent
from tempo and bar measure, we introduce the Rhyth-
mical Structure Profile (RSP) which is derived from the
un-quantized note onsets. The RSP is based on parti-
tioning each bar length into k equidistant grid points,
where different corresponding binary and ternary val-
ues of k (2-3, 4-6 etc.) have been investigated, each grid
as an un-shifted and a shifted version related to down-
and off-beat positions. Each note of the instrumental
track is mapped onto these grids that contain a grid
point aound the note’s onset time. By summing up
the note’s normalized velocities mapped to all defined
grid-points, the RSP can be calculated and saved in
form of a three-dimensional matrix. Afterwards, one
can both analyze the temporal distribution of notes
over all grids and as well as within each grid. They al-
low the calculation of the features dominant rhythmical
grid (containing the majority of all notes), dominant
rhythmical feeling (down- or off-beat) and dominant
rhythmical characteristic (binary or ternary). Further-
more an algorithm to detect syncopations within dif-
ferent rhythmical grids based on the RSP was imple-
mented.
2.2.3 Structure-related features
To describe the structure of a solo, both rhythmical
and melodic repetitions within the instrumental tracks
have been seeked. For this purpose, an algorithm for
detecting repeating patterns within character strings
(Correlative Matrix approach [10]) has been utilized.
These character strings are derived from the absolute
pitches as well as from the quantized onset and dura-
tion values. All detected patterns were mapped into a
three-dimensional representation consisting of the pa-
rameters length, incidence rate and mean distance. As
a fourth parameter, supposing to characterize the re-
call value of a detected pattern, the so called relevance
has been calculated from the normalized pattern pa-
rameter values as rP at = lP at,N orm + fP at,N orm + (1 −
dP at,N orm )2 . It is based on the simple assumption that
the recall value increases with ascending pattern length
and frequency and decreases with ascending temporal
distance whereas its impact is furthermore reduced by
the squaring operation. Basic statistical features like
mean, median, standard deviation, minimum and max-
imum value are calculated for each of the four pattern
parameters as well as the number of patterns related to
the overall number of notes of the current track. After
all, 63 feature values contain manifold information on
the distribution of both rhythmic and melodic patterns
within the solo.
2.2.4 Interaction-related features
To describe the interaction between the soloist and
the accompanying musicians, two approaches have
been followed. By calculating the euclidean distance
between bar-wise RSPs one can determine whether
two musicians play rhythmically in unison or use
complementary rhythms. The aforementioned chord-
tone ratio (see 2.2.1) is furthermore calculated bar-
wise to characterize the progression of the harmony-
relatedness of the solo melody. For both vectors, both
mean and standard deviation are calculated as fea-
tures.
2.3 Evaluation
The partitioning of the data sets into training and
test data generally has been performed class-wise to
a proportion 50% - 50% randomly for each iteration,
whereas a total of 50 iterations were passed through
for each evaluation scenario.
2.3.1 Genre classification
For the genre classification experiments, a 6-fold-
taxonomy has been utilized, consisting of the music
genres Swing (SWI), Latin (LAT), Funk (FUN), Blues
(BLU), Pop-Rock (POP), Metal-Hardrock (MHR). Be-
sides instrument-related single classifiers, the efficiency
of ensemble classifiers (based on a probabilistic ma-
jority decision) was investigated. Two different ap-
proaches have been chosen, a common support vector
machine (SVM) classifier with preliminary linear dis-
criminant analysis (LDA) and a nearest-neighbor clas-
sifier based on the aforementioned RSP.
LDA-SVM classifier Before the evaluation, all feature
vectors are extracted from solo excerpts of a particular
data set. After a feature-wise variance normalization of
the training data, LDA has been performed for dimen-
sionality reduction of the feature space to 5 dimensions
(since we are dealing with a six-class problem). Sup-
port vector machines have been chosen as classifier ap-
proach, more precisely C-Support Vector Classification
(C-SVC) using the radial basis function (RBF) kernel
as described in [1]. Subsequent to variance normal-
ization and dimension reduction, the optimal classifier
parameters C and γ are determined using a threefold
grid-search and the classifier model is trained after-
wards. To evaluate the trained classifier, all feature
vectors from the test data passed the same two prelim-
inary steps. Finally the classifier output was compared
with the ground truth label vector.
-3-
129
Automatic genre and artist classification by analyzing improvised solo parts from musical recordings

RSP classifier The main idea behind this novel ap- LDA-SVM RSP Human
proach is to model the rhythmic context provided by Input MIDI Audio MIDI MIDI
the accompanying instruments during the solo part
(re-synth.)
which is usually specific for each music genre. There-
fore, the RSPs of the bass track, the harmony track
MEL 63.8 44.4 – 37.6
and the drum track with separate investigation of the HAR 57.3 45.1 63.7 –
bass drum (B) and snare drum (S) track are computed MEL + 71.7 – – 58.8
globally over the total length of the analyzed excerpt HAR
to extract the most frequent rhythms and to minimize BAS 70.1 51.8 66.3 –
the influence of rhythmical breaks and variations. Af- DRU 62.2 35.9 61.0 (B) –
ter their computations, the instruments’ RSP matri- 47.7 (S)
ces of each song in the training data set are stored ALL – – – 63.1
in genre- and instrument-related containers for later
ENS 84.0 63.4 73.2
use. After applying the same computation step for the
songs within the test data set, the euclidean distance
between each extracted RSP matrix and all stored ma- Table 1: Genre classification results in %
trices related to the same instrument is calculated. The
minimum distances to each container can be converted
to assignment probabilities to the according genres due lagher, Jimi Hendrix and Steve Ray Vaughan). Since
to rhythmic similarity. the accompanying musicians are not supposed to have
impact on artist classification, only the features de-
Listening test To compare the results of the two clas- rived from the melody track have been provided to the
sifier approaches with the ability of human listeners to artist LDA-SVM classifier. Training and evaluation is
assign an excerpt from a solo part to a music genre, performed as described for genre classification.
a listening test has been performed. 25 test persons
between 20 and 42 years (µ = 26 years) of age with a 3 Results
relatively high average musical background of µ = 12
years participated. Each test person had to assign 15 The genre classification results are listed in table 1.
excerpts from different solo parts (randomly selected Besides the two classifier approaches described in 2.3.1,
from the symbolic-audio genre testset) to one of the the results of the listening test related to the three
given music genres. The instrumentation of the ex- investigated scenarios test are presented in the fifth
cerpts has been unified (melody, harmony and bass in- column. The single classifiers of both the LDA-SVM
strument assigned to a piano sound) to prevent a genre and the RSP approach achieved classification scores
assignment based on commonly appearing instruments up to 71.7% for MIDI and up to 51.8% for real audio
(e.g. Metal-Hardrock with electric guitar). Three input. Using ensemble based classification, scores up
different instrumentation scenarios have been investi- to 84.0% respectively 63.4% within the aforementioned
gated, the first five pieces only consisted of the melody 6-fold genre taxonomy were achieved. We assume that
instrument, the second five pieces of the melody and partly incomplete or erroneous transcription results are
harmony instrument and the last five pieces of the com- the main reasons for lower scores for real audio data.
plete instrumentation (see 2). A simple metronome has The achieved scores for artist classification are 58.8%
been furthermore added within the first two scenarios (electric guitar) and 56.0% (saxophone).
to provide a rhythmical orientation to the test persons.
4 Summary and future work
2.3.2 Artist classification
In this paper, we presented different high-level fea-
To evaluate the features’ discriminative power to iden- tures related to the melodic, rhythmic, structural and
tify the artist who is playing a certain solo, two exper- interaction-related description of improvised solo parts.
iments have been performed. For each of them, four A simple but common instrumentation model allows an
musiciancs playing the same instrument and being allo- application of these features for a wide range of differ-
cated to related music genres have been chosen and for ent music genres. Using the extracted information of
each of them 30 excerpts of solos have been collected. all four instrumental tracks by applying an ensemble
In detail, the first set consists of four famous saxophone classifier, classification rates up to 84.0% within a 6-
players (John Coltrane, Dexter Gordon, Charlie Parker fold genre taxonomy were achieved. As the listening
and Joshua Redman) and the second one of four well- test’s results show, a genre classification solely based
known electric guitar players (Eric Clapton, Rory Gal- on the solo part of a song is a difficult task. Despite of

-4-

130
Automatic genre and artist classification by analyzing improvised solo parts from musical recordings
the dominant solo instrument, the genre assignment is
primarily based on the characteristics of the accom-
panying instruments. Considering that timbre- and
instrumentation-related features have not been taken
into account here and only the solo part has been ana-
lyzed, the results are encouraging for further research
within this topic. As the results of the artist classi-
fication reveal, describing the way of playing by using
high-level features basically allows a discrimination be-
tween different performing artists. On the other hand,
it still exists a lack of semantic information. To over-
come this, additional features to describe playing styles
in detail as well as specific instrumentation and tim-
bre aspects need to be implemented to derive better
results for artist classification. Regardless of the clas-
sification task one has to emphasize the importance of
a well-performing transcription system in order to an-
alyze real audio data by the use of high-level features
based on score parameters.
References
[1] Chih-Chung Chang and Chih-Jen Lin. LIBSVM:
a library for support vector machines. In http:
// www. csie. ntu. edu. tw/ ~ cjlin/ libsvm
(last called: 10.09.2008), 2001.
[2] P. J. Ponce de Léon and J. M. Inesta. Pattern
recognition approach for music style identifica-
tion using shallow statistical descriptors. In IEEE
Transactions on System, Man and Cybernetics -
Part C : Applications and Reviews, volume 37,
pages 248–257, March 2007.
[3] R. López de Mántaras and J. L. Arcos. AI and mu-
sic: From composition to expressive performances.
AI Magazine, 23:43–57, 2002.
[4] C. Dittmar, K. Dressler, and K. Rosenbauer. A
toolbox for automatic transcription of polyphonic
music. In Proc. of the Audio Mostly, 2007.
[5] T. Eerola and A. C. North. Expectancy-based
model of melodic complexity. In Proc. of the
6th Int. Conf. of Music Perception and Cognition
(ICMPC), 2000.
[6] T. Eerola and P. Toiviainen. Midi toolbox: Mat-
lab tools for music research. In www. jyu. fi/
musica/ miditoolbox/ (last call: 10.09.2008),
Jyvskyl, Finland, 2004. University of Jyvskyl.
[7] J. Erkkilä, O. Lartillot, G. Luck, K. Riikkilä, and
P. Toiviainen. Intelligent music systems in music
therapy. In Music Therapy Today, volume 5, 2004.
[8] F. Gouyon, L. Fabig, and J. Bonada. Rhythmic
expressiveness transformations of audio record-
ings - swing modifications. In Proc. of the 60th Int.
Conf. on Digital Audio Effects (DAFX), Septem-
ber 2003.
[9] P. Herrera, V. Sandvold, and F. Gouyon.
Percussion-related semantic descriptors of music
audio files. In Proc. of the 25th Int. AES Conf.,
2004.
[10] J.-L. Hsu, C.-C. Liu, and A. L. P. Chen. Discov-
ering nontrivial repeating patterns in music data.
In IEEE Transactions on Multimedia, volume 3,
pages 311 – 324, September 2001.
[11] T. Lidy, A. Rauber, A. Pertusa, and J. M. Iesta.
Improving genre classification by combination of
audio and symbolic descriptors using a transcrip-
tion system. In Proc. of the 8th Int. Conf. on
Music Information Retrieval (ISMIR), 2007.
[12] S. T. Madsen and G. Widmer. A complexity-
based approach to melody track identification in
midi files. In Proc. of the Int. Workshop on Artifi-
cial Intelligence and Music (MUSIC-AI), January
2007.
[13] C. McKay and I. Fujinaga. Automatic genre clas-
sification using large high-level musical feature
sets. In Proc. of the Int. Conf. in Music Infor-
mation Retrieval (ISMIR), pages 525–530, 2004.
[14] C. Saunders, D. R. Hardoon, J. Shawe-Taylor,
and G. Widmer. Using string kernels to iden-
tify famous performers from ther playing style.
In Proc. of the 15th European Conference on Ma-
chine Learning (ECML), pages 384–395, 2004.
[15] G. Widmer and W. Goebl. Computational models
of expressive music performance: The state of the
art. In Journal of New Music Research, volume 33,
pages 203–216, 2004.
-5-
131
 The Heart as an Ocean
exploring meaningful interaction with biofeedback
Pieter Coussement, Marc Leman, Nuno Diniz, and Michiel Demey

IPEM, Dept. of Musicology Ghent University, Blandijnberg 2, B-

9000 Ghent, Belgium
pieter.coussement@ugent.be

Abstract. This paper discusses the need to redefine the concept of ‘interaction’ within the context of interactive (audio) installations.
This discussion is based on the realization of ‘The Heart as an Ocean’, a media piece that explores the relationship between auditory
senses and biometric feedback.

1. Interactivity in the context of arts

‘The Heart as an Ocean’ is a new media piece (designed by the
first author) that is based on the artistic use of the participant’s
auditory senses and biometric feedback. In a broader context,
‘The heart as an Ocean’ also functions as an experimental
setting in which new forms of interactivity are explored , more
particularly in the context of media installations and new
technologies. The media piece explores the fundamentals of
meaningful interaction by looking to what extend the physiology
of the body can be both sensor and actuator in an art context.
The installation was first exhibited at Gallery Jan Colle in
Ghent, Belgium i in February 2007.
2.1. Interface and Usability
The usability and user interaction are among the most defining
factors when developing interactive media installations for art.
When dealing with usability it is important to take into account
that technical complexity of both the user- interface and the
sensory data mapping which mainly influences the experience.
On the one hand the complexity can be due to the fact that the
user interface is too complex or on the other hand that reactions
of the system having little to none of an obvious correlation with
the public’s interaction. Until now this resulted into a way of
thinking about ‘interactivity’, usability and interface design as a
subtle equilibrium between the need of easy-to-use interfaces
and a certain amount of complexity. This should result in an
exciting experience where people are challenged to explore and
play with the installation. Although this may be sufficient to
explore some technical issues surrounding new media
installations it seems not sufficient enough to explore a more
conceptualised meaningful interaction.

2.2. Meaningful interaction

Figure 1: Gallery Jan Colle, Belgium.
2. Problem definition
Within the arts, interactive media installations become more and
more prominent, although interactive media installations are
seldom part of the permanent collection of museums. Interactive
media installations have been mostly exhibited at special
festivals like the Ars Electronica festival in Austria or
SIGGRAPH in the U.S. Recently some private organizations,
started to build collections of media installations (in Belgium,
see the Verbeke Foundation ii ). Although they are oriented
towards a more general public rather than a public of specialists,
it still requires a specialised exhibition environment and some
advanced maintainability of the installations.
In the design of ‘The Heart as an Ocean’, art has been conceived
as a way to communicate between artist and public, but also to
communicate on a broader social level among the public itself.
Art communicates ideas through sense and the artistic
experience is a result of the effectiveness of this communication.
It involves a conversation between artist and art piece, and
between art piece and public. Within an ideal interaction this
relation is symbiotic both in concept as in realisation. There is a
need to differentiate between responsiveness and interaction
eventhough both may have its distinct use in digital arts and
entertainment.
The responsiveness of interactive art can be situated between a
range of 100% responsive and 100% autonomous. From that
perspective, hyperinstruments iii for example, are 100%
responsive since they always respond in the same manner to the
same stimuli. However in using hyperinstruments in interactive
installations, the public is often confronted with a learning curve
during which technical possibilities and functions of the device
have to be explored and learned. Of course this may be fun and
exciting in itself. Yet, in the end this focus on the instrument
may result in a rather limited experience of interaction, since the
interaction does not necessarily imply a goal-direction.
Therefore no effect of non-mediation or implicit conceptual
meaning can be developed. As a result, the artist may have the
feeling that the public is not able to transcend beyond the barrier

132
of the technological mediator, and the public may have the
feeling that it never experienced the artist’s intentions.
The question is whether it is possible to cope with this problem
of technological mediation and learning curves. Are there ways
to overcome the inherent limitations of hyperinstruments?
line output to create an aggregated device providing eight line
level outputs. An extra nineteen-inch screen showed the
software GUI. The speakers were hidden in order to emphasize
the atmosphere of the exhibition space, giving more room to the
audio.

3. Basic concept

In ‘The Heart as an Ocean’, the goal was to get a natural flow
of communication without the restrictions of having a too
technical interface that could obtrude the intended interaction.
The interaction had to work like an affordance. No sophisticated
explanations should be necessary to interact, and user feedback
should be based on a very strong homogeneity in ‘experiencing’.
4.2. Software.
The software is developed using Cycling 74’s MAX/MSP vi . On
the top level there is a GUI running, which enables a real-time
HD recording of the interaction. This can be rendered to a DVD
and is offered as a multiple.

3.1. In depth concept

The media piece was designed in such a way that the state of
mind of any person who interacts with the installation could be
sensed. This way it would be possible to influence that person’s
physiology through sound in such a way that the outcome would
be similar for every participant.
To achieve this goal, a synthesised ocean wave was created, that
imitated the breaking of a wave on an imaginary shore. The
intensity, level, duration and amplitude of the wave are all
derived from the heart rate of the person who interacts with the
installation. The way in which the musical parameters relate to
heart rate is as follows: an agitated person, with a strong and fast
heart rate, would generate strong loud and fast waves. A calm
person, with a weak and slow heart rate, would generate slow Figure 2: GUI enabling a named recording.
and gentle waves. Since a new wave is generated at every
heartbeat the auditory illusion of a sea breaking on a shore is Beneath the GUI level there are several patches working
created. This effect is emphasised by a spatial movement of each together to capture and calculate the heart rate from the sensor,
wave in a setup with eight speakers. Each wave starts its cycle to create the waves, to take care of the spatial position of the
randomly at one position and moves through the auditory space wave and to render the recordings.
using the other speakers. The sound of the sea was initially
chosen because of its soothing effect. Secondly, water has
played an important role in the spiritual, psychological and
physical ablution throughout history. Moreover, the sound of the
sea has all frequency bands in it and therefore, it can be
conceived as a sort of a white noise signal spread out in space.
Because of this, it largely numbers out all other surrounding
sounds resulting in a very personal auditory space. Michael
Wenger, Dean of Buddhist Studies at the San Francisco Zen
Centre iv , speaks about this:

"Moving water is 'white noise,' in which you can hear many

things. Each individual may hear a different song in the water.
Just listening to the sound--not tying it to anything, just letting
sound wash over you--is a way of letting go of your ideas and
directly experiencing things as they are."

4. Technical realisation

When the installation was first presented there were some

difficulties related to the use of the heart rate sensor, which had
to be taken into consideration while programming the software. Figure 3: Calculating wave size in milliseconds according to
During a recent upgrade of the project, the heart rate sensor has heart rate.
been replaced with a wireless sensor. This gives better results
and leads to a less obtrusive interaction.

4.1. Hardware
‘The Heart as an Ocean’ consisted of seven satellite speakers,
one subwoofers and a heart rate sensor hooked up to an
Arduino v board connected to a Mac Book. The seven satellite
speakers were spread across a wall spanning eight meters. The
subwoofer was discretely placed in the room. An M-Audio
Firewire audiophile was used in conjunction with the computer

133
The Heart as an Ocean- exploring meaningful interaction with biofeedback

have a distinct spatial impression, a lot of extra reverb effects

were added to ameliorate the experience. Having a bigger
exhibition space with more speakers and more generated waves
is preferred.

7. Discussion

The reaction of the public corresponded with the intended

Figure 4: Spatial position of wave.
5. Action-reaction cycle.
In order to get a meaningful interaction, a feedback loop was
installed, based on an action-reaction cycle model for ecological
interaction (Leman, 2007) vii . The cycle consists out of 4 stages,
called: Play, Listen, Judge and Change. Play is the stage in
which sound is generated. Listen involves the perception of
sound, while Judge involves its evaluation. In the final stage the
action is changed to modify the resulting sound. Within
interactive art, both the Judge and Change stages are often left
unexplored, resulting either in responsive art intended as
responsive art or responsive art intended as interactive art. Both
have little to do with what I call meaningful interaction. Within
the states Judge and Change lies the essence of meaningful
interaction because they rely on the activity of the participant,
even if this activity is unconsciously stimulated.
In the case of ‘The Heart as an Ocean’, the Judge and Change
stages are implemented in a rather direct way. Judge is
calculated as the relationship between speed and amplitude of
previous waves and the heart rate. Change is calculated as the
amount of energy the installation will implement in the next
wave. To complete the cycle, the installation will first Listen to
the heart rate and only then start to Play.
design of this project. And after some time, depending on the
subjects, the effects of the installation on the public’s
physiology were quite similar. This leads to some points of
interest in extending our research of interactive art. First of all,
there is indeed the need to further (re)define interaction within
interactive arts. The idea that the installation needs to mediate
intelligently between the artist’s concept and what the public
conceives is of great importance. Secondly, if you choose to
implement interactivity, it should be a necessity from the point
of view of the affordance of the installation, which engages the
user in an interaction that is originally intended, rather than in an
interaction about the technological mediator. Finally, it can be
stated that interactive and responsive art are essentially different
from each other. The main difference is that interactive art
subscribes an action-reaction cycle model in which feedback has
an effect on the conditions for interaction.
References.
1
http://www.jancolle.be/ (last visited 20.08.2008)
ii
http://www.verbekefoundation.com/ (last visited 20.08.2008)
3
http://www.media.mit.edu/hyperins/ (last visited 20.08.2008)
4
http://www.sfzc.org/ (last visited 20.08.2008)
5
http://www.arduino.cc (last visited 20.08.2008)
6
http://www.cycling74.com (last visited 20.08.2008)
7
Marc Leman, Embodied Music Cognition and Mediation Tech-
ology, The MIT Press, Cambridge, Massachusetts

Eventhough the mapping is rather obvious, the results were

impressive. People interacting with the installation all made
similar comments. The participants found the interacting to have
a soothing effect. Most of the participants identified with the
sound. They liked to listen to themselves. Some of them
fantasized what kind of beach (either sandy or rocky etc.) they
were on through analysing the sound. Most of them wanted to
walk in the exhibition space instead of sitting down, what has
led me to change the sensor into a wireless sensor for future
exhibitions.

6. Problems to solve / Problems solved.

Concerning the installation, some changes have further

improved the basis for interacting. Originally, the heart rate
sensor used a three point measuring technique with the
consequence that the public had to sit down in front of the
installation. In addition, there was also a lot of noise in the
sensor data that had to be filtered out. More accurate
commercial heart rate sensors were at that time too expensive or
too hard to implement. As a result, not every heartbeat came
across and sometimes, although largely filtered out, noise was
interpreted as a heartbeat. By using new commercial wireless
heart rate sensors, these problems have now been fixed.
Although a speaker setup spanning eight meter is enough to

-3-
134
 EarMarkIt: An Audio-Only Game for Mobile Platforms

David Black, Kristian Gohlke, and Jörn Loviscach

Hochschule Bremen (University of Applied Sciences), Bremen, Germany
dblack@stud.hs-bremen.de, kgohlke@stud.hs-bremen.de, joern.loviscach@hs-bremen.de

Abstract. The fields of both mobile gaming and audio-only gaming have rapidly expanded in the last few years. In this work, we
attempt to combine these two fields and present an immersive audio-only game prototype meant for future application in mobile
gaming platforms. The game is based on the scenario of an open-air market, in which the player roams freely by actually walking
around, guided by the auralization played back via headphones. We created a wireless stride- and direction-sensing device with
attached manual input controls to simulate future mobile gaming devices that allow the user to explore an open space while playing a
game.

1. Introduction
Our objective for this project is to create an audio-only game in
which a player can navigate a virtual audio space by walking
around in the real world. By using audio cues present in the
game, the player, wearing a wireless sensor device, can detect
his position relative to the objects and interact with them. Our
project aims at developing an audio-only game that combines
the accessible and innovative features of immersive audio with
the freedom and ubiquity of mobile gaming.
station before the tram leaves the station. The player is allotted a
purse of money, which he can use to pay for the food items. In
the course of the game, there is also a thief who tries to steal
food that the player has collected. Points are awarded based on
the speed in which the player completes the entire set of
purchases and the number of food items remaining in his or her
basket.

Previous work [1] demonstrated a similar audio-only immersive

game that could be controlled via a stationary handheld input
device. We seek to extend this concept and add a mobile device
with location sensing. Given the technical issues with location
sensing based on GPS, cellular phones or WLAN for small
distances [2], we elected to use a stride and direction sensor to
create a game in which the user could interact with his own real-
life space. We keep the important elements of creating audio-
only games in mind, including the simplification of auditory
cues, meaningful motion tracking, and narrative interactivity [3].

Today’s handheld mobile electronic devices, such as phone,

PDAs, or MP3 players often provide large computing power.
Some of these devices are also equipped with sophisticated
sensing and communication capabilities. However, most of them
still have rather tiny buttons and screens and thus do not appear
to be platforms that could yield an immersive gaming
experience. But such an argument would neglect the audio
capabilities: Almost any mobile music player combined with
decent headphones can deliver an immersive audio experience
that is comparable to the output quality of most computers.

Because it does not rely on any visual feedback, audio-only
games are highly accessible for blind or visually impaired
people [3, 4]. However, common audio-only games rely solely
on buttons as the primary means of interacting with the game
[1]. What differentiates our game is the integration of a wireless
stride and direction sensor box that allows the player to interact
with the game in a more pleasurable way by taking full
advantage of his or her physical and perceptual capabilities.
2. Game Outline
The basic premise of the presented game is that of an open-air
food market in which the player navigates through a corridor
filled with different sounds of vendors, musicians, and other
typical market events, see Fig. 1. In the game, the player needs
to purchase a certain amount of food items and then find a tram
Figure 1: The user walks across the market, buying food and
trying to catch the tram in the end.
2.1. Gameplay
The game is based on a number of auditory cues that help the
player navigate the audio space of the market and provide
information about the food items that need to be collected.
“Environmental” sounds are original self-recorded or freely
available recordings of typical markets that are spaced along the
one-dimensional playing corridor to simulate a market.
“Interaction” sounds are recordings that involve the gameplay
elements including the food vendors, the tram and the thief.
Recordings of the vendors include those in which they are heard
trying to sell one of four single food items: apples, sugar,
donuts, and fish. The tram sound is that of a tram passing a
station. The thief sound is made up of footsteps that gradually

-1-
135
EarMarkIt: An Audio-Only Game for Mobile Platforms
become audible if the user does not move for a specified amount
of time.
The primary difficulty in the game is for the player to
distinguish between the environmental and interaction sounds,
and to recognize when a vendor is selling a certain food product
for an acceptable price. When the player encounters a vendor
selling food, he or she can make a purchase by pushing a
momentary switch button on the wireless sensor and input
device that correlates to that type of food. If the player presses
the wrong type of food button, the vendor becomes “upset” that
the player requested food that is not being sold at that location,
and that food vendor disappears, making that instance of food no
longer available for the player to purchase at this place. If the
player requests the correct type of food, the food item is added
to his basket, the price of the food is deducted from his purse,
and that specific vendor becomes silent. Pressing a query button
in conjunction with one of the food buttons allows the player to
check the remaining food that he must still purchase. The player
must seek out the best price for each food item so that the
money in the purse is sufficient for all of the items required.
The second difficulty in the game is that of limited time. The
player must find the tram station and arrive in time. Since the
player is required to search the market for the best prices of
food, he cannot simply buy the first item of each type of food
that he encounters. He or she must remember where certain
items were sold and possibly return to them.
The third difficulty in the game is the thief, who continually
follows the player through the corridor. If the player does not
take a certain number of steps within a specified time, the thief
advances towards the player and by random selection steals one
of the food items from the basket. After a food item is stolen, the
player must navigate again to try to find a remaining vendor -
which do not reappear after selling the player an item - in order
to meet the game requirements. The thief element is included in
the game to keep the player active and moving.
The game ends when either the player arrives at the tram station
or the tram leaves the station before. A score is calculated based
on how much money the user has left, how much time is left
before the tram leaves, and how many items from the required
list the user has collected in his basket.
2.2. Navigation
Similar to a real market layout, the auditory cues are placed
along a corridor with a set beginning and end. The player starts
the gameplay at the beginning of the corridor and is able to
navigate through the corridor using the wearable wireless sensor
device. Except for the thief sound, the auditory cues take up a
certain portion of the corridor and the user can hear them pass
by as he navigates through. The cues are panned either to the
right or left side of the player’s position. As the player passes
them, the localization of the auditory cues in the stereo field
changes based on the user’s position to them. This, coupled with
a high-shelf filter with variable cutoff frequency, simulates the
effect of the sounds traveling around the user. When the player
changes his heading (as measured by the direction sensor), the
stereo field adjusts accordingly so that sounds rotate relative to
the user’s orientation.
3. Prototyping Process and Game Hardware
Our audio-only game is meant in the future to be played on
mobile devices such as phones, PDAs, or even MP3 players.
However, for the development of the current prototype we used
-2-
flexible off-the-shelf components in both hardware and
software. The use of these well-known tools allows us to set the
focus of development on the game concept and evaluation of the
game, rather than having to deal with platform-specific
problems that might arise when developing for a wide range of
mobile devices (also see [5] for another example of rapid audio
prototyping using Max/MSP ). As has been shown in previous
audio-for-gaming implementations [6], this rapid prototyping
approach also makes it possible to evaluate the use of
technologies that are not yet implemented in the current mobile
devices for the mass market.
For the current prototype of the game, we chose sensors that are
already available in some modern mobile phones today (such as
Nokia 6210 Navigator, Apple iPhone and similar). These phones
could be a near-ubiquitous platform to control, or given enough
processing power, even run the entire game. As the gaming
hardware is thus already available to many people, the game
could be quickly deployed to any mobile phone that provides the
required technical capabilities. The vibration motor of the phone
could also be used to augment the audible feedback by basic
haptic sensation.
In our prototype, the game is controlled by a small, battery-
powered, wireless device that the user wears on the belt, a
headband or similar. The device contains different sensors that
allow us to determine the current heading of the player, detect
strides, and button presses. To achieve this, we use a three-axis
accelerometer, a magnetometer (“compass sensor”) a Bluetooth
modem and pushbuttons tied together by an Arduino
microcontroller board, see Fig. 2. The compass is used to
determine the players heading and needs to be calibrated once at
the beginning of the game. As the player walks, the
accelerometer data is used to detect footsteps [7]. Each step the
player makes in the real world also changes his position in the
game. By integrating the steps with the current heading of the
player, it is possible to move freely inside the game in two
dimensions. The soundscape is adapted to the player’s current
position by changing the stereo field and using some basic
psychoacoustic effects to simulate sounds coming from different
directions.
Figure 2: To test the prototype software, we developed a
wireless handheld controller and a sensor
To handle the incoming raw values from the sensors more
flexibly on the PC that computes the audio, we developed a slim
serial-to-OSC proxy. The proxy requests the raw sensor readings
from the microcontroller. Inside the proxy this data can be
filtered and interpreted to generate OSC messages [8]. These
messages can be streamed to any network address, thus the
different modules of the game could also run on different
computers if necessary. This approach has proven to be a robust
and easy way to interchange data between the different modules
of the prototype, while keeping the architecture of the prototype
as flexible as possible.
136
EarMarkIt: An Audio-Only Game for Mobile Platforms
The OSC stream from the proxy is sent to the actual game
software: A Max/MSP patch processes the OSC messages and
handles the game logic and sound output. Based on the user’s
current orientation and estimated position the game logic
generates the audio stream. To play the game, the player does
not have to be aware of the computer. It remains in the
background and merely processes the user input from the
wireless device to generate the audio stream that is then sent to a
set of wireless headphones.
As the interaction with the game is not bound to a specific
location, the player can use any open space to interact with the
game, yielding a user experience that is very similar to using a
mobile device. Whereas it allows simulating the experience of
using a mobile device, the prototype stays open for rapid
extensions and changes to both hardware and software during
development. Once the development of the prototype is finished,
the game can be ported to any sufficiently equipped mobile
device.
4. Obstacles
When a player steps in one place instead of actually walking,
each of these steps still moves the position of the game
character. However, the sensors we use only allow a rough
estimate of the position of the player in the real world. This is
not crucial for the game as the game logic does not need to
know the absolute distance the player moved, but only a relative
position measured in steps and the current heading. In addition,
this effect allows the game to be played in small places.
Some commercially-available pedometers that are meant to
measure absolute distances suffer from similar problems,
although some claim a quite decent positioning error of about
3% [9]. A further issue is that the one-axis magnetometer inside
the wireless device gives erroneous headings whenever it is
tilted. The integration of the compass readings with the
accelerometer data or the use of inertial sensors, such as
gyroscopes, could eliminate this obstacle.
Commonly, computer games provide some display of inventory,
score or other information concerning the state of the game.
They are often placed in the visual periphery of the player, to be
readily available. Creating a similar information channel in an
audio-only game appears rather difficult. Text-to-speech
synthesis for auditioning the player’s score could be a way to
provide information whenever it is requested. This would be a
pragmatic way to provide the data to the player, but it appears as
a brute-force approach. We are currently investigating other
means to convey the necessary information, without having to
interrupt the soundscape of the game.
5. Conclusion
In terms of our objectives of creating an immersive audio-only
game that could be applied to mobile devices, we have
succeeded insofar as we have developed a working model of the
game that can be played wirelessly via our prototype sensor and
control box.
We have demonstrated that audio-only games with an
immersive gaming experience that is ready to be implemented
on mobile devices that only have a very limited amount of
screen space, and that the process of prototyping applications for
mobile devices can be accelerated by using off-the-shelf tools.
-3-
In addition, the absence of a graphical user interface also makes
the game accessible to the visually impaired.
We will continue to develop our game to include improved
inventory-checking controls and more cohesive sets of sounds.
We will develop our wireless sensor algorithms to give a more
accurate reading, leading to improved game navigation.
The corridor-based game stage lends itself well to being played
on an actual train or tram platform, in which the user might be
able to synchronize the end of the game with the arrival of the
real life tram or train. Also, the technology demonstrated here
may fit into interactive installations such as those found in
museums or exhibits.
The sound output is optimized for stereo headphones, as they are
a ubiquitous audio playback device for mobile devices. When
being played on a personal computer, surround output would
deepen the immersion in the game.
The prototype of our game hardware could also work as a
controller for a wide range of different virtual instruments or
basically all audio applications that can receive OSC messages.
These applications could even be used as a sound source for the
game.
References
[1] Cahill, Conor, The Interactive Audio Game Project,
http://www.audiogame.com [retrieved on 27.05.08] (1997)
[2] Strahan, Robert, Location Sensing Technologies.
University College Dublin, Report No. e=mC2.2.1.1.2002,
(2002)
[3] Röber, N., and Masuch, M., Playing Audio-only Games: A
Compendium of Interacting With Virtual, Auditory Worlds.
DIGRA Conference 2005, (2005)
[4] Donker, H., Klante, P., and Gorny, P. The Design of
Auditory User Interfaces for Blind Users. Proc. NordCHI 2002,
(2002)
[5] Hobbs, Scott. http://pushyourdesign.com/Scott/
exectechnology.html [retrieved on 25.05.08]
[6] Paul, Leonard. Audio Prototyping with Pure Data
http://www.gamasutra.com/resource_guide/20030528/paul_01.s
html [retrieved on 26.05.08] (2003)
[7] Scarlett, Jim. Enhancing the Performance of Pedometers
Using a Single Accelerometer, http://www.analog.com/library/
analogdialogue/archives/41-03/pedometer.html [retrieved on
26.05.08] (2008)
[8] Wright, M. Freed, A., and Momeni, A., Open Sound
Control: State of the Art 2003, International Conference on New
Interfaces for Musical Expression, Montreal, pp. 153-159 (2003)
[9] Schneider, P. L., Crouter, S. E.; Lukajic, O., and Bassett,
D. R. Jr, Accuracy and Reliability of 10 Pedometers for
Measuring Steps over a 400-m Walk, Medince & Science in
Sports & Exercise, Vol. 35, No. 10, pp. 1779-1784, (2003)
137