See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/298979427

Automatic Musical Fountain Scenario Generation Using Musical Information

Analysis

Article · January 2009

CITATION READS

1 369

2 authors:

Min-Joon Yoo In-Kwon Lee

Yonsei University Yonsei University
13 PUBLICATIONS   42 CITATIONS    99 PUBLICATIONS   886 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by In-Kwon Lee on 10 February 2017.

The user has requested enhancement of the downloaded file.

 AUTOMATIC MUSICAL FOUNTAIN SCENARIO GENERATION
USING MUSICAL INFORMATION ANALYSIS
Min-Joon Yoo
Yonsei University
Department of Computer
Science
ABSTRACT
In this paper, we introduce our project the ‘Intelligent
Musical Fountain Authoring System’, a system that
automatically generates musical fountain scenarios by
analyzing the musical information in accompanying music.
The onset and beat information of the musical piece are
retrieved and its musical structure is analyzed to provide
temporal information through scenario generation. Then
new scenarios can be generated using a Bayesian network
established from sample scenarios. Though the guidance of
musical information in accompanying music, musical
fountain scenarios can be generated much faster than
manual generation.
1. INTRODUCTION
Majestic musical fountains such as the Bellagio Music
Fountain in Las Vegas and the Magic Fountain of Montjuic
in Barcelona are greatly appreciated by spectators. A
profusion of water jets and colored lights give these
fountains a spectacular appearance. However, what
distinguishes them is that their displays are synchronized
with accompanying music.
Musical fountains can produce a romantic or dynamic
atmosphere by changing the shape of water bursts
synchronized with music. Thus, the major difference
between a musical fountain and fountain is that a careful
choreography, a scenario of change in the shape of water
bursts set to music, has to be generated.
Usually the scenario is generated by experts because the
knowledge of water burst shapes created by various
nozzles and a careful analysis of music are required to
make a musical fountain scenario. However, since creating
such a scenario is quite challenging and time-consuming,
the cost of scenario generation is usually very high.
Due to the high cost, many music fountains play a short
and small number of musical pieces repeatedly.
Frequently repeated pieces would reduce the interests of
the spectators and thus lessen the practical utility of the
musical fountain.
Our project the ‘Intelligent Musical Fountain Authoring
System’ aims to automatically generate musical fountain
In-Kwon Lee
Yonsei University
Department of Computer
Science
scenarios by using an example-based approach and musical
information analysis.
A model is built by analyzing musical fountain scenario
samples generated by experts. Then new scenarios are
synthesized from the data in this model through the
guidance of the semantic information of arbitrarily
accompanying music. The information includes several
types of temporal data such as onset, beat and musical
structure boundaries.
There are a few software and hardware component that
the control the water burst shapes automatically according
to the musical contents. However, most previous systems
are limited in that they control the height of the water burst
shapes using only the amount of volume or the frequency
information of the accompanying music. However, in our
system scenarios of higher quality can be generated
because the generated musical fountain scenarios are based
on example scenarios made by experts and are generated
by applying several forms of semantic information of
music.
2. MUSICAL INFORMATION ANALYSIS
The quality of a fountain scenario depends heavily on the
synchronization of the music and the water burst shapes.
Our system achieves synchronization by extracting the
temporal features of the music which are appropriate for
scenario generation. The task of synchronization is
performed by experts and it is still very time consuming.
However, our system can reduce the amount of time
required for scenario generation largely by automating the
synchronization task using various musical information
retrieval techniques. Our system calculates three temporal
features of the music and the timing of changing water
burst shapes is based on these features.
2.1. Onset Detection
Locating an accurate starting time of the onset of a musical
event such as a note is a fundamental operation in any
attempt to identify musical events from audio signals.
Some types of synchronization in musical fountain
scenarios can be obtained by changing water burst shapes
at the time of onset of a musical event. A range of onset
detection algorithms are explained in [1,2]. Among the
several ways to detect onsets, we selected spectral flux,
which can be computed quickly and accurately. Spectral
flux is defined as follows:
∑ | , | | 1, | , (1)
where , is the magnitude of the th frame and th
frequency bin in a frequency domain of the audio signal
and is the half-wave rectifier function.
A peak-picking process is then used to select the exact
onsets, which must fulfil the following condition:
for all s.t. . (2)
The parameter controls how many onsets are selected.
This parameter can be chosen by the user, and is directly
related to the ‘busyness’ of the resulting fountain scenario.
2.2. Beat Tracking
Although some synchronization can be obtained by
changing water burst shapes at onset times, sometimes it is
not enough to use only the onset data. For example, one
may want to change the shape in the middle of a long calm
portion of the music, during which onsets may not be
detected. Thus, our system also uses beat times, in order to
provide more natural time information for scenario
generation.
Beat Tracking is a technique to extract beat data from
audio signals. Generally this type of tracking finds beat
times based on onset data.
Our system uses the beat tracking algorithm presented
by Ellis [3]. In this algorithm, the onset energy is
calculated from audio signals and the tempo is estimated
based on the onsets. Then an object function using onset
energy and tempo is defined and finally a beat sequence
that optimizes the object function is calculated using
dynamic programming.
The onset energy is calculated in the following ways. At
first, the frequency information of the audio signal is
extracted using Fourier Transform. Then, an acoustic
model is built by multiplying the frequency information
with MEL constants. After changing the values into
decibels, the first derivatives are calculated. These values
mean onsets; that is, functions representing increases in
energy output.
Estimations of the tempo with the most frequent time
periods between onsets are calculated using
autocorrelations of the onset energy data. By using the fact
that the average person perceives a tempo near 120 bpm,
the tempos near 120 bmp are weighted.
Now, optimized beat sequences are calculated by
defining the object function of two terms, the onset and the
tempo, and are solved by dynamic programming. Thus,
calculated beat sequences have times that match the times
of high onset energy, and are also similar to the estimated
tempo. One can refer to [3] for a more detailed description
of this algorithm.
2.3. Structure Analysis
The onset and beat data provide the local temporal
information for scenario generation. The system also uses
global temporal information by using structure analysis.
The accompanying music is segmented using a classical
self-similarity matrix and novelty scores [4], and some
changes in scenarios occur at the boundaries of the
segments. By defining the size of the kernel used to
calculate the novelty scores as 4~8 seconds, the structure
boundaries, which may not be found using only onset data,
can be obtained.
Now, we are implementing more elaborate musical
structure analysis techniques to detect the structures more
exactly. For example, we can obtain more accurate
boundaries of musical structures by using Maddage’s
method [5]. In this method, musical structure is analyzed
using chord patterns and singing voice boundaries as well
as onset data.
3. SCENARIO GENERATION
This paper focuses on musical fountain scenario
generation by analyzing musical signals and retrieving
musical information. Thus, we will first briefly explain
our scenario generation method. A more detailed method
will be presented in a regular paper which will be
published soon.
3.1. Constructing the Bayesian network
A Bayesian network is a model of probabilistic
relationships among a set of variables which is frequently
used for encoding uncertain expert knowledge [6]. We
designed a Bayesian network from sample scenarios
generated by experts, and the network can now be used to
generate new musical fountain scenario automatically.
 3.2. Selecting operated nozzles
At each recognized time – onset time, beat time or
segmentation boundary time – the operating probability of
each nozzle is calculated using the Bayesian network. In
the calculation of operating probability, current operating
states of nozzles and volumes of accompany music are
considered. Because the network is based on probabilities,
the generated scenarios can be different with each attempt.
Thus, our system generated various and diverse scenarios
with one piece of music.
4. SIMULATION SYSTEM
Since the cost of testing and verifying generated scenarios
of actual fountains is high, the system provides a 3D
simulation to test scenarios on a computer monitor.
Automatically generated scenarios can be verified
immediately using a particle model to simulate the jets of
water moving under the influence of gravity, and then
render them in 3D. We have implemented our own
particle dynamics engine, and rendering is performed by
GPU-based vertex shading. Figure 1 depicts our system.
In this system the user can generate, edit and verify
musical fountain scenarios.
Figure 1. A screenshot of the ‘Intelligent Musical Fountain
Authoring System’. Left-up : layout of nozzles, Middle-
up : 3D simulation, Right : property setting window,
Bottom-left : Scenario track.
5. CONCLUSION AND FUTURE WORK
We presented our ongoing project, the ‘Intelligent Musical
Fountain Authoring System’. This system can produce
fountain scenarios automatically by combining a Bayesian
model derived from sample scenarios with an analysis of
semantic features of the musical accompaniment.
View publication stats
Though the scenarios of the system may be not
sufficiently satisfying when compared with scenarios
generated through great effort by experts, automatic
generation is much faster and the experts can edit and
verify the automatically generated scenarios easily with the
graphical interface of our system.
Since scenario generation depends on the content
analysis of audio signals, the inclusion of better analysis
techniques should lead to improved scenario generation.
We are implementing more improved analysis techniques
that have been presented in several music information
retrieval journals and conferences.
We are also exploring interactive technique in order to
control fountains. We are planning to create fountain
shows accompanied by music playing in real-time by
implementing real-time techniques such as real-time beat-
tracking. We think that a fountain can be used as a
performance tool with various type of music, including
electronic music.
[To reviewers : This system is now applying to the
musical fountain of Seong Name Art Center in South
Korea and it will be completed in March 2009. We hope
that we can report more interested results in final version
of this paper, if accepted]
6. REFERENCES
[1] Bello, J.P., Daudet, L., Abdallah, S., Duxbury, C.,
Davies, M., and Sandler, M.B. “A Tutorial on Onset
Detection in Music Signals”, IEEE Transactions on
Speech and Audio Processing, Vol. 13(5), 2005, pp.
1035-1047.
[2] Collins, N. “A Comparison of Sound Onset Detection
Algorithms with Emphasis on Psychoacoustically
Motivated Detection Function”, AES 118th
Convention, 2005.
[3] Ellis, D.P.W. “Beat Tracking by Dynamic
Programming”, Journal of New Music Research, Vol.
36(1), 2007, pp.51-60.
[4] Foote, J. “Automatic Audio Segmentation Using a
Measure of Audio Novelty”, in Proceedings of IEEE
International Conference on Multimedia and Expo,
New York, USA, 2000, pp.452-455.
[5] Maddage, N.C. “Automatic Structure Detection for
Popular Music”, IEEE Multimedia, Vol.13(1), 2006,
pp.65-77.
[6] Heckerman, D. “A Tutorial on Learning with
Bayesian networks”, In Learning in Graphical
Models, M.Jordan, ed. MIT Press, Cambridge, MA,
1999.