NAFIPS 2005 - 2005 Annual Meeting of the North American Fuzzy Information Processing Society

Fuzzy logic in the arts: applications in audiovisual

composition and sound synthesis
Rodrigo F. Cadiz
Music Technology Programn, School of Music
Northwestern University
Evanston, Illinois, USA
rcadiz@northwestern.edu

Abstract-Nowadays, fuzzy logic applications can be found to that soundscape? Given a certain sequence of images, what
in a variety of fields, specially engineering and scientific areas. soundscape is appropriate to it?
However, applications of fuzzy logic in the artistic fields are A fuzzy logic approach to the challenge of composing both
not abundant. This is highly surprising if we consider the
fact that other related techniques such as neural networks and sound and moving image within a coherent framework was
genetic algorithms have been widely used for artistic and creative developed by the author [3]. This approach is based on a fuzzy
purposes. Two applications of fuzzy logic in the artistic domain logic model that enables a flexible mapping of either aural
are presented here: a fuzzy logic-based mapping strategy for or visual information into the other. This model is inspired
audiovisual composition and a novel audio synthesis technique by synaesthetic experiences [8] [18] [13] [1], in which a
based on sound particles and fuzzy logic.
stimulus in one sensory modality triggers a perception in a
I. INTRODUCTION second modality. In the proposed model, both aural and visual
Fuzzy logic systems have been widely used in a variety of information are parameterized and classified into several fuzzy
fields, most prominently engineering and control applications sets. These sets are used as either inputs or outputs to a system
[15] [14], but they have also been applied to other areas as di- that uses 26 fuzzy rules to control the behavior of the mapper.
verse as economics, business and finance [28], sociology [ 1] Once the rules are computed the variables are defuzzyfied and
and geology [9]. However, fuzzy logic has seldom been used converted into the corresponding output parameters.
in the artistic and creative fields, which is highly surprising, The described fuzzy logic model was used to compose ID-
given that other related theories and techniques such as neural FUSIONES, an audiovisual work created the author and visual
networks and genetic algorithms have been widely used for artist Luz Maria Cury in the period May 2000-March 2001.
artistic purposes. This work is completely constructed on the basis of successive
In the specific case of music, despite the fact that in his events in time and each one of these events has two instances,
2001 article Landy included fuzzy logic as one of the potential one in the aural space and another one in the visual. Each
areas for tomorrow's music world [16], only a few applications event is unique and has eight variables (four aural and four
related to creative activities, such as musical composition and visual) that define it completely. The proposed model is able
sound synthesis, have been reported in the literature. Several to translate these parameters from one space (visual or aural)
digital audio processing applications based on fuzzy logic have into the other.
been proposed [10] [6] [2] [19], but they are not directly In this piece, the aural parameters were considered inputs
related to the creative manipulation and transformation of and the visual outputs. This could vary with different applica-
digital sound. Monti and Sandler [20] developed a fuzzy-based tions of the fuzzy model. The relationship could be inverted
blackboard system that transforms audio into MIDI format (visual as input and aural as output) or even a mixed approach
and Elsea [12] applied concepts of fuzzy logic to problems in could be used (some visual and aural as inputs, some visual
music analysis and composition. and aural as outputs).
The author, a composer and electrical engineer, has been
A. Parameters andfuzzy sets
working for some years in the application of fuzzy logic in
two different areas related to the arts: audiovisual composition At this point it is important to mention that only eight
[3] and sound synthesis [4]. parameters were chosen for simplicity reasons. This might
seem a contradiction, as fuzzy logic is one of the better tools to
II. A FUZZY MAPPER FOR AUDIO-VISUAL COMPOSITION avoid simplicity. In theory, the proposed model could take any
Recent technological developments have enabled us to syn- number of input and output parameters with no limitation other
thesize images and sounds concurrently within single com- than computational power. However, only eight parameters
puters, even in real-time, giving birth to new and genuinely were used in order to make the compositional process simple.
integrated audio-visual art forms. But, how should we organize These parameters also coincide with those identified to be
and compose such works? In other words, given a certain perceptually relevant in the literature. Seashore mentions that
soundscape, what would be an appropriate sequence of images "...sound waves have four, and only four, characteristics;

0-7803-91 87-X/05/$20.00 ©2005 IEEE. 551

namely, frequency, amplitude, duration and form. Sounds of
every conceivable sort, from pure tone to the roughest noise,
can be recorded and described in terms of these four" [22].
By the term "form", Seashore is referring to timbre. Lipscomb
found pitch, loudness and timbre to be perceptually relevant in
the aural domain; and location, shape and color in the visual
domain [17].
.....
(a) Frequency
......

(b) Intensity

TABLE I
VISUAL AND AIURAL PARAMETERS

Aural Visual
Frequency Color // / //,
Intensity Shape
Duration Size t.
/
\/.c + \/+ f

Timbre Motion

(c) Duration (d) Timbre (spectrum)

Once the eight parameters were defined (shown in table I),
it was necessary to classify them into fuzzy sets. The majority
of the variables were classified into very low (VL), low (L),
medium (M), high (H) and very high (VH) regions. Although Fig. 1. Membership functions of the aural parameters
some of the parameters were fuzzyfied taking into considera-
tion some of their perceptual characteristics, it is important to
emphasize that this process could have been totally arbitrary.
The proposed fuzzy model is mainly a creative tool and it was
not designed to reproduce perceptual or physical phenomena. A A
Figure 1 shows the membership functions for the aural
parameters. As it can be seen in Figure l(a), frequency is
classified into five different fuzzy sets. Triangular functions (a) Color (b) Shape
were used for each set. Note the uneven distribution of the size
of the various sets, which resembles a logarithmic function.
This was done to reflect the actual perception of pitch by the
auditory system. Figure l(b) shows the membership functions
for intensity. As with frequency, triangular shapes were used
and the uneven size distribution was done to reflect the percep-
tion of loudness (in dB). Figure l(c) shows the membership
functions for duration. In this case, triangular functions were z,':f_i_-
7---- Kz _
used but evenly spaced. A time scale from 0 to 10 seconds was
X R

used. Events lasting more than 10 seconds were considered

very long (VL). Figure l(d) shows the membership functions (C) Size (d) Motion
for timbre. Sounds were classified into noisy or not-noisy.
Note that this is really a crisp classification, YES or NO, and
not fuzzy. This is another advantage of the fuzzy approach,
because it allows bivalent and multivalent variables equally. Fig. 2. Membership functions of the visual parameters

Figure 2 shows the membership functions for the visual B. Fuzzy rules
parameters. Figure 2(a) shows the membership functions for
color. Gaussian functions were used in order to emulate the The relationships between the different variables were reg-
actual frequency distribution of the visible spectrum. Figure ulated by means of fuzzy rules of decision, that controlled the
2(b) shows the membership functions for shape. Evenly spaced behavior of the whole mapper.
triangular functions were considered. Figure 2(c) shows the A rule of decision of this type could be:
membership functions for size. Only three fuzzy sets were If the frequency is very low then the color is very
used in this case, classifying size into small (S), medium (M) low, the shape is somewhat complex, then the size is
and large (L). Figure 2(d) shows the membership functions for big and the motion is medium.
motion. Evenly spaced Gaussian curves were used. or:

552
 If the frequency is very low and the amplitude is
high or the sound is noisy then the color is gray,
the shape is somewhat simple, then the size is big
and the motion does not really matter
C. Limitations and future work
The presented implementation does not work in real time
and it can only generate non-interactive mappings. A very
important direction for future work in this area would be to
extend this system to a real-time based framework. This would
allow, for instance, generation of visual images on the fly in
reaction to live music or vice-versa.
Another possible direction for future research would be to A. Parameters of the sound particles
develop a neuro-fuzzy mapping model. Such a system could
be able to learn how to react to a given unknown input, erties:
either aural or visual, and generate a mapping based on the
knowledge acquired on past experiences.
More details about this mapping approach can be found at
http://www.rodrigocadiz.com/idfusiones.
III. A PARTICLE-BASED FUZZY LOGIC APPROACH TO
simple way, without the enormous amount of work that would
be required to solve the analytical differential equations for a
dynamical system. These trajectories are affected and governed
by fuzzy behavioral rules, which specify the interaction of
particles and the effect of external forces to the system.
Parameters that describe the state of each particle on any
given time are fuzzyfied and used as input to a fuzzy logic
control system that affects the future state of the particles.
This technique enables the specification of highly complex
and non-linear trajectories, which in turn, result in complex
synthesized sounds.
In our model, each sound particle has the following prop-
. Frequency
. Intensity (amplitude)
. Charge
. Position in space (x,y,z coordinates)
. External influence factor

SOUND SYNTHESIS
In other words, each particle possess a fundamental fre-
quency, an intensity (or energy) and a position in 3D space at
Although sound is considered to be a wave, certain scientists any given instant in time. The particles are also considered to
and composers have considered the idea that sound could be charged, like their atomic counterparts, but in a fuzzy way,
also be represented and synthesized by dynamic particle ranging from a full negative charge of -1 to a totally positive
systems. Particle systems usually consist of large number charge of +1. Also, each particle is influenced by external
of point masses (particles) moving under the influence of forces, such as the center of charges of the system. All these
external forces such as gravity, vortex fields and collisions with properties are in fact fuzzy variables, which means that each
stationary obstacles [27]. The basic entities in these kinds of one of them consists on several fuzzy sets or membership
systems differ from physical "particles" in that they already functions (usually three or five) and values of the variable
have an intermediate complexity themselves. can have partial membership to them. Table II displays all
Several kinds of particle-based sound synthesis algorithms the inputs and outputs used in the proposed fuzzy inference
have been proposed in the literature [21] [25] [7] [5]. In system. Note that time is also included as an input, this
general, Newtonian or stochastic equations govern the motion allows the possibility to generate time-dependent behaviors
and collision of the point masses that produce the sounds and or trajectories.
it is necessary to solve all the analytical differential equations
and calculate the state for each particle before rendering a TABLE II
INPUT AND OUTPUT VARIABLES
sample of sound. Such a process is computationally very
expensive. Also, because analytical equations are a requisite,
only certain types of dynamical systems can be considered, Input variables Output variables
Frequency Frequency
those that can be modeled by equations. However, the behavior Charge Charge
of complex and chaotic dynamical systems sometimes cannot Intensity Intensity
be described in terms of closed-form equations. Indeed, most x x
Y Y
non-linear systems are impossible to solve analytically [23]. z z
Almost all the approaches that make use of particle-based Ext. influence
Time _I
systems to create sound require musicians to have at least
a basic understanding of physical laws, if not an advance
knowledge. Sturm even proposes the concept of the "com- It is important to notice that the fuzzy variables frequency,
poserscientist" [24] [25] [26], a new kind of professional on charge, intensity, X, Y, and Z are used both as inputs and
whom the distinction between composer and scientist blurs. outputs to the system. In other words, the system calculates
Making music and doing physics becomes the same thing, the new properties of each particle given the old ones, the
because exploring a musical idea means basically constructing influence of external factors and the appropriate time instant.
a system and solving its equations of motion. Figure 3(a) shows the fuzzy variable frequency, which
The author proposes a novel approach: by means of a corresponds to the normalized frequency of each particle at
fuzzy control system, it is possible to specify highly complex, any instant in time. In this case, five membership functions
nonlinear deterministic sound particle trajectories in a very were used, dividing the universe of discourse (ranging from

553
 F._=, .
ence rules, given in the form of simple linguistic statements.
Each particle was mapped to a sinusoidal wavetable digital
oscillator. All the particles had the same initial frequencies,
intensities and spatial positions, but random initial charges.
Some of the fuzzy rules used in this case were:
(a) Frequency (b) Charge * If frequency is HIGH and time is MEDIUM then
frequency is VERY HIGH
. If intensity is VERY LOW then frequency is
LOW, charge is HIGH, and x,y, and z are far to
the left
(c) Ineqensiy -5
. If charge is LOW then charge is HIGH
\,,F0,,l
,\ /...
w\
. If frequency is VERY HIGH then x,y, and z are
c In i' ty ..
CLOSE to the right
. If charge is HIGH and time is SHORT then
frequency is VERY HIGH
. If external influence is HIGH then charge is
(d) Space position
LOW and intensity is LOW
. If charge is HIGH and z is far to the right then
frequency is LOW and charge is MEDIUM
Figures 4(a), 4(b), and 4(c) show the frequency, charge
..
_

a.v.fe. EP
>,
..
and intensity trajectories respectively. It is possible to appre-
ciate that the trajectories are quite complex, and in general
..

... .
.. .. ..

* i .>D.
)-
(e) External influence
Fig. 3.
0 to 1) of into five regions, labeled "VERY LOW", "LOW",
"MEDIUM", "HIGH" and "VERY HIGH". Each value of the
variable frequency belongs in a particular degree to each one
of the five membership functions. For example, afrequency of
0.5 belongs a 100% to the "MEDIUM" region and afrequency
of 0.6 belongs approximately 40% to the "MEDIUM" region
and 60% to the "HIGH" region.
Similarly, Figure 3(b) shows the membership functions of
the variable charge and Figure 3(c) displays the fuzzy variable
intensity, both defined in similar ways as frequency.
Figure 3(d) shows the general spatial fuzzy variable, used
for the variables X, Y and Z, which is divided into five regions
labeled "FAR-" (far to the left), "CLOSE-" (close to the left),
"VERY CLOSE", "CLOSE+" (close to the right), and "FAR+"
(far to the right).
The normalized variable EI (external influence), shown in
Figure 3(e) contains three membership functions and figure
3(f) contains the additional input variable time, which is
normalized.
The proposed synthesis technique was used to generate 15
seconds of audio. 36 fuzzy rules were used. The trajectories
of the particles were then determined by these fuzzy infer-
the particles behave radically differently in relation to time.
;, Sometimes they behave very chaotically and others (specially
between seconds I and 6) they follow smooth and well-defined
(f) Time
trajectories. Also, they appear to be clustered into two different
groups.
Figures 4(d), 4(e), and 4(f) show the spatial trajectories in
the x, y, and z axes, respectively. The trajectories in X and Z
Membership functions are the same, due to a symmetry present in the rules, but the
trajectory in Y is different, although follows the same kind of
pattern. Once again, the particles seem to be divided in two
clusters of trajectories.
Overall, the different trajectories shown in the above figures
are very complex and sometimes chaotic. This complexity
emerges as a result of the parallel computation of the 36 simple
fuzzy inference rules used in this case. This is an example
of the power of fuzzy logic: with simple linguistic if-then
like rules and the proper fuzzyfication of the parameters of
interest, with no knowledge at all of the mathematical nature
of dynamical systems, it is possible to generate very complex
behavior.
This approach to sound synthesis had its origin in the
creative necessity of composers for having new sounds at their
disposal. The sounds that current synthesis techniques produce
are widely known in the musical community. But in order to
obtain different and fresh kinds of sounds, it is necessary to
look for answers that go beyond the field and techniques of
traditional music synthesis algorithms. In this particular case,
the proposed approach requires also the application of ideas
that come from disciplines far away from music, such as fuzzy
logic.
More details about this novel synthesis technique can be
found at http: //www. rodrigocadiz . com/f Sound.

554
 communities, so that each one could benefit from the other.
V. ACKNOWLEDGMENTS
II Thanks to Luz Marfa Cury for all her help and support. ID-
FUSIONES was partially funded by a grant for artistic creation
and research of the Andes Foundation, Santiago, Chile. Thanks
* b1 also to Dr. Gary Kendall and Dr. Scott Lipscomb for their
valuable comments and suggestions. The sound synthesis
research was partially funded by a Graduate Research Grant
(a) Frequency (b) Charges from the Graduate School of Northwestern University.
REFERENCES
[1] G. Berman. Synesthesia and the arts. Leonardo, 32:15-22, 1999.
[21 C. Breining. A robust fuzzy logic-based step-gain control for adaptive
filters in acoustic echo cancellation. IEEE Transactions on Speech and
i Audio Processing, 9(2), February 2001.
[3] R. Cadiz. A fuzzy logic model for compositional approaches to
audiovisual media. In Proceedings of the International Computer Music
Conference, Miami, FL, 2004.
[4] R. Cadiz and G. S. Kendall. A particle-based fuzzy logic approach to
-!, 1
sound synthesis. In Proceedings of the Conference on Interdisciplinary
-Intensit
Musicology, Montreal, Canada, 2005.
[5] N. Castagne and C. Cadoz. Creating music by means of "physical
(c) Intensities (d) X thinking":the musician oriented genesis enviroment. In 5th international
Conference on Digital Audio Effects (Dafx-02), pages 169-174, Ham-
burg, Germany, 2002.
[6] M. R. Civanlar. Digital restoration using fuzzy sets. IEEE Transactions
on Acoustics, Speech and Signal Processing, ASSF-34(4), August 1986.
[7] P. R. Cook. Real sound synthesis for interactive applications. A K
Peters, Natick Mass, 2002.
[8] R. E. Cytowic. Synaesthesia: Phenomenology and neuropsychology -
a review of current knowledge. In S. Baron-Cohen and J. E. Harrison,
editors, Synaesthesia. Classic and Contemporary Readings. Blackwell,
Cambridge, Massachusetts, 1997.
[9] R. Demicco and G. J. Klir. Fuzzy Logic in Geology. Elsevier Academic
Press, Amsterdam, Boston, 2004.
[10] P. Demichelis, R. D. Mori, P. Laface, and M. O'Kane. Computer
recognition of plosive sounds using contextual information. IEEE
Transactions on Acoustics, Speech and Signal Processing, ASSP-31(2),
(e) Y (f) April 1983.
Z
[11] V. Dimitrov and B. Hodge. Social Fuzziology. Stud), of Fuzziness of
Social Complexity. Physica-Verlag, Heidelberg,New York, 2002.
[12] P. Elsea. Fuzzy logic and musical decisions. http://arts.ucsc.
edu/EMS/Music/research/FuzzyLogicTutor/FuzzyTut.
html, 1995.
Fig. 4. Trajectories [13] P. Hertz. Synesthetic art - an imaginary number? Leonardo, 32:399-404,
1999.
[14] G. J. Klir and B. Yuan. Fuzzy sets and fuzzy logic: theory and
IV. CONCLUSIONS applications. Prentice Hall PTR, Upper Saddle River NJ, 1995.
[15] B. Kosko. Fuzzy Thinking. The new science offuzzy logic. Hyperion,
Two successful applications of fuzzy logic in the artistic New York, 1993.
fields have been described and discussed. Fuzzy logic provides
[16] L. Landy. From algorithmic jukeboxes to zero-time synthesis: a potential
a-z ofmusic in tomorrow's world (a conference provocation). Organised
the artist with a very intuitive and powerful way to specify Sound, 6(2):91-96, 2001.
behavior. [17]S. D. Lipscomb. Cognition of Musical and Visual Accent Structure
Alignment in Film and Animation. PhD thesis, University of California,
One of the advantages of using fuzzy logic in a creative Los Angeles, 1995.
field is that the rules do not have to make sense. The transfer [18] L. E. Marks. On colored hearing synaesthesia: Cross-modal transla-
of expert human knowledge into the fuzzy inference engine is tions of sensory dimensions. In S. Baron-Cohen andJ. E. Harrison,
editors, Synaesthesia. Classic and Contemporary Readings. Blackwell,
not a it is in control and scientific applications. Any
goal, as Cambridge, Massachusetts, 1997.
rule could be used in the system and this gives the artist a great [19] Z. Meng, K. Sakagami, M. Morimoto, G. Bi, and K. C. Alex. Extending
freedom. Stability also doesn't matter, on the contrary, unstable the sound impulse response of romm using extrapolation. IEEE
Transactions on Speech and Audio Processing, 10(3), March 2002.
behavior could be very welcomed on an artistic setting. [20] G. Monti and M. Sandler. Automatic polyphonic piano note extraction
As final conclusion, fuzzy logic seems to be a
a very using fuzzy logic in a blackboard system. In 5th International Confer-
promising technique to be used in artistic environments. The
ence on Digital Audio Effects, 2002.
[21] C. Roads. Microsound. MIT Press, Cambridge, Mass., 2001.
author hopes that these two applications described here could [22] C. E. Seashore. Psychology of Music. Dover Publications, New York,
help narrow the gap between the fuzzy logic and artistic 1967.

555
[23] S. H. Strogatz. Nonlinear dynamics and chaos: with applications to
physics, biology, chemistry, and engineering. Studies in nonlinearity.
Perseus Pub., Cambridge, Mass., 1994.
[24] B. L. Sturm. Sonification of particle systems via de broglie's hypothesis.
In Proc. of the Int. Community for Auditory Display Conf, Atlanta, GA,
USA, April 2000.
[25] B. L. Sturm. Composing for an ensemble of atoms: the methamorphosis
of scientific experiment into music. Organised Sound, 6(2):131-145,
2001.
[26] B. L. Sturm. Synthesis and algorithmic composition techniques derived
from particle physics. In Proc. of the Eighth Biennial Arts and Tech.
Symposiumn, New London, CT, USA, March 2001.
[27] R. Szeliski and D. Tonnesen. Surface modeling with oriented particle
systems. In Proceedings of the 19th annual conference on Coinputer
graphics and interactive techniques, pages 185-194. ACM Press, 1992.
[28] C. Von Altrock. Fuzz)' Logic and Neurofuzzy Applications in Business
and Finance. Prentice-Hall, Upper Saddle River, New York, 1997.

556