Sound Synthesis

and Processing

Xavier Serra
Music Technology Group
Universitat Pompeu Fabra, Barcelona

Index
Tradition of sound synthesis
Musique Concrte, Granular synthesis, Sampling
Spectral sound processing
Mosaicing, Concatenative synthesis
Gesture-based synthesis

Xavier Serra - 2008

Tradition of sound synthesis

Xavier Serra - 2008

4
Xavier Serra - 2008

Xavier Serra - 2008

Synthesis with physical models

graphical interface for Tbone (Perry Cook)

Xavier Serra - 2008

Digital implementation
of an acoustic system

Xavier Serra - 2008

Musique concrte

A class of electronic art music

produced from editing together
fragments of natural and industrial
sounds.

Pierre Schaeffer
Pierre Schaeffer,
Etude aux chemins de fer (1948)

Xavier Serra - 2008

Granular synthesis

All sound is an integration

of grains, of elementary
sonic particles, of sonic
quanta Xenakis (1971).

Helmuths
example

Xavier Serra - 2008

Sampling

Fairlight (1980)
Pioneered two innovations
that transformed music making,
namely sampling and sequencing.

Xavier Serra - 2008

Spectral processing

Original
Sound

Spectral
Fourier
Analysis
Analysis

Feature
Extraction

Transform.

Original
Feature

Feature
Addition

Spectral
Synthesis

Transformed
Feature

Original
Spectrum

Transformed
Spectrum

Xavier Serra - 2008

Transformed
Sound

Spectral representation
N

x( t ) = A0 + Ak cos( 2f k t + k )
k =1

= A0 + Re Ak e j ( 2f k t +k )
k =1

= A0 + Re Ak e jk e j 2f k t
k =1

X k j 2f k t X k* j 2f k t
= X0 +
e
+
e

2
k =1 2

where

X k = Ak e jk
12

A spectrum is a sequence of pairs of numbers:

{( X ,0), (
0

1
2

X 1 , f 1 ) , ( 12 X 1* , f 1 ), ( 12 X 2 , f 2 ) , ( 12 X 2* , f 2 ), }

The spectrum of: x( t ) = 10 + 14 cos 200t + 8 cos 500t +

3
2

13

Spectral analysis:

14

Spectral models

Short-Time Fourier Transform (STFT)

Sinusoidal Model
Sinusoidal plus Residual Model
Higher Level Attributes

15

Spectral analysis
window
F0
generation
magnitude
frequency
smoothing
spectrum
sine frequencies
window
sound
peak
pitch
peak
sine magnitudes
*
FFT
detection
detection
continuation
peak
sine phases
phase
peak
data
spectrum
data
additive
synthesis
sinusoidal
- component
smoothing residual
window window * component
generation
amplitude
correction
FFT

phase
magnitude
spectrum
spectrum
spectral
residual
approximation
spectral data

16

Feature analysis
inflexion
points
sine data
spectral data

attack,
steady-state,
and decay
detection

spectral
shape
spectral shape
extraction

amplitude normalization
and
amplitude envelope
extraction
amplitude and
amplitude
envelope

vibrato
data
amplitude
and
frequency
detrending

frequency normalization
and
pitch envelope
extraction
pitch and
pitch envelope

vibrato
extraction

loop points
selection

residual
sine data
residual
spectral data

loop points

17

Spectral synthesis
sine
frequencies
sine
magnitudes
sine
phases

magnitude
spectrum
spectral
sine
generation

polar to
rectangular
conversion

phase
spectrum

spectral data
(ex: reverberation)

spectral data

spectral
residual
generation

window
generation

complex
spectrum

synthesis
window
IFFT

sound

magnitude
spectrum

polar to
rectangular
conversion

phase
spectrum
18

Spectral transformations
Original Sound
Transformation

desired timbre envelope

Amp

fi-1

Xavier Serra - 2008

fi

fi+1

fi+2

Sound collage

John Oswald
A phonograph in the hands of a hip
hop/scratch artist produces sounds which
are unique and not reproduced - the record
player becomes a musical instrument. A
sampler is simultaneously a documenting
device and a creative device, in effect
reducing a distinction manifested by
copyright.

no official distributor/not for sale 1989

Xavier Serra - 2008

Photo mosaicing

Example of photomosaic made out of hundreds of portraits

of Americans who have died at war in Iraq during the last
few years.

Xavier Serra - 2008

Audio mosaicing

Tristan Jehan, 2005

(Schwarz, 2007)

Xavier Serra - 2008

Concatenative synthesis (1 of 2)

PHASE

TIMBRE
Spectral Shape
Interpolation Factor

SPP Spectrum n-i

Frame n-k

Frame n-1

Frame n
Spectral
Shape
Interpolation
Factor
(SSIntp)

Interpolation zone
k frames

Stretch

Stretch

i
n 1

i
n

= 2

f ni+
1 transpn 1

f ni transpn

2
+ 2 ( i + 1) t pitchn ( transpn 1)

tsync

c
=
2 pitchn transpn

Left EpR

Right EpR

differential
envelope

stretched
spectrum

Timbre
Envelope

f
F3 n-1

f
F3 n-1

F2 n-1
F1 n-1

F2 n-1
F1 n-1

F0 n-1

1-SSIntp

f
F0n-1 F1 n-1 F2 n-1 F3 n-1

EpR frequency mapping

F0 n-1

f
F0n-1 F1 n-1 F2 n-1 F3 n-1

EpR frequency mapping

Xavier Serra - 2008

Concatenative synthesis (2 of 2)
A
possible sounds produced by
the instrument
B
sounds produced by the performer
playing the instrument

Vocaloid example

recorded audio samples

Instrument sonic space

Performance
Score

performer
model

Synthesizer diagram

performance
trajectory
generator

performance
DB

Xavier Serra - 2008

Performance
trajectory

sound
rendering

Sound

Synthesis based on gestures

gestures generated
by the system
Xavier Serra - 2008

Violin gesture modeling