Gesture Overview PDF

Capturing gesture...
Frederic Bevilacqua
Real Time Musical Interactions team
IRCAM
Real-Time Musical Interactions Team
Staff
Frederic Bevilacqua ... researcher (leader)
Norbert Schnell... researcher and developer
Riccardo Borghesi ... researcher and developer
Diemo Schwarz ... researcher and developer
Arshia Cont... researcher and developer
Emmanuel Fléty...engineer
Fabrice Guédy ... researcher
Nicolas Rasamimanana... researcher
PhD Students
Julien Bloit
Tommaso Bianco
Vassilios-Fivos Maniatakos
Baptiste Caramiaux
Associated Researchers
Alain Bonardi ... délégation CNRS
IRCAM- Real Time Musical Interactions

Real Time Musical Interactions Ircam
• Gesture capture and Interfaces for performing arts
‣ augmented instruments
‣ dance
‣ installation
• Gesture Analysis
‣ Musician gestures
‣ gesture/action recognition and following
• Interaction
Interactions Musicales Temps Réel

Gesture Research for
performing arts and media arts
Augmented Musical Instruments

Gesture studies using 3D motion capture

Dance & Technology
accelerometer Wireless Interface
breathing
sensor
flex sensor

Gesture Follower

Movement studies
• Eadweard Muybridge and Étienne-Jules Marey
(1830 - 1904), Chronophotography

3D Motion Capture
cameras
capture space
University of California Irvine

Arizona State University - Art
• Motion Capture
• Bill T. Jones, Trisha Brown
• http://ame.asu.edu/motione

Variations V (1965)
John Cage - Merce Cunningham - Billy Kluver
David Tudor, Malcolm Goldstein, Frederick Lieberman et James Tenny (musiciens)

Robert Moog (antennas theremin)
Stan VanDerBeek (film)
Nam June Paik (images de télévisions altérées)
Billy Klüver (consultant technique)
Max Mathews (electronique), Beverly Emmons (lumière)
Merce Cunningham, Carolyn Brown, Barbara Lloyd, Sandra Neels, Albert Reid, Peter
Saul et Gus Solomons Jr. (danseurs)
http://www.artmuseum.net/w2vr/archives/Kluver/03_Variations.html
http://www.medienkunstnetz.de/works/variations-v/video/1/ Interactions Musicales Temps Réel
Alternative Interfaces
« The Hands », Michel Waisviz Le Méta-Instrument - Serge de Laubier

« augmented string instruments»
HyperCello
Tod Machover / Yo-Yo Ma
1991
Hybride…
Violon MIDI de Suguru

Goto
Gesture research and applications
‣ Sciences: Biomechanics, Psychology, Linguistics,
Neurosciences, ...
‣ Computer Graphics
‣ Human Computer Interaction
• Enactive interfaces (perception-action interaction)
• Embodied Interaction
• Tangible User Interfaces
‣ Augmented reality
‣ Ubiquitous computing
‣ Affective Computing number of reference
keyword = gesture
2000
‣ Game 1500
1000
‣ ... 500
0
1969-89 1990-95 1996-99 2000-04
Da Fact
• www.dafact.com

• Gestures ??????

Movement
• Body movements
‣ Tracking body parts, accelerations
‣ Silhouette features: stability, symmetry
‣ Trajectory characteristics: directness, …

Space
• Interactions based on spatial properties

‣ Absence/presence
‣ Movements in defined area,
‣ Crossing borders
‣ Proximity

Body
• Biosignals
‣ Breathing
‣ Muscle (EMG)
‣ etc
• Touch (e.g. conductivity)

Sounds
• Sound Descriptors
‣ pitch, energy profiles
‣ spectral parameters
‣ etc

Motion Capture
sensor attached on 3D optical motion capture hybrid system

the bow
• F. Bevilacqua, N. Rasamimanana, E. Fléty, S. Lemouton, F. Baschet « The • E. Schoonderwaldt, N. Rasamimanana, F. Bevilacqua « Combining
augmented violin project: research, composition and performance accelerometer and video camera: Reconstruction of bow velocity
report » NIME 06 profiles », NIME 2006

Recent Ircamʼs projects with
« augmented strings instruments »
2005
Florence Baschet, Bogenlied, for (augmented) violin and electronics
Anne Mercier: violon, Serge Lemouton: computer music
2006
Concert « Cursus » Ircam, Oct 23 2006,
Sebastian Rivas (comp.), Pierre Strauch (cello)
 José Miguel Fernandez (comp.), Michel Michalakakos (viola)
2007
Philippe Manoury, Partita 1, for (augmented) viola and electronics
Christophe Desjardins: viola,
Hector Parra, Tentatives de réalité, for (augmented) cello and electronics,
Pierre Strauch: cello
2008
Florence Baschet, StreicherKreis, augmented quartet
Quatuor Danel

Prototype I
2004 - Proof of principle
Finger pressure
Accelerometres x y z

Prototype II
 3 accelerometers
 removable module
 wireless transmission 500 Hz, 16 bit
 interface: Ethersense (OSC)

Bow Pressure sensor
• Prototype Matthias Demoucron

Prototype III

Wireless Interfaces
 Custom XBee digitizer 6
inputs – 10 bits – 5ms Receiver
 Ethernet receiver/base
unit microchip IP stack Interface
 OSC compliant data
 Li-Po battery powered 3
to 7 hour autonomy
 Battery level monitoring

• basic features
• demo bowing style

Florence Baschet - Geste Instrumental - Ircam 7 Oct. 2004
accelx
sound

Instrumental/musical gesture
Symbolic/ articulations, bowing styles

(Semantic)
?
Gesture
?
Signal Sound

Interpretation

Caractérisation de modes de jeux
Martelé
amax amin
[a.u.]
amin
Spiccato
amax
amax
[a.u.]
amin
2 violonistes, 2 tempi (60 bpm, 120 bpm)
Détaché
a
3 nuances (pp, mf, ff)
max
amin
• reconnaissance de modes jeux (temps réel)
• BogenLied, Florence Baschet
N. Rasamimanana et al., GW 2005, Lecture Notes in Artificial Intelligence 3881, pp. 145–155,
Interactions 2006.
Musicales Temps Réel
From gesture analysis data to musical material
martelé
spiccato
détaché
détaché
BogenLied for augmented violin Florence Baschet

Florence Baschet Bogenlied
Anne Mercier: Augmented Violon

Serge Lemouton: computer music design
• Première Festival Whynote Nov. 26 05, Dijon

• Maison de la Radio, Jan 7. 2006, Paris

BogenLied Form
acoustic electro acoustic
instrumental
I II III IV V VI VII VIII IX
détaché martelé spiccato
material o
o
all three
o bowing hybrids
styles
G A E
gesture bowing intensity bowing intensity
analysis
gesture
recognition
mapping
granular
synthesis of density of transposition envelope of
violin sound synthesis of synthesis synthesis
in realtime grains grains grains
Transition gesture
waveform
acceleration
time
• The end of a bow stroke conditions the execution of

the following.

Coarticulation gestuelle
détaché détaché martelé martelé
pourcentag
Impulse
Jerk (cyclique)
Jerk (discret)
Figure 7: Cell with mixed bowing techniques: percentages of each model for
the four strokes of a cell, i.e. two Détaché and two Martelé.
StreicherKreis - Florence Baschet
for augmented string quartet
Première in 2008 with Quatuor Danel
Computer Music Design: Serge Lemouton
Commissioned by IRCAM-Centre Pompidou, 2008

III- Elaboration des
modèles gestuels
scéances de février et mars
8 familles de
modes de jeu distincts
correspondant à
8 gestes d’archet
Pour violon
Quatuor Danel
Février 2007
Capteur gyromètre-1 et 2
Capteur accélérationX-Y-Z
Capteur de pression
Florence Baschet

gettato

m1-2-4.mp3
spiccato
Violon 1
m2-2-3.mp3
Violon 2

Questions....
• differences between gestures of violin, viola, cello

players ?
• differences between two violin players ?
• how do gestures vary during the learning process ?
• gestures - sound relationships ?

Gestes =
mouvement (5)
pression
28 paramètres
energie audio

! ". !"
n œœ . œœ# œœ" nœ n œ# . n œ"
&c Ó J J ‰ Œ Œ mœ 42 m œ . mœ 38
p f p f J
p !
n m œœ n m œœ# .. Gliss. µ n œœ µ n %%œ"œ .. n n œœ n n œœ ..
#
œœ œ o"
&c J J J ‰ Œ
œ 24 O œ . œ œ 38
Glissando
f F p P
œo .
#
%
n œoo# .. n m ˙o˙o m n œooœ
"
œ n m %% n m œœ# ..
&c $OJ J ‰ Œ $ On œJ œ
J ‰ Œ OO 42 OO 38
O p

p
# #
I
f b
f p I
II f
flautando II
flautando
III
21-vlc
C IRISÉ sectionC, ligne :P --> freq FS et G1--> gain+ FS

reconaissance/interprétation dans les 8 irisés
! !" !
5
! #
n œœ .. nUœ m n œœ n œ ..#
! m n œœ" .. m n œœ#
q = 45
œœ mœ ˙ œ
28 m œR ® ‰ 43 m n œœ .. m n œœ" ..
flautando
& 38 42
$
#
c& c n œœJ J m œ# m œ"
‰ ‰ ‰
J ‰ Œ Œ ‰ Œ
23
˙ œ
27 pont.
J J nœ œ
f p p f p f p f p sub. f p J
! !
! n% n œœ# .. flautando
q = 45
!# œ" m n œœ ..
#
m n œœ ..
"
m n œœ .. Glissan m m œœ m m œœ ..
#
U . mœ.
pont.
3 m
‰ 43 $ mn œœ . m œ"
42 Ó 28 ® R
do
"
& 8 m% ‰ ‰ nœ. nœ c& c ‰ œ ‰ m n œœ m n œœ Œ J J $ Œ
J p f J J p f p f p p
p p f
! ! œ
pont.
n œ# œœ
q = 45
œ œ mœ œ m n œœ
"
! œ détaché
µ n œ#œ ..
". U 28 $ m œR mœ. mœ œ
flautando
& 38 $ œœJ. œœ
J 42 Ó RÔ ®$ 43 Ó. c& c mœ
J ‰ Ó etc... J ‰ ®
mœ œ œ $
Ó ‰
p p f p p p p
o o oo U
n œo
3
détaché !
m n œœ ..
#
m n œœ Gliss. n m œœ n œ# œ m n œœ m n œœ#
flautando q = 45 flautando
œ
& 38 J 42 Œ 28 $ ®m œRÔ œR $ 43 Œ Œ ‰ J c& c ‰ Œ Ó. ‰
pont.
œ œ
œ
œ
œ ‰
nœ œ œ.
p f p f %
p p
24-sectionC 25 26-vlc 27-vl1 28-irise1 29-vla
p m œo o m œo m œo
(a) score détaché
µ n œœ# Gliss. µ n œœ
ord.
n m œœ" n m œœGlissando œ
Œ Ó Œ 43 J J ‰ Œ ® 81
30
&
p f p s. ( œ
m œœ" Glis œ )
#
!"
ord.
œ
m œ#
3 m n œ#œ J 81
m n ˙˙
pressure acceleration (x) pont.audio energy
& Ó 4 Œ Œ Œ ‰ ‰
p f p f
oo" flautando p
m œœ . m œo œo œo œo
3
.
détaché
µ n œœ µ n œœGlissn m œœ" n m ˙˙ n m œœ
ord. .
& 43 J ‰ Œ OO $ J Œ Œ ‰ 81
f p p f p
n n œoœo ..
II
ord.
"
œœ# œœ
III
& & 43 Ó. J 81
p f p
II
III
31-vl1 32-vlc-irise2
! U
œ" U !
's. m ˜ œœ# ..Glissandom n œœ" m n œœ ..Glissaanndo
ord.
m m œœ#
m m œœ $ œœ"
& 81 ‰ 42 43
Gliss
#
Ó.
pont.
Glis
c œ J c
33 do
R
% p f p
! marc. F p
! " flautando
3
m œœ"
flautando
1 m œœ# œœ" U nœ. m m œœ# m m œœ

® c Ó. Œ J 42 Ó c ‰ mœ. ‰ J 43
ord.
&8 m œœ
¨oo f p 3
f p f p
m œœ ># ˙˙ .. œœ" .. n n œœ# œœ n n œœ"
1 ® œœ U
&8 c J $ 42 Ó c Œ ‰ J 43
f p p f
n m ˙o˙o
#
n m '' n m œœ" .. U # " # œ
marc.
"
&
18 ‰ c J $ 42 Ó c Ó $m Rœ œ œ m œJ ‰ 43
p f stridente
p
flautando
34-vl1 36-vl2
Score following vs Gesture Following
score abstraction/
symbols
score following gesture following
gesture data
Olivia Grandville
Comment taire (2004)
Olivia Grandville- choreography, performing
Electroacoustic design Romain Kronenberg - Ircam-Centre Pompidou
Ircam assistant: Nicolas Leroy
Dance movement used for:

• sample triggering
• granular synthesis
• various effects controlled
Interaction
- hand following (facing camera)
- location in space of the dancer (from above
- energy and direction of the silhouette (from above)

Olivia Grandville Comment taire (2004)
Interaction: hands tracking

Olivia Grandville Comment taire (2004)
Interaction: Interaction:
space occupation tracking silhouette
stage divided in zones
1 2
8 3
7 4
6 5
3:57 12:30, 3:05

video capture
Eyesweb.org
University of Genoa - Italy
Double point +
Hanspeter Kyburz, Emio Greco, Pieter C. Scholten (2005)
3:
This Is My House (2005)
Myriam Gourfink, choreography,
Kasper Toeplitz composition


This Is My House -Myriam Gourfink
video

Section 8C

Sensors
Wireless Interface
accelerometer (WiseBox)
breathing
sensor
flex sensor

Analysis
Next section depends on the analysis of what happened
during the previous one
 Cyclicity of each dancer

 Synchronicity between dancers
Breathing, hip, shoulder, elbow, ankle, head.
 Activity (motion energy)
 Phrase recognition

Documentation/Notation project by
the dance company Emio Greco |PC
interactive installation !
publication (book and DVD)

demo lcd egpc

Installation Interactive
• 29-30 juin 2007, Amsterdam
School of the Arts (“preview”)
• 6 octobre 2007
Dutch Dance Days 2007,
Maastricht
• 29 février - 3 mars
1st Biennale der
Tanzausbildung, Berlin

Richard Siegal - If/Then Choreographic System

If/then open source

N E
D
A
C
gesture model
time

Example 3
• Dance performance

Homo Ludens (Richard Siegal - The Bakery)
2:55

If/Then Installed - ZKM Karlsruhe

The Gesture Follower project
• Can a computer recognize a “gesture” ?
• Can a computer guess if the performer is at

beginning, middle or end of a gesture ?
i.e. follow the gesture
• Can a computer tell how the gesture was performed ?

Previous works...
using the follower
Selected works
• Myriam Gourfink
‣ This is My House, 2005
• Alice Daquet
‣ I.D.L, I.D.O., 2006
• Emio Greco | PC
‣ (Capturing Intention), interactive DVD, 2007
‣ Double Mind-Double Skin Installation 2007
• Florence Baschet,
‣ Streicherkreis, augmented quartet, 2008
• Richard Siegal, 2008-
Gesture Follower- Learn
Gesture (live)
Gesture 1
Gesture 2
Learning

Gesture Follower
Play
Gesture (live)
Compare
Gesture 1
Gesture 2

Gesture Follower
Play
Gesture (live)
Compare
Gesture 1
Result
Gesture 2

Gesture Follower
Real-time time warping

performed gesture (live)
data
recorded example
time
• Synchronization
• Recognition
• Anticipation (prediction) Interactions Musicales Temps Réel
Time warping
time warped
references performed gesture
x
acceleration
time

Gesture to sound

Temporel mapping

Music Pedagogy applications

Max/MSP Implementation
• Part of MnM package for
gesture analysis
• Using the FTM library
• Free download at
http://ftm.ircam.fr
• Wiki
http://ftm.ircam.fr/
index.php/
Gesture_Follower
Music Pedagogy applications

Summary -Perspectives
• From triggering electronic events to
‣ continuous control
‣ controlling timing of digital media
‣ recognizing sequences
in progress
• movements “qualities”
• generating new material (audio/video) ?
• building complex scenario ?

Thanks to
• all IMTR team!
• Atelier les Feuillantines,
• Remy Muller, Donald Glowinski, Myriam Gourfink,
Richard Siegal, Hillary Goidell, Jean-Philippe Lambert,
Florent Berenger, Florence Baschet, Serge Lemouton,
Alice Daquet, Anthony Sypniewski,
• We acknowledge partial support of the following
projects:
•EU i-Maestro project (IST-026883),
•EU-ICT Project SAME (Sound And Music for
Everyone Everyday Everywhere Every way)

Gesture Overview PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Gesture Overview PDF

Uploaded by

Copyright:

Available Formats

Capturing gesture...

IRCAM- Real Time Musical Interactions

Interactions Musicales Temps Réel

IRCAM- Real Time Musical Interactions

Gesture studies using 3D motion capture

IRCAM- Real Time Musical Interactions

accelerometer Wireless Interface

IRCAM- Real Time Musical Interactions

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

University of California Irvine

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

David Tudor, Malcolm Goldstein, Frederick Lieberman et James Tenny (musiciens)

« The Hands », Michel Waisviz Le Méta-Instrument - Serge de Laubier

Interactions Musicales Temps Réel

Violon MIDI de Suguru

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

• Interactions based on spatial properties

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

sensor attached on 3D optical motion capture hybrid system

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Symbolic/ articulations, bowing styles

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

BogenLied for augmented violin Florence Baschet

Anne Mercier: Augmented Violon

• Première Festival Whynote Nov. 26 05, Dijon

Interactions Musicales Temps Réel

• The end of a bow stroke conditions the execution of

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

• differences between gestures of violin, viola, cello

• differences between two violin players ?

• how do gestures vary during the learning process ?

• gestures - sound relationships ?

Interactions Musicales Temps Réel

Interactions Musicales Temps Réel

StreicherKreis - Florence Baschet

C IRISÉ sectionC, ligne :P --> freq FS et G1--> gain+ FS

1 m œœ# œœ" U nœ. m m œœ# m m œœ

score following gesture following

Dance movement used for:

Interactions Musicales Temps Réel

Interaction: hands tracking

Interactions Musicales Temps Réel

stage divided in zones