The Light Matrix: An Interface for musicaI expression

and performance
Jonathan Pak
Electronic musician/Engineer

A prototype oI the Light Matrix interIace is presented here. At
the core oI this device is a rectangular array oI Light Emitting
Diodes (LED). Although designed to emit light. it is also
possible Ior LEDs to Iunction as crude photosensors. The Light
Matrix exploits the bidirectional properties oI the LED to create
a light reIlection based proximity sensor. The perIormer
interacts with the device through hand movements in Iront oI
the LED sensor matrix. The potential oI this device as an
interIace Ior real-time musical expression Ior computer based
audio systems will be discussed along with its suitability as an
instrument Ior live perIormance. Here the aspects oI
perIormativity covered include perIormer-instrument interaction
and audience perception oI the relationship between physical
interaction and resulting sound.
Music controller. LED photosensors. perIormativity. real-time
expression. mapping visualisation. programmable response.
The prototype oI the Light Matrix is a computer interIace
intended Ior electronic music perIormance. The Iace oI the
device consists oI a grid oI LEDs Iunctioning as both a
proximity sensing array and monochrome display. Figure 1
illustrates the process involved in translating hand movements
into musical expression. The intermediate Iorms oI data are
shown at each stage oI the process.
To begin with. each LED sensor measures the intensity oI
reIlected light. light that is emitted by other LEDs in the matrix.
The reIlected light intensity is proportional to the proximity oI
the perIormer`s hand to the plane oI the matrix. The raw data
Irom this device may be represented as a moving height Iield.
This data is relayed to a soItware application running on a
computer which translates Ieatures oI the height Iield into
control messages. In turn these messages may be used to control
audio applications. virtual instruments and eIIects.
The intensities oI each LED pixel are independently adiustable
which gives rise to some unique Ieatures oI this device. The
Light Matrix responds to the perIormer not through tactile
Ieedback seen in traditional instruments |10|. but through
modulating the light intensity pattern oI the LED matrix. This
optical Ieedback is derived Irom three diIIerent sources (see
Figure 1). These sources are: the height Iield data. mappings
deIined by the soItware application and the resultant audio
signal. This optic response modiIies the characteristics oI
physical interaction and provides visual cues to both perIormer
and audience as to how the instrument is operating.
Figure 1. Device concept illustrating multiple stage

Permission to make digital or hard copies oI all or part oI this work Ior
personal or classroom use is granted without Iee provided that copies
are not made or distributed Ior proIit or commercial advantage and that
copies bear this notice and the Iull citation on the Iirst page. To copy
otherwise. or republish. to post on servers or to redistribute to lists.
requires prior speciIic permission and/or a Iee.
. June 4-8. 2006. Paris. France.
Copyright remains with the author(s).
LED matrix
Height field
Audio signal
MIDI control
Sampling and
and mapping
LED light
Proceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06), Paris, France
The work oI J. Han and associates has also utilised the
bidirectional properties oI LEDs in their proiect: Multi-Touch
Sensing through LED Matrix Displays |3|. The title oI the
proiect alludes to a slightly diIIerent Iocus on multiple contact
point detection rather than a quantitative proximity
measurement. The display provides no visual Ieedback and is
not speciIic to musical applications.
The Light Matrix also has parallels with the Tenori-On |4|
musical instrument. currently under development by Toshio
Iwai. The elements common to both devices are the similar
Iorm Iactor and interactive LED display. One interacts with the
Tenori-On by pushing on individual LEDs in the display.
Another comparable device is JazzMutant`s musical controller:
the Lemur |5|. This Ieatures a high resolution pressure sensitive
multi-touch display though it lacks the depth sensing capability
oI the Light Matrix.
The Light Matrix has a tactile counterpart in D. Overholt`s
MATRIX interIace |1|. The limitations oI the MATRIX`s
spring loaded rod assembly are: the substantial Iorce required to
push down the rods and the Iact that adiacent rods prevent
lateral movement oI the perIormer`s hand.
The advent oI the DAW (Digital Audio Workstation) has
aIIorded musicians great scope to explore new sound textures
and improvisatory techniques never beIore possible. Virtual
instruments can be much more complicated than their physical
counterparts with no upper limit on the number oI parameters
controlling expression.
Using standard interIaces such as Iaders. switches. knobs and
keyboards. it is diIIicult to control even a Iew parameters
simultaneously. The consequences oI this limited Iorm oI
control in a live perIormance are a lack oI spontaneity and a
possibly unengaging perIormance.
The Light Matrix is designed to overcome these limitations by
providing a greater level oI control. The prototype has an 8 × 8
grid oI LED sensors each capable oI measuring depth with 8
bits oI precision. As a controller. the size oI the input space
makes it possible to realise many complex mapping schemes to
translate perIormative hand gestures into sound.
The low latency oI the device coupled with simultaneous
control over many parameters makes that the Light Matrix ideal
Ior real-time musical applications in synthesis and DSP (Digital
Signal Processing).
The constant low latency oI the device means that the interIace
is both predictable and responsive to the perIormer`s gestures.
The degree oI simultaneous control gives the artist a broader
palette Ior selI expression through music. Both oI these Iactors
underscore the immediacy oI perIormance Ior the audience.
3.1 Physical interaction
The absence oI tactile Ieedback arguably allows a more direct
Iorm oI expression. Freedom Irom the physicality oI playing a
traditional instrument eliminates physical constraints and
reduces the eIIort required to play the instrument. Any eIIort
required to operate the interIace translates directly into
expressive movement.
The device is capable oI detecting the presence oI any obiect
that is reIlective but in this case the use oI hands as the means
to convey expression is only logical. The capacity oI the human
hand to communicate semantic and emotional content is second
only to Iacial expressions. The inherent expressiveness oI
human hand gestures presents enormous potential Ior conveying
musical expression.
3.2 Optical feedback
The Light Matrix uses a unique Iorm oI Ieedback through light
intensity. A more intense light source will enable the LED
sensors to detect hand positions Iurther away by causing a
stronger reIlection. Conversely a weaker light source will limit
the usable range oI control.
3.2.1 Performer interaction
By continuously updating the intensities oI each LED in
response to the motions oI the perIormer`s hand. the interactive
characteristics oI the device can be changed. Figure 2 illustrates
some oI the basic responses used to alter the parameters oI
physical interaction.

Figure 2. Optical feedback schemes driven by physical
3.2.2 Functional visualisation of mapping
This technique is used to adiust the interIace`s responsiveness
which is best illustrated with an example:
A practical application oI the interIace could be to manipulate
the timbre oI a synthesiser. An illustration oI the LED matrix
Ior this example is shown in Figure 3. The volume oI low
Irequency components oI the synthesiser`s sound is increased
by moving closer to the surIace on the leIt side oI the matrix.
Progressively higher Irequencies are controlled by moving
towards the right side oI the matrix.
OIten the volume oI high Irequencies needs to be tempered in
relation to the low tones so as not to sound shrill. The
realisation oI this mapping should be immediately apparent
Irom the gradient in light intensity. The reduced intensity on the
right corresponding to a reduced emphasis on the higher
Irequencies. making the synthesiser sound less bright. The
Reflected light intensity
Change in
Change in
Proximity driven feedback
Movement driven feedback
Pr oceedi ngs of the 2006 I nter nati onal Confer ence on New I nter faces for Musi cal Expr essi on (NI ME06), Par i s, Fr ance
visual representation oI this mapping is not only Iunctional but
also inIormative Ior the perIormer and the audience.
Figure 3. Example of weighted frequency response through
visible mapping
3.2.3 Auditorv feedback
Auditory Ieedback is also possible with this system by using
sound to control the intensities oI the LEDs in the matrix. This
can be illustrated by extending the example in section 3.2.2.
Figure 4 shows how the sound spectrum oI the hypothetical
synthesiser may be divided into Irequency bands and mapped
onto the matrix as variations in light intensity. The result is that
the interIace reacts more strongly to the most dominant
Irequencies in the sound spectrum. This type oI Ieedback is a
very direct and intuitive Iorm oI control as the perIormer is
interacting with some aspect oI the sound they are producing.

Figure 4. Example of audio-optic feedback

Figure 5. Functional block diagram of hardware
3.3 Audience interaction
The musical purist will always put the sonic outcome Ioremost
in any perIormance. However a perIormance not only entails
listening but also observing. A recital will always be more
engaging iI it is interesting to watch. This interest is partly
maintained through the audience`s understanding oI the
instrumentalist`s craIt: how physical interaction with an
instrument relates to the sound it produces. It is this cross
perceptual reinIorcement that enhances the experience oI a live
musical perIormance.
When it comes to electronic music perIormances one oI the
primary tools is the laptop computer. As a musical instrument
the operation oI a computer is generally unengaging |11|
because there are no motions discernable by the listeners that
translate to sound output. The Light Matrix remedies this
problem by providing an interIace to the computer that removes
the physical barrier oI the computer screen. Much like a
traditional instrument. the gestures and motions oI a musician
operating the Light Matrix are clearly visible. In addition the
interIace has a certain aesthetic appeal because the glowing
surIace oI the device highlights the perIormer`s hands.
The maior components oI the device prototype are shown in
Figure 5. The device communicates with the computer via USB
(Universal Serial Bus) protocol. The microcontroller contains
inbuilt USB interIace circuitry and is also serially
programmable over USB.
The height Iield data is constructed by scanning each LED
sequentially in a raster pattern. There are two phases to each
LED pixel read operation:
1. A reIerence measurement: This is perIormed under
'dark¨ conditions where all the LEDs in the matrix
are switched oII during the sampling period.
2. The actual measurement: While sampling. all other
LEDs in the matrix remain in a light emitting state.
of sound
Display driver
Pr oceedi ngs of the 2006 I nter nati onal Confer ence on New I nter faces for Musi cal Expr essi on (NI ME06), Par i s, Fr ance
The reIerence measurement Iorms the baseline Ior comparison
with the actual measurement. Using the diIIerence between the
reIerence measurement and actual measurement reduces the
interIerence caused by ambient lighting.
The scanning operation is perIormed at the maximum rate
allowable by the ADC (Analogue Digital Converter) chip.
Consequently the strobing caused by alternating between these
two measurement phases is not perceptible due to the response
time oI the human visual system. The net eIIect being that the
device is on continuously even though it is actually switched oII
Ior some oI the time.
4.1 LED matrix
LEDs as sensors in this application oIIer several advantages.
Firstly they are cheaper than photodiodes or phototransistors
and the bidirectional property oI each element simpliIies wiring.
mechanical construction and circuit complexity. This in turn
reduces cost and allows Ior increased matrix density and size.
When the LED is Iorward biased. the current Ilow causes the
device to emit light as per normal. Under certain conditions
LEDs can be operated as photosensors |6||7|. As photosensors.
LEDs act as a narrow band light sensors. generally picking up
wavelengths shorter than those emitted |2|. The prototype
utilises high intensity red LEDs which exhibit relatively good
sensitivity because the emission spectrum is close to the
wavelengths in the absorption spectrum.
4.2 Photosensor amplifier/ADC
The role oI the ampliIier is to convert the photocurrent Irom
each LED into a voltage. the current being proportional to the
light intensity impinging on the diode iunction.
The output voltage Irom the ampliIier is then digitised by the
ADC which is a 13 bit switched capacitor variety. Calibration
oI the ADC is done electronically by a pair oI DACs linked to
the microcontroller. These provide the upper and lower
reIerence voltages needed by the ADC.
4.3 Display driver/multiplexer
The display driver consists oI a bank oI LED driver chips
interIaced with the microcontroller that provide constant current
PWM dimming. The multiplexer circuitry is responsible Ior
interleaving the display and sensing phases.
The soItware is an application written in C# that perIorms the
Iollowing tasks:
Communicating with the Light Matrix device over
USB. This includes reading raw height Iield data Irom
the microcontroller and setting the LED intensity
Mapping height Iield data to MIDI control messages
and sending these messages to a speciIied MIDI
output port.
Mapping height Iield data to LED intensity values.
Frequency and loudness analysis oI an audio signal.
Mapping Irequency and loudness data to LED
intensity values.
An audio signal is Ied into the Light Matrix control soItware
through a purpose built VST |12| eIIect plug-in. This plug-in
acts as a bridge between the host audio application and the
Light Matrix control application.
Exploration oI the optical Ieedback and mapping possibilities oI
the Light Matrix has only iust begun. More Ilexible soItware is
needed to deIine these mappings.
Use oI the antiquated MIDI protocol limits the amount oI
control available. Open Sound Control (OSC) |8| or a more
direct Iorm oI communication should be employed.
More investigation is needed to determine an arrangement oI
LEDs that is more conducive to perIormance gestures made
with hands. The arrangement oI the LEDs in a square matrix is
probably not the most suitable.
|1| Baker. Bonnie C.. Keeping the Signal Clean in
Photosensing Instrumentation.
|2| Dietz. P.. & Yerazunis. W.. & Leigh. D.. Jerv Low-Cost
Sensing and Communication Using Bidirectional LEDs
|3| Han. J.. Multi-Touch Sensing through LED Matrix
|4| Iwai. T. Yamaha Tenori-On.
|5| JazzMutant Lemur.
|6| Mims. Forrest M.. III. Siliconnections. Coming of Age in
the Electronic Era. McGraw-Hill. New York. NY. 1986.
|7| Mims. Forrest M.. III. LED Circuits and Proiects. Howard
W. Sams and Co.. Inc.. New York. NY. pp. 60-61. 76-77.
|8| Open Sound Control.
|9| Overholt. D. The MATRIX. A Novel Controller for
Musical Expression
|10|Roads. C. The Computer Music Tutorial. The MIT Press.
1996. pp. 656-657.
|11|Stuart. C. The Obiect of Performance. Aural
Performativitv in Contemporarv Laptop Music
|12|VST. Steinberg`s Virtual Studio Technology.

Proceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06), Paris, France

Sign up to vote on this title
UsefulNot useful