Josh Reiss

Centre for Digital Music

Queen Mary, University of London

INTELLIGENT DIGITAL AUDIO

EFFECTS FOR AUTOMATIC MIXING
All demonstrations available from
www.elec.qmul.ac.uk/digitalmusic/automaticmixing/
Motivation
Musicians who’d rather get
on with making music than
gget too deep There’s no reason whyy a
p into
band
can] recording
engineering
[audio interfaces come using reasonably
with a 'gain learn'conventional
mode... instrumentation
DAWs could optimiseshouldn’t
theirbe EQ’d and balanced
own mixer and plug-in automatically
gain by advanced
structure while preservingDAW the software.
same mix balance.
Classification
Is this art or
engineering?
 Approach
ƒ Create tools
l that
h automate complex
l mixing tasks
k
ƒ Develop effects which dynamically adapt to multi‐
channel input of a mixer
ƒ Analyse content of each channel with respect to other
channels
ƒ Must be suitable for automatic live mixing tasks
¾ Operate real‐time
¾ Preserve system stability

Cross Adaptive Effects

Cross-Adaptive
Also called inter-channel dependent
p or
MIMO (multi-input / multi-output) effects
 Cross-Adaptive Effects

ch1 to chN are audio

signals.

fv1 to fvN are features

extracted from audio
channels
h l

cv1 to
t cvN are control
t l
vectors that drive digital
audio
di processors

General cross‐adaptive
cross adaptive effect diagram
 Automatic mixing host

Auto Panner Normalised-EQ

Time Offset Correction Auto Fader

Channel
Enhancer Hardware Control
Surface
Automatic Mixing Host
Automatic stereo panner
 Automatic Panner
ƒ Side chain processing for real
time applications
ƒ Multiple channel processing
dependency
ƒ Accumulative spectral
decomposition

Multi-input
M lti i t
processing
dependency
Automatic Panner Stereophonic Masking

Di ti
Directional
l spectral
t l masking
ki

Panning space directional masking

minimization
i i i ti

ƒ Reducing directional spectral

masking improves intelligibility.
Automatic Panner Results

Lissajous
j p
plot of sterophonic
p output
p
of automatic panner
Panning factor for 4 input channels using
the automatic panner

ƒ Spectral decomposition rules manage to converge and adapt to inputs.

ƒ Reduction in directional maskingg maximizes stereophonic
p separation.
p
ƒ The automatic panner requires a better priority scheme
Automatic Panner Evaluation
DEMONSTRATION
Automatic Panner
Spectral Enhancer
 Objective
Minimising signal masking through the characterisation
of spectral channel interdependence

What do we define as spectral masking?

Spectral masking is the loss of spectral content perception

of one or more channels when they are mixed together.
‰ not the same as psychoacoustic masking
‰ results
lt from
f signal
i l and
d noise
i occupying
i ththe same ffrequency bi
bins
‰ noise spectrum masks spectrum of the signal
 Inter‐channel dependent spectral enhancer

ƒ Master channel ‘writes’

spectral enhancing
contour and has no
signal processing
applied to it.
ƒ Slave channels ‘read’
spectral contour and
receive a processing
control
t l value.
l

•Master
M t channel
h lhhas continuous
ti iinformation
f ti regarding
di
properties of all channels
 Understanding the display

Spectral
Filter bank division =k2 enhancing
contour

Attenuation

Freq enc
Frequency
 Attenuation Q
Affects all
+ Bypass - Ch
Channels l jjustt
like a normal
ducker would do

Affects Only

- + Channels just
with same
spectral content
DEMONSTRATION
Spectrall Enhancer
h

Un-masked
Masking
Limit

Masked
Automatic
A i gain
i maximisation
i i i
and feedback prevention
Feedback prevention
p

ƒ Feedback Suppressors:
à Frequency shifting
à Auto‐notch filtering
à ...
ƒ Goal of mixing engineer to adjust frequency response
Don't
while stayingTreat
belowthe Symptoms-
threshold for feedback
à Flatten
Treat pthe Cause!
response
à Maximise gain
ƒ Need gain limiter which does not introduce artifacts or
prevent EQ
 Feedback explained
Given the following acoustic model:
ƒ HETOT(x)
( ) iis feed‐forward
f df d
transfer function of the
system
à product of individual
transfer functions of
microphone, equaliser,
amplifier and speaker.
ƒ HATOT(x) is acoustic
feedback transfer
function of the system
The system will
introduce undesired
acoustic feedback if:
HATOT(x)·HETOT(x)>1
 Feedback prevention
Given the following acoustic model:

ƒ if equalizer gain is 0dB

when flat and system is
on marginal condition
before undesired
feedback,
feedback
ƒ can still use the
equalizer by forcing
HeEQ(x) to remain
below 0dB.
DEMONSTRATION
Feedback p
prevention / Gain maximisation
Multi-Channel Time Offset
Correction for Mixing
 Undesired destructive interference of
signals,
i l kknown as comb-flittering,
b flitt i iis
common in day-to-day audio mixing.
Comb-filtering caused by:
•Using
U i more th
than one microphone
i h tto record
d a single
i l
sound source

•Mixing an electric instrument direct box signal with its

acoustic amplified signal
signal.

•Use of pparallel signal

g p
paths with different latency
y times

•Digital implementations which do not compensate for

difference in latency time of different algorithms
 Comb Filter

Peaks at frequencies
1/d, 2/d, 3/d…

Notches at frequencies
1/(2d ), 3/(2d), 5/(2d)…

Where d is delay between

signals

Comb Filtering effect: a white noise signal added to a

copy of itself, delayed by 1ms
 Problem

+
 Aim

Delay / Polarity
Correction

+
Delay / Polarity
Correction
 Implementation

Delay - distance between impulse

response
p absolute maxima and the instant
when the impulse was input into the
system

Polarity - whether the impulse response is

positive or negative will determine the
polarity of the system

General algorithm flow diagram for an automatic

mixture cross-adaptive time offset corrector.
 Results

Impulse response of signal before correction (top) and after the correction
((bottom)) for a weakly
y correlated signal.
g
 Results

Impulse response of signal before correction (top) and after correction

((bottom)) for a highly
g y correlated signal.
g
DEMONSTRATION
Time offset and p
polarityy correction
Automatic faders
Auto-gain

Auto-faders
Auto‐faders

SPL Application
DEMONSTRATION
Automatic Faders and Gain Control
DEMONSTRATION
Putting it all together
NO automatic tools
Auto‐Gain ON
Auto‐Fader ON
Auto‐Pan ON
Varying Auto‐Pan control
Normalise Equaliser
q ON
Transfer function of Equaliser
Equaliser
q Flat
Equaliser Boost
ALL Automatic tools Runningg
 Conclusions
• Automatic Mixing is underdeveloped
• can be achieved byy usingg side pprocessingg
• Automatic stereo panning reduces spatial/spectral masking
• Cross‐channel gain maximisation can avoid feedback
• Fader settings can be automated for equal loudness
• EEnhance
h a channel
h l using
i inter‐dependent
i t d d t spectral
t l features
f t
• Minimises spectral masking of a channel by other channels
• Allow
ll user controll off effect
ff by b modifying
df Q and
d attenuation parameters
• Time offset corrector reduces comb‐filtering due to delays between
audio signals in a mixer
• correct delays with +/‐4 sample accuracy, optimize polarity for all channels
N d a cross‐adaptive,
Need d i inter‐channel
i h l dependent
d d effect
ff host‐
h
VST3?
 Current Work
ƒ Automatic
A t ti Monitor
M it Mixing
Mi i – accepted
t d for
f AES JJournall
ƒ Reverse engineering the mix – submitted to DAFX
ƒ Adaptive noise gating – submitted to DAFX
ƒ Automatic fader and gain control – submitted to WASPAA
ƒ Turing Test

Future Work
ƒ Hardware and VST3 implementations
ƒ Targeted
d mixing
i i
ƒ Auto‐EQ
ƒ automatic ambisonics source placement
à based on musician position tracking
à Based
B d on source content
t t
ƒ Feedback prevention with environmental measurements
 Thanks
ƒ Enrique Perez Gonzalez
ƒ Yonghao Wang
ƒ Michael Terrell
ƒ Martin Morrell
ƒ Daniele Barchiesi

Demonstrations:
www.elec.qmul.ac.uk/digitalmusic/automaticmixing/
Research group:
www.elec.qmul.ac.uk/digitalmusic/
q / g /
automaticmixing/audioengineering.html