Acoustics, Digital Media

and Audio Engineering

PERCEPTUAL TESTING FOR ACOUSTICS

AND AUDIO

Dr#Bruno#Fazenda#
 Acoustics, Digital Media
and Audio Engineering

Basic Considerations

PERCEPTUAL TESTING

ACUSTICAUACH#2015##Dr#Bruno#Fazenda#
 Acoustics, Digital Media
and Audio Engineering

Psychoacous>c#Methods#
A#sensory#evalua>on#where#Human#is#the#measurement#instrument#
Quan>fy#the#percep>on#of#a#given#aspect#by:#
Varying#parametric#factors#that#make#up#that#aspect#
Iden>fying#Human#response#to#it#
detec>on#
mood#
Preference#
So,#for#a#given#study#we#need#to:#
Iden>fy#the#factor#under#study#
Appropriately#present#a#dierent#number#of#cases#to#a#panel#of#listeners##modelling#and#
audi>ons#
Obtain#a#meaningful#response#from#each#listener##data#collec>on#
Analyse#if#there#are#any#signicant#results#from#the#panels#responses##sta>s>cal#analysis#

FACTOR
UNDER
QUANTIFIABLE LISTENER SUBJECTIVE RESPONSE
STUDY
ACUSTICAUACH 2015 Dr Bruno Fazenda
 Acoustics, Digital Media
and Audio Engineering

START

Decision to Define research

undertake question and Literature Study
subjective test hypothesis

Define initial Statistical design Conduct pilot

conditions considerations experiment

Check design with Document

Report results dummy data experimental set-
up

Run main
Conclusions Statistical analysis
experiment

END ACUSTICAUACH 2015 Dr Bruno Fazenda

 Acoustics, Digital Media
and Audio Engineering

Do#you#really#want#to#do#subjec>ve#tes>ng?#
Subjec>ve#tes>ng#is#a#lengthy#process#which#requires#prepara>on#and#
knowledge#in#order#to#provide#credible#results#
When#to#do#listening#tests?#
Find#preference#of#a#system/method#over#others#
Iden>fy#problems#introduced#by#a#system/method#
Quan>fy#subject#response#to#a#given#s>muli#
When#NOT#to#do#listening#tests?#
If#a#similar#test#has#been#done#
If#there#is#an#(automa>c)#objec>ve#way#of#doing#it#
i.e.#measure#using#a#model/method/apparatus#

ACUSTICAUACH 2015 Dr Bruno Fazenda

 Acoustics, Digital Media
and Audio Engineering

The#research#ques>on#and#hypothesis#
What#is#it#that#we#are#trying#to#test?#
1. Quality#dierences#between#products/samples#(i.e.condi&ons)#
2. Preference#of#one#condi>on#over#others#
3. Detec>on#of#an#artefact#
4. Performance#over#given#condi>ons#
Examples:#
1. How#does#the#sound#quality#of#our#loudspeaker#A#compare#with#that#of#
loudspeaker#B?#
2. Which#audio#mix#do#you#prefer,#A#or#B?#
3. Can#you#hear#the#signal?#Is#there#a#dierence#between#sample#A#and#B?#
4. How#long#does#it#take?#
ACUSTICAUACH 2015 Dr Bruno Fazenda
 Acoustics, Digital Media
and Audio Engineering

Dening#the#Test#`#Ini>al#Condi>ons#
The#ini>al#condi>on#is#dened#by#a#set#of#variables#that#will#test#the#
hypothesis#
Variables#can#be:#
Categorical##eg:#Age,#Loudspeaker(A,B,C,D)#
Numerical##eg:#ra>ng,#cuto#frequency,#SPL#
Test#design,#deployment#and#sta>s>cal##analysis#techniques#will#be#
dependent#on#these#
Variables#in#a#sta>s>cal#analysis:#
Dependent##the#answer#provided#by#the#subject#
Independent##those#controlled#by#the#experimenter#
Examples:#
 Acoustics, Digital Media
and Audio Engineering

Examples#of#Test#Variables#
Test Independent Variables Dependent Variables

Example 1: Loudspeaker !Loudspeakers A and B !Subject s preference in the form of

!MusicalGenre/Samples ranking or rating scores
comparison
!Loudspeaker position in room

!Presentation Levels

Example 2: Ambisonic vs !Ambisonic and surround systems !Subject s localisation as perceived

!Musical Genre/Samples angle
Surround
!Transient/Frequency characteristics of
signals (eg: snare hits vs bowed cello)

MP3 codecs !Bitrate !Subject s preference in the form of

!MusicalGenre ranking or rating scores
!Playback type !Subject s response in terms of signal
degradation
 Acoustics, Digital Media
and Audio Engineering

The#research#ques>on#and#hypothesis#
You#need#to#dene#the#null#hypothesis#(H0)#and#alterna>ve#hypothesis#(H1)#for#
appropriate#sta>s>cal#analysis.#
For#1#and#2#:#
H0:The#perceived#sound#quality#of#loudspeaker#A#is#iden>cal#to#loudspeaker#B.#
#H1:#There#is#a#signicant#dierence#in#perceived#quality#between#loudspeaker#A#and#B.#
H0:There#is#no#signicant#preference#of#one#mix#over#another.#
#H1:#There#is#a#signicant#dierence#in#preference#of#one#mix#over#another.#

For#3#and#4:#we#are#simply#trying#to#quan>fy#the#thresholds#or#the#performance.#
#A#score#will#be#the#result##but#a#sta>s>cal#test#might#s>ll#be#necessary#
The#primary#aim#is#to#collect#data#and#check#whether#it#rejects#the#Null#Hypothesis##
this#is#called#Null+Hypothesis+Signicance+Tes&ng+(NHST)#
A#decision#criteria#must#be#dened#in#advance#
#Most#commonly#for#work#in#audio#this#is#set#at#5%#
ACUSTICAUACH 2015 Dr Bruno Fazenda
 Acoustics, Digital Media
and Audio Engineering

The#research#ques>on#and#hypothesis#
The#primary#aim#is#to#collect#data#and#check#whether#it#rejects#the#Null#
Hypothesis##this#is#called#Null+Hypothesis+Signicance+Tes&ng+(NHST)#
A#decision#criteria#must#be#dened#in#advance#
Most#commonly#for#work#in#audio#this#is#set#at#5%#
A#parametric#test#such#as#ANOVA#or#t`test#checks#the#probability,#p,#that#
the#measured##results#from#two#or#more#condi>ons#(the#means#obtained#
from#each#test)#came#from#the#same#popula>on#(H0)#
If#p<0.05#then#we#can#reject#H0#and#state#that#the#results#do#indeed#report#a#
dierence#in#percep>on#between#the#condi>ons#tested#
An#idea#of#how#strong#your#result#is#can#be#obtained#from#the#eect+size+
a#value#which#allows#you#to#see#how#much#your#independent#variable#(IV)#has#
aected#the#dependent#variable#(DV)#in#an#experimental#study#
It#looks#at#how#much#variance#in#your#DV#was#a#result#of#the#IV##
ACUSTICAUACH 2015 Dr Bruno Fazenda
 Acoustics, Digital Media
and Audio Engineering

Sta>s>cal#Design#Considera>ons#
Most#tests#will#be#looking#for#sta&s&cal+signicance+in#the#
results#
#i.e.#preference#of#loudspeaker#A#is#higher#than#that#for#Loudspeaker#B.#
#or,#The#threshold#of#detec>on#measured#at#dierent#frequencies#is#signicantly#dierent#
#Null#Hypothesis#(H0)#states##no#dierence#in#preference#between#any#of#the#
loudspeakers#
Appropriate#methods#for#tes>ng#sta>s>cal#signicance#must#
be#employed#
#
Loudspeakers A, B, C, D Quality Rating of Loudspeakers
A B C D 10
John 4.5 9.6 3.1 10
Tom 6.7 9.4 2.9 9.6 8
Hannah 3.2 9.6 2.2 9.1 6
Matt 6.6 9.2 3.4 9.4
Jenny 5.3 8.7 3.2 8.2 4 Mean
Peter 5.2 9.3 2.8 8.9
2
Mean 5.2500 9.3000 2.9333 9.2000 0
St. Dev 1.32 0.33 0.42 0.62 A B C D
 Acoustics, Digital Media
and Audio Engineering

Sta>s>cal#Design#Considera>ons#
How#many#subjects#are#needed#for#sta>s>cal#signicance?##
This#is#dicult#to#dene##chicken#and#egg#problem##
usually#try#to#get#as#many#subjects#as#you#can#to#improve#
chances#of#success#
For#very#small#perceptual#dierences#or# very#subjec>ve #tasks,#
such#as#quality#ra>ngs,#you#may#need#a#very#large#number#of#
subjects#(>30)#to#ensure# normal#distribu>on (which#will#allow#
you#to#use#more#powerful#parametric#stats#tests)#
You#can#do#a#power+test#to#check#whether#the#number#of#
samples#(i.e.#subjects#x#nr#audi>ons)#is#enough#for#sta>s>cal#
signicance#
Pre`hoc:#this#usually#requires#an#es>ma>on#of#parameters#in#data#such#as#the#
variance#
Post`hoc:#will#tell#you#whether#a#sta>s>cal#non`signicance#is#due#to#low#power#
or#indeed#not#having#an#eect#from#the#experiment#
 Acoustics, Digital Media
and Audio Engineering

Defining the Test - Subjects

Describing Properties of Products

Elici>ng#a#set#of#
Sensory#Proling# akributes#
(descriptors)#
10#to#15#trained#
Trained#Panels# listeners#

Sensory#Dierence#
Tes>ng# Detec>ng#dierences#
Subjec>ve#Test#

Preferences#
Hedonic#or#Aec>ve# 100#to#150#naive#
Consumers#(naive)# Tests# Trends# listeners#
Market#studies#

Describing Degree of Liking

 Acoustics, Digital Media
and Audio Engineering

Sensory Panel Methodologies

Descriptive Sensory Analysis (sensory Profiling)
Identifying product properties through a consensus language
Eliciting (creating) attributes to use for describing product
differences
Product examples are used to elicit descriptors
Scales may also be drawn with consensual end points (i.e.
strong vs weak for bass in a mix)
Discrimination Tests
Can users detect small differences between products?
 Acoustics, Digital Media
and Audio Engineering

Discrimina>on#Tests#
! Triangle test
! Two identical and one different sample

! Which one is different?

! ABX
! A double blind test methodology
! Sample X is a random choice between A and B
! Each subject runs test for a number of trials (>10)
! Number of trials gives statistical significance
based on binomial distribution
! Software:
" MATLAB
" WinABX
" Paired comparison (Thurstone)
" Which of the samples has/is more <attribute> ?
 Acoustics, Digital Media
and Audio Engineering

Consumer#Tests#
Rating Based
Consumer provides a score for each product on a
given scale
Ranking Based
All samples (products) are presented simultaneously
Consumers asked to rank them in order of preference
Can use (Thurstone s) paired comparison but this will
lead to a large number of auditions
 Acoustics, Digital Media
and Audio Engineering

Ra>ng#Based#Scales#
! Mainly used for preference
! Does system A sound
better than B and if so
by what degree?
! Also used for noticeability
of degradations

From: Bech, Zacharov, Perceptual Audio Evaluation, Wiley, 2006

 Acoustics, Digital Media
and Audio Engineering

Ra>ng#Based#Scales#
! A common example in audio is the MUltiple Stimuli with
Hidden Reference and Anchor MUSHRA
! ITU-R recommendation BS.1534-1

From: Bech, Zacharov, Perceptual Audio Evaluation, Wiley, 2006

 Acoustics, Digital Media
and Audio Engineering

Numerical#Con>nuous#Scale#
! Numerical continuous scale
! Eg: After listening to the sample please indicate
perceived direction in degrees (use diagram below as a
guide)
0
45

90

135
180
 Acoustics, Digital Media
and Audio Engineering

Experimental#Setup#
Documenting Setup
Experimental details
Calibration
Levels
Nr Subjects
Positioning of sources/listener(s)
sample names/numbers
working plan
health and safety
Etc
Everything you would need if you or someone else were to
set up the experiment again or rerun the analysis in the
future