You are on page 1of 5
 
Brief Communications
Rapid Recalibration to Audiovisual Asynchrony 
Erik Van der Burg,
1
David Alais,
1
and John Cass
2
1
School of Psychology, The University of Sydney, Sydney, NSW 2006, Australia, and
 2
School of Psychology, The University of Western Sydney, Sydney, NSW2214, Australia
To combine information from different sensory modalities, the brain must deal with considerable temporal uncertainty. In naturalenvironments, an external event may produce simultaneous auditory and visual signals yet they will invariably activate the brainasynchronouslyduetodifferentpropagationspeedsforlightandsound,anddifferentneuralresponselatenciesoncethesignalsreachthereceptors.Onestrategythebrainusestodealwithaudiovisualtimingvariationistoadapttoaprevailingasynchronytohelprealignthesignals. Here, using psychophysical methods in human subjects, we investigate audiovisual recalibration and show that it takes placeextremelyrapidlywithoutexplicitperiodsofadaptation.Ourresultsdemonstratethatexposuretoasingle,briefasynchronyissufficientto produce strong recalibration effects. Recalibration occurs regardless of whether the preceding trial was perceived as synchronous, andregardlessofwhetheraresponsewasrequired.Weproposethatthisrapidrecalibrationisafast-actingsensoryeffect,ratherthanahigher-levelcognitiveprocess.Anaccountintermsofresponsebiasisunlikelyduetoastrongasymmetrywherebystimuliwithvisionleadingproducebiggerrecalibrations than audition leading. A fast-acting recalibration mechanism provides a means for overcoming inevitable audiovisual timingvariationandservestorapidlyrealignsignalsatonsettomaximizetheperceptualbenefitsofaudiovisualintegration.
Introduction
We live in a multisensory world where single external events canproduce correlated signals across our various sensory systems.Combining sensory information across the senses can markedly improvetheprecisionandspeedofperception(Morein-Zamiretal., 2003; Van der Burg et al., 2008b; Alais et al., 2010), although these benefits decline if the signals are asynchronous (Van derBurgetal.,2008b,2010).Audio-visualsignalsareoneparticularly  common cross-sensory pairing that we experience many timeseach day, such as when crossing a road or engaging in conversa-tion. Audition and vision are also the senses we rely on to detectdistant objects and events, although the different speeds of lightand sound mean there is a distance-dependent audiovisual asyn-chrony that makes optimal integration difficult to achieve. Anadaptivephenomenonknownastemporalrecalibration(Fujisakiet al., 2004; Vroomen et al., 2004) helps realign signals, as shown in a number of studies demonstrating that repeated exposure toan audiovisual asynchrony shifts the point of subjective simulta-neity (PSS) in the direction of the leading sense.Recalibration studies have typically used lengthy asynchro-nous cross-modal adaptation procedures lasting up to severalminutes, suggesting that the recalibration process is sluggishand requires long periods of adaptation (Fujisaki et al., 2004;Vroomen et al., 2004; Navarra et al., 2009; Roseboom and Ar- nold, 2011). Ideally, recalibration would occur very rapidly, en-abling the observer to benefit immediately from multisensory integration rather than having to wait several minutes for a slow realignment to occur. Here, we examine whether audiovisualtemporal recalibration can occur over brief time-scales. Using asimultaneity judgment (SJ) task, participants judged whether aluminance onset and tone were synchronized or not across arange of stimulus onset asynchronies (SOAs). If temporal recali-brationoccursrapidly,thenweexpectthatthePSSonthecurrenttrial to be contingent upon the audiovisual asynchrony on thepreceding trial.Our results demonstrate that temporal recalibration can oc-cur extremely quickly, far more rapidly than previously thought.We find that the PSS is highly contingent on the modality orderand SOAs of the preceding trial. We also find that passively ob-serving prior asynchronies is sufficient to trigger recalibration,and that preceding trials judged to be asynchronous still producerecalibration. Further, we show that temporal recalibration isasymmetrical with respect to stimulus order, occurring over abroader SOA range when vision leads on the previous trial thanwhen audition leads. Finally, we show that participants who re-port simultaneity over a broad range show greater PSS recalibra-tion than those with narrower simultaneity ranges. We present atemporal recalibration model to explain these results.
Materials and Methods
Experiment1.
Fifteenparticipants(8females,7males;13naive,meanage:26.6 years) were paid AU $20/h to participate. Trials started with a whitefixation dot on a black screen for 1000 ms after which a surroundingwhite ring appeared (radius 2.6°; width 0.4°) and a tone was presented(500Hz;50msduration).Thetoneeitherprecededorfollowedthering’sonset by a SOA drawn randomly from the set (
0,
64,
128,
256,
512 ms). The task was to judge whether the ring’s onset was synchro-nized with the tone. The advantage of a SJ-task is that the modality orderisirrelevanttothetaskandtheoutcomelesssusceptibletoaresponsebias
Received March 18, 2013; revised July 1, 2013; accepted July 31, 2013.Author contributions: E.V.d.B. and J.C. designed research; E.V.d.B. performed research; E.V.d.B. analyzed data;E.V.d.B., D.A., and J.C. wrote the paper.ThisworkwassupportedbyAustralianResearchCouncilGrantDE130101663(E.V.d.B.)andDP120101474(D.A.and J.C.).CorrespondenceshouldbeaddressedtoDrErikVanderBurg,UniversityofSydney,Sydney,NSW2006,Australia.E-mail: erik.vanderburg@sydney.edu.au.DOI:10.1523/JNEUROSCI.1182-13.2013Copyright © 2013 the authors 0270-6474/13/3314633-05$15.00/0
The Journal of Neuroscience, September 11, 2013
 •
 33(37):14633–14637 •
 14633
 
than other methods such as temporal order judgments (Van der Burg et al., 2008a; Spence and Parise, 2010). The ring remained until theunspeeded response was recorded and the fix-ation dot was present throughout. Participantscompleted120trialsforeachSOAconditioninthree sessions of 400 trials. The monitor (85Hz)wasviewedfrom80cmandthetonesweredeliveredoverheadphones.Eachsessionbeganwith two practice trials.
Experiment 2.
 In Experiment 2, nine partici-pants (4 males, 5 females; 7 naive; mean age:25.8years)participatedinanexperimentiden-tical to Experiment 1 except that the audiovi-sualstimuliwerebothbrief(50ms)andthatonhalf of the trials, a cue informed participantsthat no response was required and instructedthem to passively observe the stimuli. A greenfixation dot indicated test trials, and a red dotindicatedpassivetrials.Afterapassivetrial,thenext (test) trial was initiated after 700 ms. Par-ticipants performed one session of 600 trials.
Results
We conducted separate intertrial analysesfor Experiments 1 and 2 to examinewhether the modality order on a givenprevious trial (Trial t-1) affected the dis-tribution of “simultaneous” responses onthe current trial (Trial t). Simultaneity data form a peaked distribution centeredon the PSS and are well modeled by Gaussian distributions. For each partici-pant, we fitted the data for both kinds of t-1 trial order (i.e., vision led or auditionled) with Gaussians to estimate PSS (dis-tribution mean), amplitude, and simulta-neity bandwidth (the distribution’s SD).
Experiment 1.
a
 plots meanproportionofsimultaneousresponsesasafunction of SOA for both t-1 trial orders,with continuous lines showing best-fitting Gaussian distributions. Figure 1
b
 shows the adaptationeffect size for each individual plotted against their simultaneity bandwidths. The adaptation effect is defined as the PSS whenvision leads on Trial t-1 minus the PSS when audition leads onTrial t-1. When the tone preceded the luminance onset on theprevious trial (green curve), the PSS was significantly shorter (15ms)thanwhenthetonefollowedtheluminanceonset(redcurve;35 ms),
 t 
(14)
3.6,
 p
0.003 (two-tailed
 t 
 test). That is, partici-pants experienced luminance onsets presented
 
35 ms beforethe tone’s onset as “synchronous” when vision occurred first onthe previous trial. Conversely, when audition occurred firston the previous trial, luminance onsets presented 15 ms earlierthan the tones were reported as synchronous. Thus, even thoughwe used brief stimuli and did not use an explicit adaptation pro-cedure entailing repeated exposure to a particular asynchrony,thesestrongintertrialeffectsshowthatadaptationoccursrapidly,with the direction of PSS shift contingent upon the modality order of the previous trial. Modality order did not affect band-widths (
(14)
1.3,
 p
0.210) or amplitudes (
(14)
0.6,
 p
0.534).We established that PSS on a particular trial is strongly deter-mined by the order of audiovisual stimuli on the previous trial(Figs.1
a
,
b
).Takingthisfurther,weaskwhetherthemagnitudeof temporal recalibration depends on the specific SOA of the previ-ous trial. We estimated Gaussian parameters for each partici-pant’ssimultaneitydataasabove,butappliedtheanalysistoeachSOA presented on Trial t-1 (
120 trials to estimate the PSS perSOA on Trial t-1). Data were pooled for
 t 
-1 SOA equals 0 ms.ANOVAs were run on PSS, bandwidth, and amplitude with t-1SOA as the independent variable and Huynh–Feldt corrected
 p
 values reported wherever relevant. As Figure 1
 shows, theANOVAforPSSyieldedareliableeffectoft-1SOA(
(8,112)
6.1,
 p
0.001). We further analyzed this effect by comparing the PSSforeacht-1SOAwiththedefaultPSS(thePSSwhen
-1SOA
0ms) and found that the PSS was significantly shorter when audi-tion preceded vision on the t-1 trial by 64 ms (
(14)
2.3,
 p
0.034) but not for longer SOAs (ps
0.373). In contrast, whenvision led on Trial t-1, PSSs were significantly longer than thedefault for all SOAs (
ts
(14)
3.1, ps
0.008). The ANOVA foramplitude yielded no significant effect (
(8,112)
2.1,
 p
0.07).TheANOVAforbandwidthyieldedasignificanteffect(
(8,112)
3.7,
 p
0.005) as bandwidths tended to increase with SOA onTrial t-1, independently of modality order.Together, these results paint a clear picture of rapid tempo-ral recalibration. Perceived audiovisual simultaneity on a giventrialdependsstronglyonboththemodalityorderintheprevious
Figure1.
 ResultsofExperiment1.
a
,MeanproportionofsimultaneousresponsesasafunctionofSOA.NegativeSOAsindicatethe tone precedes the luminance onset, positive SOAs indicate that the tone followed the luminance onset. The data have beenbinnedbasedonthemodalityorderintheprevioustrial(t-1):greensymbolsindicatethatauditionleadsonTrialt-1,whereasredsymbols indicate that vision leads. Continuous lines are best-fitting Gaussian distributions to the two types of t-1 trial order. ThePSSforeacht-1trialorderisgivenbytheGaussianmean.
b
,Thesimultaneitybandwidth(meanoftheGaussianSDforbothkindsof t-1 trial order) as a function of the adaptation effect (PSS when vision leads-PSS when audition leads on Trial t-1) for eachparticipant.
,MeanPSSofbotht-1trialtypesasafunctionofSOAonTrialt-1.ThedottedlineshowsthemeanPSSforaSOAof0ms on Trial t-1.
, Mean PSS of both t-1 trial types as a function of the preceding response.
14634
 •
 J. Neurosci., September 11, 2013
 •
 33(37):1463314637 Van der Burg et al.
Rapid Recalibration
 
trial and the SOA between those asynchronous stimuli. Intrigu-ingly, temporal recalibration is asymmetrical with respect tostimulus order. When audition led on Trial t-1, only the shortestSOAshowedareliableshiftinPSS,yetwhenvisionledonTrialt-1,significant PSS shifts were observed for all SOAs. The asymmetry based on stimulus order is also evident in the magnitude of PSSshifts, with adaptation being stronger when vision led on Trial t-1(maximum PSS
default PSS
36 ms) compared with when au-ditionledonTrialt-1(minimumPSS
defaultPSS
8ms).The asymmetrical pattern can be easily captured by a simpleasymmetricalGaussianmodel(Fig.2
a
,right).TheGaussianpeaissettoSOA
0ms,anddeclinesrapidlyontheleftside(SOAs
0ms;auditionleadsonTrialt-1)andmoregraduallyontheright(SOAs
 
 0 ms; vision leads on Trial t-1). The asymmetricalGaussiandescribeshowthemagnitudeofrecalibration(PSSshiftrelative to default PSS) varies as a function of SOA on Trial t-1and captures several fundamental observations in our data, spe-cifically: (1) temporal recalibration is most pronounced whenSOAis small,(2) the magnitude of recalibrationdeclines system-atically with increasing SOA, but more rapidly when auditionleads, and (3) recalibration approaches zero for very large SOAs,an important functional limit as audiovisual signals with largeSOAsareunlikelytohaveacommonsourceand should not lead to recalibration. In thismodel, the negative bandwidth (
 
 A
) con-trols the SOA range over which recalibra-tionoccurswhenauditionleadsonTrialt-1,and the positive bandwidth (
 
) controlsthe SOA range supporting recalibrationwhenvisionleads.Becauseanauditoryleadis defined as a negative SOA (Fig. 1
), weinvert the sign of the negativ e side of theasymmetrical Gaussian (Fig. 2
b
, right), andthen add a single source of Gaussian-distributed noise to produce the curves inFigure2
,right).The model in Figure 2
b
 is defined by Equation 1. Equation 2 was used to quan-tify the amount of random noise in themodel. Figure 2
 is the convolution of Equations 1 and 2.
PSS
SOA
1
 SOA
1
 
0,
 
e
SOA
1
2
2
 
 A
2
SOA
1
 
0,
 e
SOA
1
2
2
 
2
,(1)Noise
 
 e
 x 
2
2
 
noise
2
,
 
 
 
 
 
.(2)
The model has four parameters, the reca-libration bandwidths for audition and forvision leading on Trial t-1 (
 
 A
 and
  
),the bandwidth of the noise (
 
noise), andthe normalization factor (PSSnorm). Theonly constraint was that
 
 A
 and
 
 couldnotexceed2500ms.Thenoiseisnormally distributed random noise and representsvariationduetoneuralnoise,latencyvari-ation, response errors, individual differ-ences, etc. PSSnorm scales the magnitude of the asymmetricalGaussian to the observed PSS shifts.The model was fitted to the data in Figure 2
 by minimizingthe root-mean-square-error. Before fitting, the unit amplitudeversionshowninFigure2
wasnormalizedbyshiftingitsasymp-tote to the default PSS. The best fitting asymmetrical model wasobtained when the parameters
 
noise and PSSnorm were 92 and33 ms, respectively, and when the recalibration bandwidth foraudition leading on Trial t-1 (
 
 A
) was 88 ms and for visionleading (
 
) was 2500 ms. The
  
 value corresponds to theconstraining limit of our asymmetric fit. We propose that futurestudies include longer SOAs to better determine this parameter.Nevertheless,ourasymmetricalmodelprovidedaverygoodfittothe data (
R
2
0.93), far better than the symmetrical model (Fig.2, left) where the best fit was obtained for
 
 A
 
1316 ms,
 
noise
56 ms and PSSnorm
39 ms. Overall, although thesymmetrical model describes the data reasonably well (
R
2
0.78),itoverestimatesshiftsinPSSamplitudeandfailstocapturethe asymmetry inherent in the data (Fig. 2
, left).The results show that brief audiovisual stimuli lead to recali-brationevenwhentheauditoryandvisualsignalcomponentsare
Figure2.
 Modeloftemporalrecalibration.Left,ThebehaviorofasymmetricalGaussianmodel.Right,Theasymmetricalmodel.
a
, The green and red regions correspond, respectively, to audition leading on Trial t-1 and vision leading on Trial t-1. In thesymmetrical model, a single parameter controls the SOA range over which recalibration can occur, whereas the asymmetricalmodelhasseparaterecalibrationbandwidthparametersfornegative(auditionleads)andpositive(visionleads)SOAs.
b
,Thesamefunctions as in
a
except that the sign of the Gaussian’s negative side was inverted to accommodate our convention that negativeSOAs refer to audition leading vision.
 c 
, The Gaussian functions were convolved with normally distributed random noise whosebandwidthwasafreeparameter.
,Theexperimentaldata(Fig.1
)showingtheshiftinPSSasafunctionofSOAonTrialt-1.Theblack continuous line shows the best-fitting Gaussian model, whose amplitude is controlled by a PSSnorm parameter.
Van der Burg et al.
Rapid Recalibration J. Neurosci., September 11, 2013
 •
 33(37):14633–14637
 • 14635

Reward Your Curiosity

Everything you want to read.
Anytime. Anywhere. Any device.
No Commitment. Cancel anytime.
576648e32a3d8b82ca71961b7a986505