You are on page 1of 26

Research on Music Perception and Action at Haskins Laboratories: A Fast-Forward Review (Or: 25 Years in 8 Minutes

Bruno H. Repp

Haskins Laboratories

300 George Street 270 Crown Street (1971-2005) “Haskins Laboratories is an independent, international, multidisciplinary community of researchers conducting basic research on spoken and written language.” *What am I doing there?+

Memory & Cognition.. & Repp. R. 123-135. working on speech perception and acoustic phonetics. & Repp. H. melodies are more difficult to recognize if they are combined with different lyrics. 25. M. G.. Davidson. R. However. H.G. L. R.. (1984). 16. J. Serafine. B. Cognition. B. On the nature of melody-text integration in memory for songs... Physical interaction and association by contiguity in memory for the words and melodies of songs. L. Crowder.Beginnings: Melody and Text I started out as a speech researcher. Serafine. Robert G. B. Crowder (1939-2000) Mary Lou Serafine (recent picture) Our research showed that in a test of memory for unfamiliar songs... in the 1980s I started a collaboration with Mary Lou Serafine and Robert Crowder on a musical topic of common interest. 285-303. H. L. Integration of melody and text in memory for songs. G. (1990). M. Serafine. Crowder. 469-476. Journal of Memory and Language. . M. 18. (1986). A series of experiments explored the conditions under which this integration of melody and text occurs. & Repp. Crowder.

The results were mixed. H. 88. H. With his help. B. B. 423-434. 8. 7. I encountered Manfred Clynes and his theory of composer-specific expressive microstructure. Composers' pulses: Science or art? Music Perception. Repp. (1990a). B.The “Composer’s Pulse” At a conference in 1985. Expressive microstructure in music: A preliminary perceptual assessment of four composers' "pulses". 243-274. Manfred Clynes Repp. I synthesized piano performances exhibiting “appropriate” and “inappropriate” microstructure. Music Perception. 6. H. (1990c). (1989). which intrigued me. 622-641. H. Repp. Patterns of expressive timing in performances of a Beethoven minuet by nineteen famous pianists. Journal of the Acoustical Society of America. 1-33. Music Perception. and analyzed the timing of expert pianists’ performances of a Beethoven sonata movement. (1990b). Repp. Further perceptual evaluations of pulse microstructure in computer performances of classical piano music. obtained aesthetic judgments from listeners. . B.

Journal of the Acoustical Society of America. B.Performance Analyses I Now I was hooked. Objective performance analysis as a tool for the musical detective. No. H. which enabled me to give objective descriptions of commonalities and individual differences among famous artists’ interpretations. B. B. 641-650. Repp. (1992b). . 100. Repp. Expressive timing in Schumann's "Träumerei": An analysis of performances by graduate student pianists. 1203-1204. The dynamics of expressive piano performance: Schumann's "Träumerei" revisited. 15. H. 2413-2427. I decided to abandon speech research and focus on music. Diversity and commonality in music performance: An analysis of timing microstructure in Schumann's "Träumerei". Later I analyzed both expressive timing and dynamics in MIDI recordings of graduate student pianists’ performances. (1995c). 98. 2546-2568. B. H. Repp. (1993b). Journal of the Acoustical Society of America. which were demonstrably less individual than the expert performances. (1996b). Journal of the Acoustical Society of America. Repp. H. 7. Stimulated by my analysis of Beethoven performances. op. 92. I conducted detailed measurements and statistical analyses of expert pianists’ expressive timing in performances of Robert Schumann’s “Träumerei”. 93. Journal of the Acoustical Society of America.

56. (1992c). 221-242. 269-284. Psychology of Music. Relational invariance of expressive microstructure across global tempo changes in music performance: An exploratory study. A constraint on the expressive timing of a melodic gesture: Evidence from performance and aesthetic judgment. and the perceived tempo of an expressively timed performance. (1994b). H. the scaling of expressive timing with changes in tempo. 10. Psychological Research. B. 39-57. Repp. 157-167. They investigated the optimal shape of a ritardando. B. H. 22. Repp. Quantitative effects of global tempo on expressive timing in music performance: Some perceptual evidence. H. (1995b). On determining the basic tempo of an expressive music performance. Music Perception. H. 13. (1994a). Repp. Repp. B. Music Perception. B.Perception of Tempo and Timing My “Träumerei” analyses led to several perceptual and performance studies using this music. .

H. H. B. B. (1997c).) Repp. 164-172. Pedal timing and tempo in expressive piano performance: A preliminary investigation. The effect of tempo on pedal timing in piano performance. (1996e). Repp. (1997f). Psychology of Music. . The art of inaccuracy: Why pianists' errors are difficult to hear. Repp. B. arpeggio timing. Psychology of Music. (1997g). Psychological Research. 530-534. H. (1996c). B. Journal of the Acoustical Society of America. 14. 4.Performance Analyses II Making use of extensive MIDI performance data I had collected from Yale graduate student pianists. (1996c). Repp. 3917-3932. 133-148. Expressive timing in a Debussy Prelude: A comparison of student and expert pianists. Music Perception. I measured and analyzed everything I could lay my hands on. B. 1. 199-221. Some observations on pianists' timing of arpeggiated chords. Patterns of note onset asynchronies in expressive piano performance. Repp. H. (Manfred Clynes called me “Reppmesser”. H. Musicae Scientiae. 257-268. 24. 161-184. including pedal timing and its dependence on tempo. and error patterns. Repp. asynchronies within and between hands. 60. Psychonomic Bulletin & Review. B. B. 100. 25. H. (1997d). Repp. Variability of timing in expressive piano performance increases with interval duration. H.

perception. Journal of the Acoustical Society of America. B. (1993a). . B. B. Repp. Some empirical observations on sound level properties of recorded piano tones. (1998f). H. B. and of the production and perception of legato and staccato articulation on the piano. Repp. Journal of the Acoustical Society of America. 93. and production of legato articulation on a computer-controlled grand piano. Repp. using simple musical materials. and production of legato articulation on a digital piano. (1995a). 1878-1890. Repp. H. H. 1136-1144. Unpublished manuscript. 97. H. I also conducted a few studies of piano acoustics. Acoustics. 102. (1997b). 3862-3874.Piano Acoustics and Technique During that time. Journal of the Acoustical Society of America. perception. Perception and production of staccato articulation on the piano. Acoustics.

645-665. 26.Digression: The “Tritone Paradox” A stay in the Netherlands in 1993 enabled me to pursue research on the auditory illusion called “tritone paradox” and its presumable connection to language and speech. Perception. Spectral envelope and context effects in the tritone paradox. The tritone paradox and the pitch range of the speaking voice: A dubious connection. (1994c). 12. Music Perception. B. H. 227-255. . B. Diana Deutsch Repp. H. (1997e). proposed by Diana Deutsch. Repp. My results did not settle the issue but led to some new paradoxes.

Repp. B. I synthesized composite music performances by averaging the timing and dynamic patterns of individual performances and found that they. Repp. 419-444.“Average” Performances Inspired by recent findings in the psychological literature showing that composite faces constructed by averaging digitized images are judged to be more attractive than most individual faces. Music Perception. B. (1997d). I also found that the average timing patterns of groups of expert and student pianists were remarkably similar. were preferred to most individual performances (when realized on a digital piano). 257-268. H. too. H. suggesting a common norm. Musicae Scientiae. 1. The aesthetic quality of a quantitatively average music performance: Two preliminary experiments. . 14. (1997a). Expressive timing in a Debussy Prelude: A comparison of student and expert pianists.

241-281. Psychological Research.Action-Perception Parallels in Expressive Timing In a series of perceptual studies I demonstrated that the detectability of a small local change in the timing of an otherwise metronomic synthesized music performance is closely related to the average timing pattern of performances of the same music: If an interval is typically lengthened in performance. B. Relationships between performance timing. Repp. presumably because it sounds “normal”. its artificial lengthening is difficult to detect. H. B. Detectability of duration and intensity increments in melody tones: A partial connection between music perception and performance. . 1217-1232. H. Obligatory "expectations" of expressive timing induced by perception of musical structure. Repp. 51. 57. perception of timing perturbations. B. (1995d). (1999g). 188 203. B. B. Journal of Experimental Psychology: Human Perception and Performance. H. H. and perceptual-motor synchronization in two Chopin preludes. Repp. Variations on a theme by Chopin: Relations between perception and production of deviations from isochrony in music. 33-43. Australian Journal of Psychology. (1992a). (1998c). Cognition. (1998a). 44. Repp. 791-811. 24. Probing the cognitive representation of musical time: Structural constraints on the perception of timing perturbations. 61. Repp. H. Perception & Psychophysics.

I measured the timing of 115 recorded performances of the opening phrase of Chopin’s Etude in E major. No. B. 105. Op. Repp. Repp. A microcosm of musical expression: II. 104. H. B. Journal of the Acoustical Society of America. and statistically extracted four underlying independent timing patterns whose weighted combinations yielded good approximations of the observed patterns. . (1999f). H. 3. A microcosm of musical expression: I. Repp. B. Contributions of timing and dynamics to the aesthetic impression of pianists' performances of the initial measures of Chopin's Etude in E major. Journal of the Acoustical Society of America. 10. A microcosm of musical expression: III. I also analyzed the measured dynamic patterns and investigated the (very weak) relation between these objective measures and aesthetic judgments of the performances. Quantitative analysis of pianists' dynamics in the initial measures of Chopin's Etude in E major. 469-478. (1998d). (1999a). 1972-1988. H. Quantitative analysis of pianists' timing in the initial measures of Chopin's Etude in E major.Performance Analyses III In my most ambitious performance analysis project. Journal of the Acoustical Society of America. 1085-1100. 106.

including experiments on performance imitation. In W. Journal of Motor Behavior. (2002a). H. 18. 265-290. (2000). Repp. Perception & Psychophysics. Repp. (1998b). B. Repp. Pattern typicality and dimensional interactions in pianists’ imitation of expressive timing and dynamics. (1999d). Detecting deviations from metronomic timing in music: Effects of perceptual structure on the mental timekeeper. The detectability of local deviations from a typical expressive timing pattern.). 64. (1999c). B. 529-548. 61. H. Oxford. Repp. Control of expressive and metronomic timing in pianists. Music Perception. H. (2002e). B. Repp. Effects of auditory feedback deprivation on expressive piano performance. B. H. H. B. Repp. Prinz & B. and pattern learning. Perception of timing is more context sensitive than sensorimotor synchronization.: Oxford University Press. (1999b). 145-164. Music Perception. U. H. . 245–265). mental imagery. H. Repp. The embodiment of musical structure: Effects of musical context on sensorimotor synchronization with complex timing patterns. 16. Music Perception. 409-438. B. 173–211.More Chopin Etude Studies The Chopin performance data enabled me to conduct other studies that relied on them.K. 31. B. Hommel (Eds. 15. 703–716. Perception & Psychophysics. Common mechanisms in perception and action: Attention and Performance XIX (pp. the role of auditory feedback.

Phase correction in sensorimotor synchronization: Nonlinearities in voluntary and involuntary responses to perturbations. Psychological Research. Repp. B. I discovered that small timing changes in the music that were barely audible were automatically compensated for in tapping. Journal of Motor Behavior. From now on . Repp. Repp. my research focused on simple rhythms and only rarely used real music. (2000a). . 63. H. (2001a). Human Movement Science. 28. H. 410–430. Journal of Experimental Psychology: Human Perception and Performance. B. H. This awakened my interest in the phase correction process underlying sensorimotor synchronization. Phase correction.First Tapping Studies In some of my recent music studies I had used finger tapping as a method of tracking expectations about expressive timing. Phase correction following a perturbation in sensorimotor synchronization depends on sensory information. 27. B. I confirmed my finding in several studies using phase perturbation methods. Journal of Experimental Psychology: Human Perception and Performance. 600–621. (2002d). 1–37. Automaticity and voluntary control of phase correction following event onset shifts in sensorimotor synchronization. 291–298. (2002c). 106–128. B. and phase shifts after subliminal timing perturbations in sensorimotor synchronization. phase resetting. 21. H. Compensation for subliminal timing perturbations in perceptual-motor synchronization. H. B. (2002b). 34. Repp. Repp.

Germany) has remained a close collaborator. 20. Keller Repp. Quarterly Journal of Experimental Psychology. Processes underlying adaptation to tempo changes in sensorimotor synchronization. . Peter (now at the Max Planck Institute for Human Cognitive and Brain Sciences in Leipzig. B. Repp. (2004). who spent one year with me as a post-doc. 277–312. Adaptation to tempo changes in sensorimotor synchronization: Effects of intention. E. & Keller. H. Peter E.Phase and Period Correction Together with Peter Keller. Human Movement Science. B. attention. (2001b). 57A. I investigated how synchronized tapping adapts to tempo change in a metronome (i. period correction). H.e. P. 499–521. We found period correction to be less automatic and more cognitively controlled than phase correction.. and awareness..

J. H. Chen. Repp.. (2004). 28. San Diego). The influence of metricality and modality on synchronization with a beat.Synchronization in Two Modalities Amandine Penel. B. 163. & Repp.. Chen. Y. Iversen. H.. another post-doc... Human Movement Science. (2002). H. H. I also started collaborating with Aniruddh Patel and John Iversen (Neurosciences Institute. 1085–1099.. B. Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Aniruddh D. Auditory dominance in temporal processing: New evidence from synchronization with simultaneous visual and auditory sequences. and I found that people had difficulty synchronizing with visual sequences. Patel. (2002). R. (2005). Journal of Experimental Psychology: Human Perception and Performance. 515–532. Patel John R. A. Psychological Research. 252–270. D. Y. & Patel. B. 21. Iversen Repp. B. Repp. 68. . D. Spectral decomposition of variability in synchronization and continuation tapping: Comparisons between auditory and visual pacing and feedback conditions. Experimental Brain Research.. A. 226–238. & Penel. A. especially in the presence of auditory distractors. & Penel. We found that it is difficult to extract a beat from a visual rhythm. A.

a surprisingly difficult task.. 23. H. 121. (2006a). Journal of Experimental Psychology: Human Perception and Performance. (2004).. Repp. Psychological Research. . Does an auditory distractor sequence affect self-paced tapping? Acta Psychologica. Keller. H. (2003a). E. Staying offbeat: Sensorimotor syncopation with structured and unstructured auditory sequences. H. (2005). (2004b). H. On the nature of phase attraction in sensorimotor synchronization with interleaved auditory sequences. B. & Repp. Keller. 29. 290– 309. Quarterly Journal of Experimental Psychology. 57A. P. E. & Repp. Repp. I investigated the influence of auditory distractors on synchronization with an auditory metronome or on selfpaced tapping. P. Human Movement Science. When two limbs are weaker than one: Sensorimotor syncopation with alternating hands. B. B. H. B. 81–107. 389–413. 69. 292–309.Distractors and Syncopation In a series of experiments. 1085-1101. Repp. Phase attraction in sensorimotor synchronization with auditory sequences: Effects of single and periodic distractors on synchronization accuracy. B. Peter Keller took the lead in investigations of off-beat synchronization with alternating hands.

This “synchronization threshold” reflects a sensorimotor processing limit around 8-10 Hz. 23. 355–370. (2005a). 434-454. The role of different kinds of accent. Repp. Repp. Music Perception. Qualitative observations. Repp. (2005b). H. 529-543. B. Perception & Psychophysics. (2007b). B. Rate limits of on-beat and off-beat tapping with simple auditory rhythms: 2.The Synchronization Threshold In several studies. H. B. 167–189. Rate limits in sensorimotor synchronization with auditory and visual sequences: The synchronization threshold and the benefits and costs of interval subdivision. Hearing a melody in different ways: Multistability of metrical interpretation. Rate limits of on-beat and off-beat tapping with simple auditory rhythms: 1. H. 102. 69. 479–496. Repp. B. 35. I determined the fastest rate at which musicians are able to maintain synchrony with a metronome or a simple rhythm. Repp. reflected in rate limits of sensorimotor synchronization. Journal of Motor Behavior. Cognition. . Perceiving the numerosity of rapidly occurring auditory events in metrical and non-metrical contexts. 22. B. (2003b). Music Perception. H. H. (2007a).

Sensorimotor synchronization: A review of the tapping Musical synchronization. B.). 2. Altenmüller.Review Papers Halfway through the decade I published three review papers. . Repp. Repp. B. Advances in Cognitive Psychology (http://ac-psych. Oxford. Wiesendanger. UK: Oxford University Press. Rate limits of sensorimotor synchronization. Repp. H. (2005c). motor control. 12. H. and the brain (pp. Psychonomic Bulletin & Review. 55–76). Music. Kesselring (Eds. M. In E. H. 163181. (2006c). (2006d). B. & J. 969– 992.

B. & Keller. 102-111. Newark. 202. Consciousness and Cognition. 16. H. . & Repp. Self versus other in piano performance: Detectability of timing perturbations depends on personal playing style. 15. later in Leipzig. Experimental Brain Research. when I did not have any grant support. H. Perceiving action identity: How pianists recognize their own performances. Günther Knoblich (now at Radboud University Nijmegen) taught me much about perception and action and later invited me to work with him for two years at Rutgers University.. Günther Knoblich In three studies involving real music. Psychological Science.Perception and Action: Pianists In 2003 I began regular visits to the Max Planck Institute for Human Cognitive and Brain Sciences in Munich. 101-110. (2007). we showed that pianists can recognize their own performances and predict their own characteristic timing. Keller. Pianists duet better when they play with themselves: On the possible role of action simulation in synchronization.. (2010). & Knoblich. B. 604–609. P. Repp.. Repp. G. G. (2004). P.. B. Knoblich. E. E. H.

we investigated the feeling of agency in tapping and showed that pressing piano keys can affect pianists’ perception of relative pitch in the tritone paradox. G.. (2006b). (2007b). Toward a psychophysics of agency: Detecting gain and loss of control over auditory action effects. & Knoblich. 469-482. H. H. 111. B. Repp. (2007a). 493–504. Repp. 168. G. & Knoblich. 18. Quarterly Journal of Experimental Psychology. Experimental Brain Research. Repp. Knoblich. 248-262. Journal of Experimental Psychology: Human Perception and Performance.Perception and Action: Tapping and Listening In other studies. Inferring agency from sound. B. B.. G. H.. B. Does an auditory perceptual illusion affect on-line auditory action control? The case of (de)accentuation and synchronization. & Knoblich. Psychological Science. 2156-2170. 62. B. Action can affect auditory perception. . H. Performed or observed keyboard actions affect pianists’ judgments of relative pitch. & Repp. 33.. Repp. Cognition. (2009). (2009). 6-7. H. G.

B.. 61–78. Phase correction in sensorimotor synchronization with nonisochronous sequences. London. P... & Desain. P. 565– 593. London. H. . E. I investigated effects of tempo. Repp.. B. 26. 1-18. (2002). Psychological Research. H.. J. J. B. London.. Influences of metrical structure and grouping on the kinematics of rhythmic finger tapping. Effects of tempo on the timing of simple musical rhythms. 23. & Saltzman. & Patel. H. J. L. J. Production and synchronization of uneven rhythms at fast tempi. (2005). & Keller. Repp. E. Repp. H. Tracking an imposed beat within a metrical grid. meter. Music Perception. Windsor. H. beat tracking. Repp. R. D. Music Perception. A. P. H. Unpublished manuscript... Multiple temporal references in sensorimotor synchronization with metrical auditory sequences. (2002). and interval structure on the accuracy of rhythm production and perception. (submitted). Music Perception. 79-98. 26. (2008). L. Repp. 171-175. E. P. 19. & Keller. Repp.Rhythm Production and Perception With various collaborators. Repp. E. & Keller.. (2008a). and phase correction in synchronization with moderately complex rhythms. Music Perception. 72.. Iversen. B. B. B. B. (2008). H. Perception-production relationships and phase correction in synchronization with two-interval rhythms..

. & Repp. 128. Keller. 378-386. (2008). Multilevel coordination stability: Integrated goal representations in simultaneous intra-personal and inter-agent coordination. (2008). Sensorimotor synchronization with adaptively timed sequences. Acta Psychologica.. 423-456. I have done some preliminary work with him along these lines. Repp. Human Movement Science. B. H. E. H. 27. B. P. E..Inter-Agent Synchronization Synchronization between individuals (as in music performance) has as yet been little investigated but is a special interest of Peter Keller. & Keller. P.

H. Is recognition of emotion in music performance an aspect of emotional intelligence? Music Perception. Music Perception. & Jendoubi. This has resulted in a number of coauthored publications on various topics.. B. B. & Marcus. H. Context sensitivity and invariance in perception of octave-ambiguous tones. (in press). B. R. H. Music Perception. R. (2009). B... (in press). (2010). Resnicow. A filled duration illusion in music: Effects of metrical subdivision on the perception and production of beat tempo. . (2007). Flexibility of temporal expectations for triple subdivision of a beat. Journal of Motor Behavior. 24. 145–158. I have been sought out by an increasing number of Yale undergraduates who wish to do research on music cognition. M. Advances in Cognitive Psychology. & Doggett. No sustained sound illusion in rhythmic sequences. 27-41. J. Repp. 114-134. (2009). & Repp. and phase correction. 367-376. & Bruttomesso. S. B. Repp. H.. H. Salovey. H.Yale Undergraduate Collaborators In recent years. Tapping to a very slow beat: A comparison of musicians and non-musicians. Repp. Repp. (2004). E. B. & Thompson.. R. B. P. Advances in Cognitive Psychology. Psychological Research. J. continuation. 5. 22. M. J. Simultaneous event-based and emergent timing: synchronization.. H. & Steinman. Repp. H. 5. 111-126... 42. Repp.

Repp. as well as perception of rhythm.. Repp. Top-down control of rhythm perception modulates early auditory responses. 2306-2312. Annals of the New York Academy of Sciences. 62. B. Journal of Motor Behavior. (2010). (2009). Repp. Rhythmic sensorimotor coordination is resistant but not immune to auditory stream segregation. H. & Patel. Sensorimotor synchronization and perception of timing: Effects of music training and task experience. (2008c). B. 40. Quarterly Journal of Experimental Psychology. B. B. Self-generated interval subdivision reduces variability of synchronization with a very slow metronome. B. 29. 26. H. 62. 426-434. J. Do metrical accents create illusory phenomenal accents? Attention. Repp. (2009b). Segregated in perception. Metrical subdivision results in subjective slowing of the beat. H. Repp. (2008b). Human Movement Science. Quarterly Journal of Experimental Psychology. D. & Psychophysics. H. (in press). 19-39. 363-367. H. H. integrated for action: Immunity of rhythmic sensorimotor coordination to auditory stream segregation. Repp. Repp. Music Perception. 58-73. H. B. and accentuation. 200-213. B. Repp. Iversen. Perception. Music Perception. A. . 1169. meter. H. R..Current Research I am continuing to investigate sensorimotor synchronization in various contexts. (in press). B. Perfect phase correction in synchronization with slow auditory sequences. (2009a).

for your attention . for allowing me to stay even though my research is unrelated to the Laboratories’ mission • NIH and NSF for providing financial support • The “Master Tappers” (my regular participants: nowadays graduate students from the Yale School of Music. previously also including undergraduates and other graduate students).Thanks to: • Haskins Laboratories. for their cheerful cooperation and excellent rhythmic skills • You. the audience.