EP 0 750 776 B1 Sena Pa OIOAMIN UA a oO) cunpeanPaen ine Office européen des brevets (1) EP 0 750776 B1 (12) EUROPEAN PATENT SPECIFICATION (45) Date of publication and mention 61) intci7: G10H 5/00, G10H 1/20, 0.09200" Butetn 20186 GI0H 7108 (21) Aeplication number: 96900484.1 Se cae (22) Dato of fing: 18.01.1996 (87) Intemational publication number: Wo 96/2259 (25.07.1996 Gazette 1996/34) (54) METHOD AND APPARATUS FOR CHANGING THE TIMBRE AND/OR PITCH OF AUDIO SIGNALS \VERFAHREN UND VORRICHTUNG ZUR ANDERUNG DES KLANGES UND/ODER DER TONHOHE VON AUDIOSIGNALEN PROCEDE ET APPAREIL POUR CHANGER LE TIMBRE ET/OU LA HAUTEUR DE SIGNAUX AUDIO (84) Designated Contracting States: ‘+ JUBIEN, Christopher, Michael ATBE CH DE DKESFR GBGRIEITLILUMCNL Vietoria, British Columbia V8T 1R4 (CA) PTSE + RODEN, Brian, John Victoria, British Columbia V8X 1N2 (CA) (80) Prionty: 18.01.1995 US 974110 (74) Representative: VOSSIUS & PARTNER (43) Date of publication of application: Siebertstrasse 4 02.01.1997 Bulletin 1997/01 81675 Munchen (DE) (73) Proprietor: VL TECHNOLOGIES LTD. (66) Relerences cited Victoria, B.C. VaM 126 (CA) EP-A- 0504 684 GB-A-2087 123 US-A- 4991 484 US-A-5 194681 (72) loventors: + GIBSON, Brian, Charles Vietoria, British Columbia V8T 1R4 (CA) Note: Within nine months from the publication of the mention ofthe grant ofthe European patent, any person may give notice to the European Patent Office of opposition tothe European patent granted, Notice of opposition shall be fled in ‘a witien roasoned statement. It shall nt be deamed to have been filed untl the oppasttion fee has bean paid. (Art. {99(1) European Patent Convention).1 EP 0750776 B1 2 Description [0001] The presentinvention relates generally elec: tronic audio effects and in particular to musical effects that shift the timbre and/or pitch of audio signals. [0002] In any periodic musical note, there is a funda: ‘mental frequoncy that datermines the particular pitch of the note, as well as numerous harmonies which provide character or timbre to the note. Ii the particular com bination ofthe harmonic frequencies withthe fundamen tal frequency that make, for example, a guitar and avi alin playing the same note sound diferent fram one an ‘ther, The relationship of the amplitude of the funda: ‘mental frequency component to the amplitudes of the harmonics created by an instrument or voice is referred to as the spectral envelope. In @ musical instrument such as a gular, flute, or saxophone, the spectral enve- lope of a note played by the instrument expands and Contracts more or less proportionally as the pitch of tho note is shifted up or down, [0003] Electronic pitch shifters are musical effects that receive an input note and produce an output note With a ifferent pitch, Such effects are often used to al: low a single musician to sound ike several. For musical Instruments, one can change th pitch of anote by sam: pling and recording the sound from the instrument and playing back the sampled and recorded sounds ata rate thats faster o slower than the rate at which the samples wore recorded. The output notes created by this tech nique sound fairy natural because the spectral enve- lope ofthe pitch shed sounds mimics how the spectral ‘envelope ofthe sounds produced by the instrument var ies with pitch [0004] in contrast to notes produced by musical in struments, the spectral envelope of vocal notes or sounds does not vary proportionally asthe pitch of the vocal note varias. However, the relative magnitudes ot the individual frequencies thal make up this spectral en velope may change. Shifting the pitch of a vocal note by ‘sampling a note as itis sung or spoken and playing the ‘samples back a a diferent speed, does not sound nat: tural because the process varies the shape ofthe spec: tral envelope in proportion to the amount of pitch shit. To shift realistically the pitch ofa vocal sound, amethod is required for varying the fundamental frequency while only slightly varying the overall shape ofthe spectral en velope. [0005] A device that shifts the pitch of vocal notes to Create harmonies in real time is described in our prior US. Patent No. 5,231,871 (Ihe "'671 patent). The ‘methodof pitch shiling describedin the'671 patent was ‘adapted trom an article, Lent, K. “An Efficient Method for Pitch Shifting Digtally Sampled Sounds,” Computer ‘Music Journal, Volume 13, No. 4, (1989) (the "Lent ‘method’. The Lentmethodallows the pitch ofalgitally ‘sampled sound tobe shifted without changing the spec: tral envelope. Briefly stated, the Lent method can be used to shift the pitch of a vocal note by replicating por: tions of a stored input signal at a rate that is faster or slower than the fundamental frequency input note. While this method of shifting the pitch of vocal notes works wel, the pitch shifted notes do not sound com pletely natural, because the spectral envelope remains fixed as the pitches of the notes are varied. [0008] As deseribed above, there are two methods of electronically shifting the pitch ofa note. The first meth- 04, referted to as resampling, modifies the spectral en- velope in proportionto the amount of pitch shit, The sec- ‘ond, Lent method more or less maintains the spectral ‘envelope regardless of tha amount of pitch shift. Nethar ‘of these two methods allow the spectral envelope to be varied in @ controllable manner. Therefore, thera is @ ‘eed for a method of altering the spectral envelope of a ‘musical note that is not dependent on the pitch of anote, With such a method, more realistie harmonies can be Created. In addition, by changing the timbre of the note With or without changing the output pitch, i is possible to make one instrument sound lke another, or one per- son's voice sound ike another [0007] To shit the timbre of both vocal notes and ‘notes produced by musical instruments, the present in- vention uses a navel combination of pitch shifting by al- tering the sampling rate ofa signal and pitch shifting ac cording othe Lent method, The method and apparatus ofthe present invention s further disclosedinthe claims, Inthe preferred embodiment, the input signalis sampled ata fist rate, and the resulting digital representation is stored in amomory buffer. The stored digital input signal 's then resampled at a second rate that is determined by a user. The resampled input signal is then stored in ‘second memory buffer. The pitch of the resampled in- pout signals then shttedby scaling the resampled input signal with a window function ata rate equal to the fun damental frequency of the output note dosired, Itt is dsiredto only shiftthe timere ofa note andnot the pitch ‘of a note, then the rate at which the resampled input signal is scaled with the window function isthe same as the fundamental frequency of the input note. If is de- sired to change the pitch of the output note as well as its timbro, then the rato at which the resampled input signal is scaled with the window function ciffers from the fundamental frequency of the input nate. [0008] According to another aspect ofthe invention, fan effect generator s disclosed that can mediy the tin bre andior pitch ofan input audio signal to match a pteh recewved on a MIDI channel. Proferably, the effect gen- crator is used with @ MIDI karaoke system that provides a stream of melody or harmony notes to the effect gen- trator. The elfect generator reads the notes on the MIDI channel and automatically assigns the note an amount (of timbre shit. The assignment can be made by com paring the pitch of the harmony note with one or more thresholds or with the pitch of an input audio signal re- Cclved from a user of the karaoke system. The amount of timbre assigned to each note can make the harmony ‘notes sound diferent from the input audio signal or can3 EP 0750776 B1 4 ‘mimic how the input auaio signal would change i raised or lowered in pitch [0003] The foregoing aspects and many ofthe attend- ‘ant advantages ofthis invention will become more read: lly appreciated as the same becomes better understood by reference to the following detalled description, when taken in conjunetion with the accompanying drawings, wherein: FIGURES 1A-1D are representative graphs of the spectra of vocal signals showing how the spectral fenvolopes change as a result of prior at timora/ pitch shiting techniques as wel as the timbralpiteh shifting technique of the present invention FIGURE 2A is flow chart of the steps performed by the present invention to shift the timbre and/or pit of an input note; FIGURE 28 is a flow chart of the steps performed by tho present invention to create timbre shifted, harmony notes from an input vocal note; FIGURE 3 is a block diagram of a musical effect ‘generator for producing vocal harmonies according tothe method ofthe present invention: FIGURE 4A and FIGURE 4B are graphs and corre sponding diagrammatic memory charts showing how an input vocal signal is resampled according to step of the method ofthe present invention; FIGURE 5 is abiock diagram showing the functions pporformed by a cigital signal processor that is pro: {grammed according to the method of the present invention: FIGURE 6s a block diagram showing the functions performedby a windowed audio generator unit with: in the digital signal processor; FIGURES 7A and 78 aro a graphic representations ‘ofthe method of shiting the pitch of a cigtally sam- pled vocal signal according othe present invention; FIGURES 8A and 88 show how a Hanning window is created and stored in memory in the method of the present invention; and FIGURES 9A and 9B are block diagrams of music cffects that dynamically select the amount of timbre shift that is applied to a note. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT [0010] Tho present invention provides a system for shiing the timore of a note in a way that sounds more realistic than timbre shits produced by known systems, Inits simplest form, the method can be used to shift the timbre of a note but nat the pitch of note, For example, the method can be used to make a vocal signal sung or ‘spoken by aman sound as ifthe same note were sung ‘or spoken by a woman. In addition to shiting the timbre fof a note, the method of the present invention can be used to change the pitch and timbre of a note. For ex- ‘ample, the present invention canbe usedtomakea note sung by a woman sound like another note sung by @ ‘man. Finally, the presently preferred embodiment of the invention is sed to create timore shifted, harmony ‘notes from an input note. Athough the fllowing descrip tion is primarily directed to producing harmony notes from an input vocal note, t willbe realized that the note ‘nged not be a vocal note but may be produced from any soutce, and the output note need not be different from (or harmonious with the input pitch, [0014] FIGURES 1A-1D compare how the spectral ‘envelope of a voeal nate changes when the pitch ofthe note is shifted according to prir art techniques and by the method ofthe present invention. FIGURE 1A shows ‘frequency spectrum 30a thats representative of atyp- ical vocal note. The overall shape ofthe spectrum is de~ fined by one or more formants or peaks 32a, The char- ‘acter or timbre ofthe vecal note is definedby the relative ‘magnitude and postion ofthe fundamental froquency of the note and the harmonies (represented by the arrows 34a). [0012] To shift realistically the pitch of a vocal note, t Is necessary to shift the fundamental frequency of anote ‘while maintaining the formants ofthe spectrum close to those of the original vacal note. FIGURE 1B shows spectrum 306 of a pitch shifted vocal note that is a mu sical fifth below the note associated with the spectrum shown in FIGURE 1A, The note associated with the spectrum Sob was created by slowing the playback rate cof the sampled original vocal note. As can be soon, tho entire spectral envelope definedby the formants 22b as oll as the individual harmonics 34b is compressed and shited to a lower frequency. The result of shifting the formants makes the pitch shifted vocal note sound un- ratural [0013] FIGURE 1c shows a spectrum 20¢ of a pitch shifted vocal note that is a musical ith below the note associated with spectrum shown in FIGURE 1A and \which was generated in accordance with the method set forth in the '671 patent. The pitch shifted vocal note as- sociated with the spectrum 30c was created by replica: Ing aportion of the input vocal note ata rato that is slow: cr than the fundamental frequency of the original input vocal note. In the spectrum 80, only the frequencies of the harmonics 84c have changed, as described in the ‘671 patent. The overall shape ofthe spectrum remains the same as the spectrum shown in FIGURE 1, The Pitch shifted vocal note associated with the spectrum 0c sounds more natural than the pitch shifted vocal note producedby the note associated withthe spectrum ‘308 shown in FIGURE 1B. However, the pitch shifted vocal note does not sound completely natural. Pitch shifted vocal notes produced by the method described Inthe'67' patent tend to have timores that are vary sim ilar tothe input vocal signal from which they are created. Therefore, all the pitch shifted vocal notes sound like altered variations ofthe original [0014] To ater the timbre of a note in @ manner that Sounds realistic, the present invention uses @ novel