You are on page 1of 70

US007940914B2

(12) Unlted States Patent (10) Patent N0.2 US 7,940,914 B2


Petrushin (45) Date of Patent: May 10, 2011

(54) DETECTING EMOTION IN VOICE SIGNALS (56) References Cited


IN A CALL CENTER
' ' U.S. PATENT DOCUMENTS
(75) Inventor: Valery A- Petrushln: Arhngton Helghts, 5,572,576 A 11/1996 Klausner et a1 " 379/67
IL (US) 5,918,222 A 6/1999 Fukui et al.
_ 6,212,550 B1 4/2001 Segur .......................... .. 709/206
(73) Ass1gnee: Accenture Global Services Limited, 6,638,217 B1 10/2003 Liberman ................... .. 600/300
Dublin (113) 6,795,808 B1 9/2004 Strubbe et al.
7,181,693 B1 2/2007 Anderson et al. ........... .. 715/745
(*) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 OTHER PUBLICATIONS
U~S~C- 15403) by 3093 days- Petrushin, V. Emotion in Speech: Recognition and Application to
Call Centers, Arti?cial Neural Networks in Engineering, Nov. 7-10,
(21) Appl. No.: 09/833,301 1999
_ Foreign communication/of?ce action dated Dec. 3, 2008 (in Hebrew
(22) Flled: Apr 10 2001 and English translation citing above US. Patent No. 5,572,576 (5p).
65 P ' P bl' t' D t
( ) nor u lea Ion a a Primary Examiner * Harry S Hong
Us 2003/0033145 A1 Feb- 13 2003 (74) Attorney, Agent, or Firm *Brinks Hofer Gilson &
L.
Related US. Application Data lone
(63) Continuation of application No. 09/388,909, ?led on (57) ABSTRACT
Aug. 31, 1999, noW Pat. No. 6,275,806. _ _
A system, method and article of manufacture are prov1ded for
(51) Int, Cl, detecting emotion using statistics. First, a database is pro
H04M 3/51 (200601) vided. The database has statistics including human associa
H04M 3/523 (200601) tions of voice parameters With emotions. Next, a voice signal
G101, 17/00 (200601) is received. At least one feature is extracted from the voice
(52) US. Cl. ............ .. 379/265.06; 379/265.07; 704/270; Signal Then the extracted V0106 feature is Compared to the
704/E17_002 voice parameters in the database. An emotion is selected from
(58) Field of Classi?cation Search 379/88_01i88_04 the database based on the comparison of the extracted voice
379/112.01, 142.01, 265.02*265.08, 266.01*266.03, feature to the voice parameters and is then output.
379/309; 704/270, 272, E17.002
See application ?le for complete search history. 3 Claims, 35 Drawing Sheets

RECEIVING AVOICE SIGNAL /200

EXIRACTING AT LEAST ONE FEATURE OF THE VOICE SIGNAL /202

DETERMINING AN EMOTION ASSOCIATED WITH THE VOICE /204


SIGNAL BASED ON THE EXTRACTED FEATURE

OUTPUTTING THE DETERMINED EMOTION


US. Patent May 10, 2011 Sheet 1 0135 US 7,940,914 B2

gomz
1
E
2
E
0:
Q:
.\ $523, f

3
EE

@5228
20
Em2
aw r
1

5%Em a$52; E-2E:HI FTJ


my5 FGE
\ N:
US. Patent May 10, 2011 Sheet 2 0135 US 7,940,914 B2

RECEIVINGAVOICE SIGNAL /200

EXTRACTINGAT LEAST ONE FEATURE oETHE VOICE SIGNAL /202

DETERMININGANEMOTIONASSOClATED WITH THE VOICE /2o4


SIGNAL BASED ON THE EXTRACTED FEATURE

OUTPUTTING THE DETERMINED EMOTION /206

FIG. 2
US. Patent May 10, 2011 Sheet 3 0f 35 US 7,940,914 B2

.2I5T02 E{0m2l:

6E
m
SA3aoi01ME a":OE2":ON":

(Wmvanoov
US. Patent May 10, 2011 Sheet 4 0135 US 7,940,914 B2

.ZIEmToz 5l%mT: >mLoTz.< Q<wlxl 9I&x.2 .5I2T.

_
_

ioiE 2":CEa":

3
ocoooooooo
OODwNCDWQ'C'DNF'

(wovanoav
C)
US. Patent May 10, 2011 Sheet 5 0135 US 7,940,914 B2

NOR+M-AL HAmP~Y AN+GRY SAD-><- AFR*AlD TOT-AL

FIG.
5

gageaasaaso
(%)mvano0v
US. Patent May 10, 2011 Sheet 6 0135 US 7,940,914 B2

PRDvIDING A DATABASE HAvING STATISTICS INCLUDING


STATISTICS OF HUMAN ASSOCIATIONS OF VOICE PARAMETERS /600
WITH EMoTIoNS

RECEIVING AvoICE SIGNAL /602

EXTRACTINGAT LEAST ONE FEATURE OF THE VOICE SIGNAL /604

COMPARING THE ExTRACTED voICE FEATURE T0 THE VOICE A06


PARAMETERS IN THE DATABASE

SELECTING AN EMDTIDN FROM THE DATABASE BASED ON THE


COMPARISON OF THE EXTRACTED voICE FEATURE ToTHE /608
voICE PARAMETERS

DUTPUTTING THE SELECTED EMDTIDN /610

FIG- 6
US. Patent May 10, 2011 Sheet 7 0135 US 7,940,914 B2

RECEIVING VOICE SIGNALS FROM A PERSON DURING A fym)


BUSINESS EVENT

ANALYZING THE VOICE SIGNALS FOR DETERMINING ALEVEL OF [702


NERVOUSNESS OF THE PERSON DURING THE BUSINESS EVENT

OUTPUTTING THE LEVEL OF NERVOUSNESS OF THE PERSON [704


PRIOR TO COMPLETION OF THE BUSINESS EVENT

FIG. 7
US. Patent May 10, 2011 Sheet 8 0135 US 7,940,914 B2

804 806 808


/ / /

FILTER FM DISCRIMINATOR DETECTOR

s00 / l

WAVEFORM GATE 1DCAMP. PROCESSOR\ ouTPuT 0Ev|cE\


812 814 816

FIG. 8

92 904 906
V
908
r/
900
PROCESSOR\
918

910 916

912 914

FIG. 9
US. Patent May 10, 2011 Sheet 9 0135 US 7,940,914 B2

RECEIVING AVOICE SIGNAL REPRESENTATIVE OFA COMPONENT fmo


OFACONVERSATION BETWEEN AT LEAST TWO SUBJECTS

II

DETERMINING AN EMOTION ASSOCIATED WITH THE VOICE / 1002


SIGNAL

II

PROVIDING FEEDBACK TOATHIRD PARTY BASED ON THE /1004


DETERMINED EMOTION

FIG. 10
US. Patent May 10, 2011 Sheet 10 0135 US 7,940,914 B2

PROVIDING AVOICE SIGNALANDAN EMOTION ASSOCIATED f1100


THEREWITH

AUTOMATICALLY DETERMINING THE EMOTION ASSOCIATED WITH /1102


THE VOICE SIGNAL

STORING THEAUTOMATICALLY DETERMINED EMoTIoN M104

RECEIVINGAUSER DETERMINED EMoTIoN ASSOCIATED WITH fm?


THE VOICE SIGNAL DETERMINED BYA USER

COMPARING THE AUTOMATICALLY DETERMINED EMOTION WITH / 1103


THE USER DETERMINED EMOTION

FIG. 11
US. Patent May 10, 2011 Sheet 11 0135 US 7,940,914 B2

1211 1212 1213 1214

6+ M1CROPHONE __> TAPE LNPUT + LNPUT LEVEL + ELL-[ER DRWER


AMPLIFIER JACK CONTROL AMPLIFIER

1218 1217 1215 1215


v / / / V /
SAMPLE START SWEEP RATE AND
FILTER BANK AND SWITCH AND STOP SAMPLE TIME STYLE DIMENSION
1219/1- DIGITIZER OVERRIDE SWITCH ASSEMBLY

1220 1221 1225 12.24


PEAK FREQUENCY Z COMPRESSION
REPRESENTATION
SIGNAL Z ASSEMBLYAND
ELEMENT Z sTYkgggwlEB?'oN 3
CORRECTION AND STORAGE STORAGE
J
/
1226
l
/
1228 1227 1222 1223
/ v / / /
RESEARCH
OUTPUT OICITAL OUTPUT MEASUREMENT RESEARCH __
<- SELECTION COEFFICIENT
READOUT SELECTION
SWITCH SWITCH KEYPAD

FIG. 12
US. Patent May 10, 2011 Sheet 12 0135 US 7,940,914 B2

E SPECTRUM SPREAD \
' ANALYSIS AND STORAGE

1331
E SPECTRUM BALANCE
' ANALYSIS AND STORAGE

V
SPECTRUM ENVELOPE \
ANALYSIS AND STORAGE

SIGNAL STYLE
COMPRESSION \ V VARAB'LTY \ \> DIMENSION
AND STORAGE ANALYS'S AND STORAGE ASSEMBLY

V
PAUSE
ANALYSIS AND STORAGE

_ SYLLABLE CHANGE \
T ANALYSIS AND STORAGE

V
HIGH FREQUENCY \
ANALYSIS AND STORAGE

FIG. 13
US. Patent May 10, 2011 Sheet 13 0135 US 7,940,914 B2

141o\_ D'SPLAY \1412


PC

AUDIO J41?
CARD

RS232

1432\ PRE- MICROPHONE N418


PROCESSOR

KYPAD
/
1439
BIO- aim
MONITOR \1430
1435
.

FIG. 14

1542
BIO-SIGNAL --------
US. Patent May 10, 2011 Sheet 14 0135 US 7,940,914 B2

012011011 1 : 0501101112
T P 02
1050 _ _ +V
+ 14@ g |B_Q2 l($.Q2
: 27K
_ L1
""6 1430 1 16471
1011; 5 J01 31' ?k
668
1
22K ~
5 0.040
c2 47"
1010
Ar 1000500K VR acfmvxx; __1@___
0.010 _
1004 'i < 1 1
1K l11.01 ) 5 Q3 V Cm 1548

+ cs 5 (Q-3) 1K
(TBv $33K $1111: 5 '
_v_1002 5 \
V0.00
FIG. 16

TO AUDIO FROM
CARD 1417 1790 1786 1780 M'CRg?glONE
I [jg Ci] 01011111 SPEECH in
W 0101110 '
1
1700 1104 1702
MEMORY j 01011111 010010111111
"39 FROM
< RS232 m 010-1101111012
1011c 143
1410

FIG. 17
US. Patent May 10, 2011 Sheet 15 0135 US 7,940,914 B2
US. Patent May 10, 2011 Sheet 16 0135 US 7,940,914 B2

l q,
|_N|T l
YES 1930/1 SETTOOFORALLT J
1932K BEST TEsT NO
N YES
1934 /I DETERMNETREH J
T

1938/! REQUES'SAMPLE J
1940 /{ SAMPLEIEAPITCH J
r
1942 /| INSTRUCT DsPTo INCREASE PITCH I YES
REQUESTTESHAND WAIT FOR END
1944/ COMMAND

1946 /l INSTRUCT DSPTO+JECREMENT PlTCH J


REQUEsTTEsT2AND WAIT FOR END
1943/ COMg/IAND
19% 1950/! REQUEST \OTE1OR2 1 1956
sET REF1P|TCH= 2 1 sETREE1P|TcH=
oLD-DEDREMENT {1952) BEST TEST >_ OLD+DEIREMENT
1962\1 ESTABLISH PITDNNDDTETDATIDN LINE J
L/ D'SABLE /<I& MDDTEYDDMNAND )/19
COMMAND /< VALUE oF+TYEgEE1v
= . )-__->
41966 N0 958 1N0
19m /{ ESTABLISH +POINT REEz T
1974 /{ INSTRUCT DSP H1 lNCREASE PITCH J
REQUEST TEsT 1 AND WAIT FOR END
1975/ COMMAND
V

1978 /| INSTRUCT DSP T0 +:TECRENENT PITCH ]


REQUEST TEsT2AND WAIT FOR END

19 a
1930/ COMQAAND
a 1982 /1 REQUESHOTHORZ J 1986
sET REF2PITCH= 2 1 sET REF2P|TCH=
DLD-TNDRENENT 75$ BESTTEST >_ OLD+INCREMENT J

1990
\{TSTABLISH NEW PITCIHV MODIFICATION LINE 1 F | G_ 1 9
US. Patent May 10, 2011 Sheet 17 0135 US 7,940,914 B2

PROVIDING A VOICE SIGNAL ANDAN EMOTION ASSOCIATED / 2000


THEREWITH

AUTOMATICALLY DETERMINING THE EMOTION ASSOCIATED WITH / 2002


THE vOICE SIGNAL

II

STORING THE AUTOMATICALLY DETERMINED EMOTION /2004

RECEIVING AuSER DETERMINED EMOTION ASSOCIATED WITH [2005


THE vOICE SIGNAL DETERMINED BYA USER

COMPARING THE AUTOMATICALLY DETERMINED EMOTION WITH [2003


THE USER DETERMINED EMOTION

FIG. 20
US. Patent May 10, 2011 Sheet 18 0135 US 7,940,914 B2

PROMPTING A USER FOR A VOICE SAMPLE OvER A NETWORK / 2100

RECEIVING THE VOICE SAMPLE FROM THE USER OvERTHE [2102


NETWORK

RETRIEVING REGISTRATION INFORMATION ABOUT THE USER 2104


INCLUDINGAVOICE SCAN OVER THE NETWORK /

COMPARING THE VOICE SAMPLE AND THE VOICE SCAN FOR / 2106
VERIFYING AN IDENTITY OF THE USER

GRANTING DATAACCESS TO THE USER OVER THE NETWORK IF / 2103


THE IDENTITY OF THE USER IS VERIFIED

DENYING DATAACCESS TO THE USER IF THE IDENTITY OF THE / 2110


USER IS NOT VERIFIED

FIG. 21
US. Patent May 10, 2011 Sheet 19 0135 US 7,940,914 B2

2220 2224
/ /
SPEAKER > SECURITY CENTER
2222
/
> SECURED SYSTEM <

2226
/
POSmVE ID

FIG. 22

2362 2350 2364


/ 531% j / 239%
I /2366 I

4358a _ / ~2358b
2354a!" \\ 2354b / I,

L 2352 J

REMOTE COMM. /2356


MODE

FIG. 23