Dynamic Time Warping (DTW)

J.-S Roger Jang ( 張智星 )

jang@mirlab.org
http://mirlab.org/jang
MIR Lab, CSIE Dept
National Taiwan University
 Dynamic Time Warping

 Goal
 To align two sequences under certain
constraints, such that the distance
between these two sequences is as small
as possible.
 Method
 Dynamic programming

-2-
Distance between Same-length Sequences

x   x1 , x2 , xn  and y   y1 , y2 , yn 
 Distance between  

 n
  xi  yi , p  1
1
 i 1
 n p p 
Lp  norm( x, y )     xi  yi     n
 
 xi  yi  , p  2
2
 i 1 
i 1

max xi  yi , p  
 i
 Alignment x   x1 , x2 , x3 , x4 , x5 , x6 , x7 , x8 , x9 
y   y1 , y2 , y3 , y4 , y5 , y6 , y7 , y8 , y9 


-3-
Distance between Different-length Sequences
x   x1 , x2 , xm  , y   y1 , y2 , yn 
 

We want to find an alignment path   p , q  ,  p , q  , p , q  

1 1 2 2 k k

(subject to the alignment constraints), such that the distance along the
path is minimized:
k

 x 
2
dist ( x, y )  min pi  yqi ,
 pi , qi  , i 1k i 1

with the bounary conditions:

Anchored beginning:  p1 , q1    1,1

 Anchored end:  pk , qk    m, n 

-4-
 Alignment Constraints: Type 1

 Temporal constraints
p1  p2    pk , q1  q2    qk
 Other alignment constraints
 One-to-one mapping
 No consecutive skip-over
Alignment path =
x1 x2 x3 x4 x5
  1,1 ,  2,3 ,  3,5 ,  4, 6  ,  5,8  
y1 y2 y3 y4 y5 y6 y7 y8

-5-
 Alignment Constraints: Type 2

 Temporal constraints
p1  p2    pk , q1  q2    qk
 Other alignment constraints
 1-to-1, 1-to-many, or many-to-1 mapping
 No skip-over
Alignment path =
x1 x2 x3 x4 x5   1,1 ,  2, 2  ,  2,3 ,  3, 4  ,  3,5  ,  4, 6  ,  5, 7  ,  5,8  
y1 y2 y3 y4 y5 y6 y7 y8

-6-
 Type-1 DTW: Table Fillup
x, y: input vector/matrix
j Local paths: 27-45-63 degrees

DTW formulation:
D (i, j ) 1. Optimum-value function D(i, j ):
y(j)
y(j-1) The DTW distance between x(1: i ) and y (1: j )
2. Recurrent equation for D(i, j ) :
 D(i  1, j  2) 
 
D(i, j )  x(i )  y ( j )  min  D(i  1, j  1)  ,
 D(i  2, j  1) 
 
with D (1,1)  x(1)  y (1) .

x(i-1) x(i)
3. Answer  D(len( x), len( y ))
i
-7-
 Type-2 DTW: Table Fillup
x, y: input vector/matrix
j Local paths: 0-45-90 degrees

DTW formulation:
y(j) D (i, j ) 1. Optimum-value function D(i, j ):
y(j-1) The DTW distance between x(1: i) and y(1: j )
2. Recurrent equation for D(i, j ) :
 D (i, j  1) 
 
D(i, j )  x(i )  y ( j )  min  D(i  1, j  1)  ,
 D (i  1, j ) 
 
with D(1,1)  x(1)  y (1) .
x(i-1) x(i) i 3. Answer  D(len( x), len( y ))

-8-
 Local Path Constraints
 Type 1:  Type 2:
 27-45-63 local paths  0-45-90 local paths
D i, j 

D i, j 
D i  1, j 

D i  2, j  1 D i  1, j  1
D i, j  1
D i  1, j  1

D i  1, j  2 

D(i, j )  x(i )  y ( j )  D(i, j )  x(i )  y ( j ) 

 D (i  1, j  2)   D(i, j  1) 
   
min  D(i  1, j  1)  min  D(i  1, j  1) 
 D (i  2, j  1)   D(i  1, j ) 
   
-9-
 Path Penalty for Type-1 DTW

 Alignment path of type-1 DTW

 45-degree paths are likely to be avoided since we
are minimizing the total distance.
 So we can add penalty for paths deviated from 45-
degree. D i, j 


 D(i  1, j  2)    0 
  D i  1, j  1
D(i, j )  x(i )  y ( j )  min  D(i  1, j  1)  D i  2, j  1

 D(i  2, j  1)   
 
D i  1, j  2 

-10-
 Path Penalty for Type-2 DTW

 Alignment path of type-1 DTW

 45-degree paths are likely to be taken since we are
minimizing the total distance.
 So we can add penalty for paths of 45-degree.
0 D i, j 
D i  1, j 
 D(i, j  1)  
  0
D(i, j )  x(i )  y ( j )  min  D(i  1, j  1)    D i, j  1
 D(i  1, j )  D i  1, j  1
 

-11-
 Other Minutes about DTW

 Typical applications
 Speech recognition: MFCC matrices as inputs
(where x(i) is the MFCC vector of frame i)
 Query by singing/humming: Pitch vectors as
inputs (where x(i) is the pitch value of frame i)
 Abundant variants for various applications
 Recurrent formulas
 Local path constraints

-12-
Applications

 Applications of DTW
 DTW for speech recognition
 DTW for query by singing/humming

-13-