Recursion and Divide-and-Conquer: CS2800: Design and Analysis of Algorithms

Recursion and Divide-and-Conquer
CS2800: Design and Analysis of Algorithms
Yadu Vasudev
yadu@cse.iitm.ac.in
IIT Madras
Module plan
1. Asymptotic analysis
2. Integer Multiplication Redux
3. Order statistics
4. Fast Fourier Transform
5. Closest pair of points
1
Asymptotic analysis
Recall: Asymptotic analysis
• Often, we are only interested in the the rate of growth of a

function w.r.t to other functions
2

• Does 2n grow faster than n100 ?
• Does 2n grow faster than 20.9n ?
2

• For an algorithm A, which function f(n) captures the rate of
growth of the running time of A the best?
2

Definitions of O, Ω, and Θ
• A function f(n) = O(g(n)) if ∃c > 0 and n0 ≥ 1 such that
f(n) ≤ cg(n), ∀n ≥ n0
• A function f(n) = Ω(g(n)) if ∃c > 0 and n0 ≥ 1 such that
f(n) ≥ cg(n), ∀n ≥ n0
• A function f(n) = Θ(g(n)) iff f(n) = O(g(n)) and f(n) = Ω(g(n))
2

Little-oh
• A function f(n) = o(g(n)) if ∀c > 0, there exists an n0 s.t
f(n) < cg(n), ∀n ≥ n0
2

Little-oh
f(n) < cg(n), ∀n ≥ n0
• Equivalently,
f(n)
lim =0
n→∞ g(n)
2

Little-oh
f(n) < cg(n), ∀n ≥ n0
• Equivalently,
f(n)
lim =0
n→∞ g(n)
• f(n) is a strictly slower growing function than g(n)
2

Little-omega
• A function f(n) = ω(g(n)) if ∀c > 0, there exists an n0 s.t
f(n) > cg(n), ∀n ≥ n0
• Equivalently,
f(n)
lim =∞
n→∞ g(n)
• f(n) is a strictly faster growing function than g(n)
2
• O is like ≤, o is like <, Θ is like =, but...
3
Consider a hypothetical algorithm for checking primality of a

number n
• If n is even (check the LSB), answer ”no”

• Else run an algorithm with running time log n
3

number n

(
1 if n is even,
f(n) =
log n otherwise
3

number n

(
1 if n is even,
f(n) =
log n otherwise
• f(n) = O(log n), but not Θ(log n)

• f(n) = Ω(1), but not Ω(g(n)) for any function that grows with n
• f(n) 6= o(log n)
3
Some interesting functions
4
·2
··
22
• non-elementary functions - f(n) = 2 : height of the tower of
2s depends on the input size
4
·2
··
22
• iterated logarithm - log∗ n: number of times log has to be taken
before n is at most 1
• log∗ n ≤ 4 for all practical values of n
4
·2
··
22
• iterated logarithm - log∗ n: number of times log has to be taken
before n is at most 1
• log∗ n ≤ 4 for all practical values of n
• Ackermann function
• α(n, 0) - increment by 1
• α(n, 1) - perform incremement n times: addition
• α(n, 2) - perform addition n times: multiplication
• α(n, 3) - perform multiplication n times: exponentiation
• α(n, 4) - perform exponentiation n times: ? . . .
4
Integer Multiplication Redux
Recall: Peasant multiplication
5
Input: Integers a and b

Output: c = a · b
Set c ← 0
repeat
if a is odd then c ← c + b;
b←2·b
a ← b a2 c
until a = 0;
Running time:
5

Output: c = a · b
Set c ← 0
repeat
b←2·b
a ← b a2 c
until a = 0;
Running time:
• The main while loop runs for log a many iterations
5

Output: c = a · b
Set c ← 0
repeat
b←2·b
a ← b a2 c
until a = 0;
Running time:

• Each operation in the while loop takes at most log a + log b steps
5

Output: c = a · b
Set c ← 0
repeat
b←2·b
a ← b a2 c
until a = 0;
Running time:

• Each operation in the while loop takes at most log a + log b steps
• O(n2 ) time to multiply two n-digit numbers
5
Faster integer multiplication
Recursive solution:
(10n/2 a + b )(10n/2 c + d ) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )
6
Recursive solution:
Numbers with n/2 digits
6
Recursive solution:
Numbers with n/2 digits Recursive calls
6
Recursive solution:
Correctness: From the definition of multiplication
6
Recursive solution:
Correctness: From the definition of multiplication

Running time?
6
Analyzing recurrence relations
(10n/2 a + b)(10n/2 c + d) = 10n (ac) + 10n/2 (ad + bc) + (bd)
7
n
T(n) = 4 T( ) + O(n)
2
7
n
T(n) = 4 T( ) + O(n)
2
# recursive calls adding the sub-products
running time on inputs of size n

size of sub-problems
7
Analyzing recurrence relations: Recursion trees
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
8
n cn
n n n n
2 2 2 2 2cn
n n n n n n n n
4 4 4 4 ··· 4 4 4 4 ··· 22 cn
.. .. .. .. .. .. ..
. . . . . . .
2log n cn
Running time: cn(1 + 2 + 22 + · · · + 2log n ) = O(n2 )
8
Recursion trees: The master theorem
Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

n f(n)
n n n n
b b b b af( bn )
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
logb n−1 n

T(n) = Θ(nlogb a ) + ai f
X
bi
i=0 9

n a f(n)
n n n n
b b b b af( bn )
a a
n n n n n n n n
b2 b2 b2 b2 ··· b2 b2 b2 b2 a2 f( bn2 )
.. .. .. .. .. .. ..
. . . . . . .
alogb n f(1)
logb n−1 n

X
bi
i=0 9
When
logb n−1 n
X
,
bi
i=0
10
When
logb n−1 n
X
,
bi
i=0
• If f(n) = O(nlogb a− ), then T(n) = Θ(nlogb a )

• If f(n) = Θ(nlogb a ), then T(n) = Θ(nlogb a log n)
• If f(n) = Ω(nlogb a+ ) and af(n/b) ≤ cf(n) for some
constant c < 1, then T(n) = Θ(f(n))
10
When
logb n−1 n
X
,
bi
i=0
• If f(n) = O(nlogb a− ), then T(n) = Θ(nlogb a )

• If f(n) = Θ(nlogb a ), then T(n) = Θ(nlogb a log n)
• If f(n) = Ω(nlogb a+ ) and af(n/b) ≤ cf(n) for some
constant c < 1, then T(n) = Θ(f(n))
Points to remember:
• Floors and ceilings are not important in proving the upper
bounds (for most cases)
• Use recursion trees directly when the master theorem cannot
be applied
10
Integer multiplication: Karatsuba’s method
11
Karatsuba’s observation: bc + ad = ac + bd − (a − b)(c − d)
11
(10n/2 a + b)(10n/2 c + d) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )
11
(a − b)(c − d)
11
(a − b)(c − d)
Running time: T(n) = 3T(n/2) + O(n)
11
(a − b)(c − d)
Is the master theorem applicable here?
11
(a − b)(c − d)
Is the master theorem applicable here? T(n) = Θ(nlog 3 )
11
Integer multiplication: A short history
12
Integer multiplication: A short history
Peasant multiplication - O(n2 ) 1600 BC

2
Lattice multiplication - O(n ) 1200
Karatsuba’s method - O(nlog 3 ) 1960
Schönhage-Strassen - O(n log n log log n) 1971
Conjecture: Θ(n log n)
∗
n)
Fürer/De-Saha-Kurur-Saptharishi - O(2Θ(log n log n) 2008
∗
Harvey-van der Hoeven - O(22 log n
n log n) 2018
Harvey-van der Hoeven - O(n log n) 2019
“...our work is expected to be the end of the road for this problem, al-
though we don’t know yet how to prove this rigorously.“
12
Order statistics
Order statistics
Given an array A[1, 2, . . . , n], find the kth element in the sorted array
13
Order statistics
• Sort the array, and return A[k]
13
Order statistics
• Sort the array, and return A[k] O(n log n)
13
Order statistics
• Sort the array, and return A[k] O(n log n)

• If we find a pivot partitioning A into two parts of size αn and
(1 − α)n in O(n) time, then we have a recursive algorithm with
running time
T(n) = T(αn) + O(n)
13
Median of medians
14
Median of medians
n
5
• Divide the array A into n/5 subarrays of length 5 and find their
medians
14
Median of medians
n
5
medians
• How good is the median of these medians as a pivot?
14
Median of medians
n
5
medians
14
Median of medians
n
5
medians
• All the blue elements are ≤ m and all the green elements are
≥ m - How many are there?
14
Median of medians
n
5
3n
10
m
3n
10
medians
• All the blue elements are ≤ m and all the green elements are
≥ m - How many are there?
14
A recursive algorithm
Input: Array A[1, 2, . . . , n], k

if n ≤ 10 then
brute-force search
else
set d ← n/5
for 1 ≤ i ≤ d do
set M[i] ← median of A[5i − 4, . . . , 5i]
set m ← median of M recursively
r ← Partition(A[1, 2, . . . , n], m)
if k < r then
recursively search in A[1, 2, . . . , m − 1] for k
else
if k > r then
recursively search in A[m + 1, . . . , n] for k − r
else
return m
15
A recursive algorithm
Input: Array A[1, 2, . . . , n], k

if n ≤ 10 then
brute-force search
else
set d ← n/5
for 1 ≤ i ≤ d do
set M[i] ← median of A[5i − 4, . . . , 5i]
set m ← median of M recursively T( n5 )
r ← Partition(A[1, 2, . . . , n], m) O(n)
if k < r then
recursively search in A[1, 2, . . . , m − 1] for k
else T( 7n
10 )
if k > r then
recursively search in A[m + 1, . . . , n] for k − r
else
return m
15
Running time - median of medians algorithm
• The running time is given by

n
7n

T(n) = T +T + O(n)
5 10
16

n
7n

T(n) = T +T + O(n)
5 10
• What happens if we write this as
7n

T(n) ≤ 2T +O(n)?
10
16

n
7n

T(n) = T +T + O(n)
5 10
7n

T(n) ≤ 2T +O(n)?
10
O(nlog10/7 2 )
16

n
7n

T(n) = T +T + O(n)
5 10
7n

T(n) ≤ 2T +O(n)?
10
O(nlog10/7 2 )
• Is our algorithm bad or analysis not tight?
16
Running time using recursion trees
n 7n
T(n) = T +T + O(n)

5 10
17
n 7n
T(n) = T +T + O(n)

5 10
n 7n
5 10
n 7n 7n 49n
25 50 50 100
.. .. .. ..
. . . .
17
n 7n
T(n) = T +T + O(n)

5 10
n n
n 7n 9
5 10 10 n
n 7n 7n 49n 9 2
n

25 50 50 100 10
.. .. .. ..
. . . . 9 i
n

≤ 10
17
n 7n
T(n) = T +T + O(n)

5 10
n n
n 7n 9
5 10 10 n
n 7n 7n 49n 9 2
n

25 50 50 100 10
.. .. .. ..
. . . . 9 i
n

≤ 10
2 !
9 9

T(n) ≤ 1+ + + ··· n = O(n)
10 10
17
Fast Fourier Transform
Convolution of vectors
Given a = (a0 , a1 , . . . , an−1 ), b = (b0 , b1 , . . . , bn−1 ), compute the value
a ∗ b = (c0 , c1 , . . . , c2n−1 ), where

X
ck = ai bj
i+j=k
18
a ∗ b = (c0 , c1 , . . . , c2n−1 ), where

X
ck = ai bj
i+j=k
• Polynomial multiplication: If a and b represent the coefficients

of the polynomials, then the convolution is the coefficients of
their product
18
a ∗ b = (c0 , c1 , . . . , c2n−1 ), where

X
ck = ai bj
i+j=k

their product
a = (2, 3, 4) ≡ 2 + 3x + 4x2
b = (1, 2, 3) ≡ 1 + 2x + 3x2
a ∗ b = (2, 7, 16, 17, 12) ≡ 2 + 7x + 16x2 + 17x3 + 12x4
18
a ∗ b = (c0 , c1 , . . . , c2n−1 ), where

X
ck = ai bj
i+j=k

their product
a = (2, 3, 4) ≡ 2 + 3x + 4x2
b = (1, 2, 3) ≡ 1 + 2x + 3x2
a ∗ b = (2, 7, 16, 17, 12) ≡ 2 + 7x + 16x2 + 17x3 + 12x4
• Signal processing: - smoothing discrete measurements,
analyzing response of a linear time-invariant system
18
The naive algorithm:
19
• Compute ck using at most k multiplications and additions, for

k ∈ {0, 1, . . . , 2n − 2}
X
ck = ai bj
i+j=k
19

k ∈ {0, 1, . . . , 2n − 2}
X
ck = ai bj
i+j=k
• Running time: O(n2 )
19

k ∈ {0, 1, . . . , 2n − 2}
X
ck = ai bj
i+j=k
Fast Fourier transform:
19

k ∈ {0, 1, . . . , 2n − 2}
X
ck = ai bj
i+j=k
• Running time of O(n log n)
19

k ∈ {0, 1, . . . , 2n − 2}
X
ck = ai bj
i+j=k
• Running time of O(n log n)

• First described by Gauss in 1805
• Modern version by Cooley and Tukey in 1965
19
Vectors and polynomials
20
Convolution ≡ Polynomial multiplication
20

Pn−1 Pn−1
P(x) = i=0 ai xi , Q(x) = i=0 bi xi ⇒ a ∗ b ≡ P(x)Q(x)
20

Pn−1 Pn−1
P(x) = i=0 ai xi , Q(x) = i=0 bi xi ⇒ a ∗ b ≡ P(x)Q(x)
Naive algorithm: Multiply ai xi with Q(x) for each i and then add
20
Polynomials - an alternate view
A polynomial P(x) of degree n − 1 is uniquely defined by its

evaluation on n points
21

n−1
ai xi
X
P(x) =
i=0
a0 P(α0 )
   
 a   P(α ) 
 1   1 
 a2  ≡  P(α2 ) 
   
vector of coefficients vector of evaluations
 ..   ..
   
 .   .


an−1 P(αn−1 )
21

n−1 n−1
ai xi Q(x) = bi xi
X X
P(x) =
i=0 i=0
a0 b0 P(α0 )Q(α0 )
     
 a   b   P(α )Q(α ) 
 1   1   1 1 
 a2  ∗  b2  ≡  P(α2 )Q(α2 ) 
     
 ..   ..   ..
     
 .   .   .


an−1 bn−1 P(α2n−2 )Q(α2n−2 )
21
Convolution - outline of an algorithm
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0
(a0 , a1 , . . . , an−1 ) P(α0 ), P(α1 ), . . . , P(α2n−2 )

(b0 , b1 , . . . , bn−1 ) Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
P(α0 )Q(α0 ),
P(α1 )Q(α1 ),
P(x)Q(x)
...
P(α2n−2 )Q(α2n−2 )
22
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0
(a0 , a1 , . . . , an−1 ) P(α0 ), P(α1 ), . . . , P(α2n−2 )

(b0 , b1 , . . . , bn−1 ) Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
P(α0 )Q(α0 ),
P(α1 )Q(α1 ),
P(x)Q(x)
...
P(α2n−2 )Q(α2n−2 )
22
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0
(a0 , a1 , . . . , an−1 ) evaluation P(α0 ), P(α1 ), . . . , P(α2n−2 )

(b0 , b1 , . . . , bn−1 ) Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
P(α0 )Q(α0 ),
P(α1 )Q(α1 ),
P(x)Q(x)
...
P(α2n−2 )Q(α2n−2 )
22
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0

(b0 , b1 , . . . , bn−1 ) Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
pointwise product
P(α0 )Q(α0 ),
P(α1 )Q(α1 ),
P(x)Q(x)
...
P(α2n−2 )Q(α2n−2 )
22
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0

(b0 , b1 , . . . , bn−1 ) Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
pointwise product
P(α0 )Q(α0 ),
interpolation P(α1 )Q(α1 ),
P(x)Q(x)
...
P(α2n−2 )Q(α2n−2 )
22
Polynomial evaluation
Pe (x2 ) - polynomial of degree Po (x2 ) - polynomial of degree

at most n2 at most n2
23
Naive evaluation of a polynomial on O(n) points requires O(n2 )

time

23

time
Can we do better by choosing αi s carefully?

23

time
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·

23

time
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·
= (a0 + a2 x2 + a4 x4 + · · · ) + x(a1 + a3 x2 + a5 x4 + · · · )

23

time
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·
= (a0 + a2 x2 + a4 x4 + · · · ) + x(a1 + a3 x2 + a5 x4 + · · · )

23
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·
= (a0 + a2 x2 + a4 x4 + · · · ) + x(a1 + a3 x2 + a5 x4 + · · · )

24
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·
= (a0 + a2 x2 + a4 x4 + · · · ) + x(a1 + a3 x2 + a5 x4 + · · · )

±α0 , ±α1 , ±α2 , . . . , ±αn/2−1
24
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·
= (a0 + a2 x2 + a4 x4 + · · · ) + x(a1 + a3 x2 + a5 x4 + · · · )

±α0 , ±α1 , ±α2 , . . . , ±αn/2−1
P(αi ) = Pe (αi2 ) + αi Po (αi2 )

P(−αi ) = Pe (αi2 ) −αi Po (αi2 )
24
P(x) = a0 + a1 x + a2 x2 + a3 x3 + · · ·
= (a0 + a2 x2 + a4 x4 + · · · ) + x(a1 + a3 x2 + a5 x4 + · · · )

±α0 , ±α1 , ±α2 , . . . , ±αn/2−1
P(αi ) = Pe (αi2 ) + αi Po (αi2 )

P(−αi ) = Pe (αi2 ) −αi Po (αi2 )
evaluation of two n/2-degree polynomials at n/2 points 24

Polynomial evaluation - a “complex” approach
P(αi ) = Pe (αi2 ) + αi Po (αi2 )

P(−αi ) = Pe (αi2 ) −αi Po (αi2 )
evaluation of two n/2-degree polynomials at n/2 points
A divide-and-conquer algorithm works if αi2 are plus-minus

pairs!
25
P(αi ) = Pe (αi2 ) + αi Po (αi2 )

P(−αi ) = Pe (αi2 ) −αi Po (αi2 )

pairs!
4π/n
Complex roots of unity: 2π/n
25
P(αi ) = Pe (αi2 ) + αi Po (αi2 )

P(−αi ) = Pe (αi2 ) −αi Po (αi2 )

pairs!
4π/n
1, ω, ω 2 , . . . , ω i , . . . , ω n−1 1
e2πi/n
25
4π/n
1, ω, ω 2 , . . . , ω j , . . . , ω n−1 1
e2πi/n
26
4π/n
1, ω, ω 2 , . . . , ω j , . . . , ω n−1 1
e2πi/n
n
• For each j, ω 2 +j = −ω j when n is even
26
4π/n
1, ω, ω 2 , . . . , ω j , . . . , ω n−1 1
2π/n + π
e2πi/n
n
n 2πi n
πi 2πi j
ω 2 +j = e n 2 +j =e (e n )
= −ω j
26
4π/n
1, ω, ω 2 , . . . , ω j , . . . , ω n−1 1
2π/n + π
e2πi/n
n
n 2πi n
πi 2πi j
ω 2 +j = e n 2 +j =e (e n )
= −ω j
• (ω j )2 are n2 -th roots of unity
26
Polynomial evaluation - the Fast Fourier Transform
27
Input: polynomial P(x) as a coefficient vector (a0 , a1 , . . . , an−1 )

(n = 2k ), ω - nth root of unity
Output: P(ω 0 ), P(ω), P(ω 2 ), . . . , P(ω n−1 )
27

if ω = 1 then return P(1) base case of recursion
27

express P(x) = Pe (x2 ) + xPo (x2 )
27

Evaluate Pe (ω 2 )
Evaluate Po (ω 2 ) recursive calls - divide step
27

Evaluate Pe (ω 2 )
foreach j ∈ {0, 1, . . . , n − 1} do
P(ω j ) = Pe (ω 2j ) + ω j Po (ω 2j )
combining solutions - conquer step
27

if ω = 1 then return P(1) base case of recursion O(n)
Evaluate Pe (ω 2 )
foreach j ∈ {0, 1, . . . , n − 1} do
P(ω j ) = Pe (ω 2j ) + ω j Po (ω 2j )
27

Evaluate Pe (ω 2 )
Evaluate Po (ω 2 ) recursive calls - divide step 2T( n2 )
foreach j ∈ {0, 1, . . . , n − 1} do
P(ω j ) = Pe (ω 2j ) + ω j Po (ω 2j )
27

Evaluate Pe (ω 2 )
Evaluate Po (ω 2 ) recursive calls - divide step 2T( n2 )
foreach j ∈ {0, 1, . . . , n − 1} do
P(ω j ) = Pe (ω 2j ) + ω j Po (ω 2j )
O(n)
27
Where are we now?
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0

(b0 , b1 , . . . , bn−1 ) O(n log n) Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
pointwise product O(n)
P(α0 )Q(α0 ),
P(x)Q(x)
...
P(α2n−2 )Q(α2n−2 )
28
Polynomial evaluation - a matrix view
n−1
aj ωij
X
P(ω) =
j=0
29
n−1
aj ωij
X
P(ω) =
j=0
P(1) 1 1 1 1 a0
    
···
 P(ω)  1
   ω ω2 ··· ω n−1    a1 
 
 P(ω 2 )  = 1 ω2 (ω 2 )2 ··· 2 n−1  
(ω )   a2 
   
.. .. ..   .. 
    
. . .  . 
  
   ··· ···
n−1
P(ω ) 1 ω n−1 n−1 2
(ω ) ··· (ω n−1 )n−1 an−1
29
n−1
aj ωij
X
P(ω) =
j=0
P(1) 1 1 1 1 a0
    
···
 P(ω)  1
   ω ω2 ··· ω n−1    a1 
 
 P(ω 2 )  = 1 ω2 (ω 2 )2 ··· 2 n−1  
(ω )   a2 
   
.. .. ..   .. 
    
. . .  . 
  
   ··· ···
n−1
P(ω ) 1 ω n−1 n−1 2
(ω ) ··· (ω n−1 )n−1 an−1
29
n−1
aj ωij
X
P(ω) =
j=0
P(1) 1 1 1 1 a0
    
···
 P(ω)  1
   ω ω2 ··· ω n−1    a1 
 
 P(ω 2 )  = 1 ω2 (ω 2 )2 ··· 2 n−1  
(ω )   a2 
   
.. .. ..   .. 
    
. . .  . 
  
   ··· ···
n−1
P(ω ) 1 ω n−1 n−1 2
(ω ) ··· (ω n−1 )n−1 an−1
29
n−1
aj ωij
X
P(ω) =
j=0
P(1) 1 1 1 1 a0
    
···
 P(ω)  1
   ω ω2 ··· ω n−1    a1 
 
 P(ω 2 )  = 1 ω2 (ω 2 )2 ··· 2 n−1  
(ω )   a2 
   
.. .. ..   .. 
    
. . .  . 
  
   ··· ···
n−1
P(ω ) 1 ω n−1 n−1 2
(ω ) ··· (ω n−1 )n−1 an−1
29
n−1
aj ωij
X
P(ω) =
j=0
P(1) 1 1 1 1 a0
    
···
 P(ω)  1
   ω ω2 ··· ω n−1    a1 
 
 P(ω 2 )  = 1 ω2 (ω 2 )2 ··· 2 n−1  
(ω )   a2 
   
.. .. ..   .. 
    
. . .  . 
  
   ··· ···
n−1
P(ω ) 1 ω n−1 n−1 2
(ω ) ··· (ω n−1 )n−1 an−1
29
n−1
aj ωij
X
P(ω) =
j=0
P(1) 1 1 1 1 a0
    
···
 P(ω)  1
   ω ω2 ··· ω n−1    a1 
 
 P(ω 2 )  = 1 ω2 (ω 2 )2 ··· 2 n−1  
(ω )   a2 
   
.. .. ..   .. 
    
. . .  . 
  
   ··· ···
n−1
P(ω ) 1 ω n−1 n−1 2
(ω ) ··· (ω n−1 )n−1 an−1
Vandermonde matrix V(ω)
29
Polynomial evaluation - a geometric view
Question: What linear transformation does the Vandermonde

matrix correspond to?
30

1 1 1 1 0
  
···
1
 ω ω2 ··· ω n−1   0
 
1 ω 2 (ω 2 )2 · · · (ω 2 )n−1   1 
  
.. ..   .. 
  
. . .

 ··· ···
n−1 n−1 2 n−1 n−1
1 ω (ω ) · · · (ω ) 0
30

1 1 1 1 0
  
···
1
 ω ω2 ··· ω n−1   0
 
1 ω 2 (ω 2 )2 · · · (ω 2 )n−1   1 
  
.. ..   .. 
  
. . .

 ··· ···
n−1 n−1 2 n−1 n−1
1 ω (ω ) · · · (ω ) 0
30

1 1 1 1 0
  
···
1
 ω ω2 ··· ω n−1   0
 
1 ω 2 (ω 2 )2 · · · (ω 2 )n−1   1 
  
.. ..   .. 
  
. . .

 ··· ···
n−1 n−1 2 n−1 n−1
1 ω (ω ) · · · (ω ) 0
Multiplying the Vandermonde matrix with the ith vector of the

standard basis gives the ith column of the Vandermonde matrix
30
Lemma: The columns of the Vandermonde matrix V(ω) are

orthogonal vectors
31

orthogonal vectors
complex conjugate
 
1
 ω −k 
 
 n−1
 (ω −k )2  X


1 ωj (ω j )2 (ω j )3 ··· (ω j )n−1  (ω −k )3  = ω i(j−k)
 
 i=0
 
 ..
.
 
 
(ω −k )n−1
31

orthogonal vectors
complex conjugate
 
1
 ω −k 
 
 n−1
 (ω −k )2  X


 
 i=0
 
 ..
.
 
 
(ω −k )n−1
Pn−1
• If j = k, then i=0 ω i(j−k) = n
• If j 6= k,
n−1
1 − ω n(j−k)
ω i(j−k) =
X
1 − ω j−k
i=0 31

orthogonal vectors
complex conjugate
 
1
 ω −k 
 
 n−1
 (ω −k )2  X


 
 i=0
 
 ..
.
 
 
(ω −k )n−1
Pn−1
• If j = k, then i=0 ω i(j−k) = n
ω n = 1 - nth root of unity
• If j 6= k,
n−1
1 − ω n(j−k)
ω i(j−k) =
X
=0
1 − ω j−k
i=0 31

orthogonal vectors
• The Vandermonde matrix V(ω) rotates the standard basis to the

Fourier basis
• The Fourier basis consists of the columns of the Vandermonde
matrix V(ω)
31

orthogonal vectors

Fourier basis
matrix V(ω)
• Interpolation amounts to performing the reverse
transformation
31

orthogonal vectors

Fourier basis
matrix V(ω)
• Interpolation amounts to performing the reverse
transformation
V(ω)V(ω)∗ = nI
complex conjugate
31
Interpolation - Inverting the Vandermonde matrix
32
Lemma: V(ω)−1 = n1 V(ω)∗
1 1 1 1
 
···
1
 ω −1 (ω −1 )2 ··· (ω −1 )n−1 

V(ω)∗ = 1 ω −2 (ω −2 )2 ··· (ω −2 )n−1
 

.. ..
 
. .
 
 ··· ··· 
−(n−1) 2 −(n−1) n−1
1 ω −(n−1)
(ω ) ··· (ω )
32
1 1 1 1
 
···
1
 ω −1 (ω −1 )2 ··· (ω −1 )n−1 

V(ω)∗ = 1 ω −2 (ω −2 )2 ··· (ω −2 )n−1
 

.. ..
 
. .
 
 ··· ··· 
−(n−1) 2 −(n−1) n−1
1 ω −(n−1)
(ω ) ··· (ω )
• Conjugates of the roots of unity are themselves roots of unity!
32
1 1 1 1
 
···
1
 ω −1 (ω −1 )2 ··· (ω −1 )n−1 

V(ω)∗ = 1 ω −2 (ω −2 )2 ··· (ω −2 )n−1
 

.. ..
 
. .
 
 ··· ··· 
−(n−1) 2 −(n−1) n−1
1 ω −(n−1)
(ω ) ··· (ω )
ω −j = ω n−j
32
1 1 1 1
 
···
1 ω n−1 (ω n−1 )2 ··· (ω n−1 )n−1 
 
V(ω)∗ =  1 ω n−2 (ω n−2 )2 ··· (ω n−2 )n−1 
 
.. ..
 
. .
 
 ··· ··· 
1 ω (ω)2 ··· (ω) n−1

• Compute the FFT, and reverse the output!
ω −j = ω n−j
32
Convolution - the final algorithm
n−1 n−1
i
bi xi
X X
P(x) = ai x Q(x) =
i=0 i=0
(a0 , a1 , . . . , an−1 ) O(n log n) P(α0 ), P(α1 ), . . . , P(α2n−2 )

(b0 , b1 , . . . , bn−1 ) evaluation Q(α0 ), Q(α1 ), . . . , Q(α2n−2 )
pointwise product O(n)

FFT
P(α0 )Q(α0 ),
P(x)Q(x)
O(n log n) ...
P(α2n−2 )Q(α2n−2 )
33
Closest pair of points
Pairwise distances of points
Given n points {pi = (xi , yi )}1≤i≤n on the plane, find a pair of points
that are closest to each other
34
(1, 1)
(−1, 0.5) (1.5, 0.5)
(−0.5, −0.5)
34
(1, 1) √
0.5
(−1, 0.5) (1.5, 0.5)
(−0.5, −0.5)
34
(1, 1) √
0.5
(−1, 0.5) (1.5, 0.5)
(−0.5, −0.5)
• (Most?) Important primitive for geometric algorithms
34
(1, 1) √
0.5
(−1, 0.5) (1.5, 0.5)
(−0.5, −0.5)

• Question: What is the simplest algorithm?
34
(1, 1) √
0.5
(−1, 0.5) (1.5, 0.5)
(−0.5, −0.5)

For each pair of points, find the distance; then find the pair that is
closest
34
(1, 1) √
0.5
(−1, 0.5) (1.5, 0.5)
(−0.5, −0.5)

For each pair of points, find the distance; then find the pair that is
closest O(n2 )
34
Pairwise distances - the 1D case
Given n points on the real line - find the pair of points that are
closest to each other
35
• Sort the points

• Find the i such that pi+1 − pi is the smallest in the sorted order
35
• Sort the points O(n log n)

O(n)
35
• Sort the points O(n log n)

O(n)
Question: Does a similar idea work in the 2D case - sorting based on

the x and y coordinates
35
36
Sorting according to x-coordinate and y-coordinate and checking in

order
36

order
(1, 3)
(3, 1)
(−3, −1)
(−1, −3)
36

order
(1, 3)
(3, 1)
√
2 10
(−3, −1)
√
2 2
(−1, −3)
36
Pairwise distances - a divide and conquer approach
Assumption: No two points have the same x-coordinate
37
37
• Divide the set of points into two (almost)equal halves
37
δ1
δ2

• Find the closest pair of points in each part recursively
37
δ1
δ3
δ2

• Find the closest pair of points, one on each side
37
δ1
δ3
δ2

• Return min(δ1 , δ2 , δ3 )
37
δ1
δ3
δ2

• Find the closest pair of points in each part recursively 2T(n/2)
37
δ1
δ3
δ2

• Find the closest pair of points in each part recursively 2T(n/2)
• Find the closest pair of points, one on each side O(n2 )?
37
δ1
δ2
38
δ1
δ = min(δ1 , δ2 )
δ2
38
2δ
δ1
δ = min(δ1 , δ2 )
δ2
• Only points within δ of the midpoint line need to be considered
38
2δ
δ1
δ = min(δ1 , δ2 )
δ2

might contain all the points!
38
2δ
δ1
δ = min(δ1 , δ2 )
δ2

might contain all the points!
Question: For a fixed point p on one side, how many candidate
points are there on the other side?
38
Pairwise distances - bounding the number of points
δ δ
39
How many points can lie inside the

δ δ
δ × δ square?
39

δ δ
δ × δ square?
• Every two points inside a δ × δ square must be at least δ apart!
39

δ δ
δ × δ square?

there can be no more than 4 points inside a δ × δ square
39

δ δ
δ × δ square?

there can be no more than 4 points inside a δ × δ square
• For any point p in the 2δ strip, need to consider at most 15
points next to it, when the points are sorted according to the y
coordinate
39
Closest pair of points - a divide and conquer algorithm
Input: Set of points pi = (xi , yi )

Output: Pair of points closest to each other
Find the line x = xm dividing into two equal halves P1 and P2
δ1 ← distance between closest pair of points in P1
δ2 ← distance between closes pair of points in P2
δ ← min(δ1 , δ2 )
Sort the points Pδ between x = xm − δ and xm + δ according to
the y coordinates
foreach point p in the sorted list of Pδ do
foreach point p0 that is at most 15 positions away from p do
δ 0 ← min(δ 0 , d(p, p0 ))
return min(δ 0 , δ)
40
Closest pair of points - a divide and conquer algorithm
Input: Set of points pi = (xi , yi )

Output: Pair of points closest to each other
Find the line x = xm dividing into two equal halves P1 and P2 O(n)
δ1 ← distance between closest pair of points in P1
δ2 ← distance between closes pair of points in P2 2T( n2 )
δ ← min(δ1 , δ2 )
Sort the points Pδ between x = xm − δ and xm + δ according to
the y coordinates O(n log n)
foreach point p in the sorted list of Pδ do O(n)
foreach point p that is at most 15 positions away from p do
0
δ 0 ← min(δ 0 , d(p, p0 ))
return min(δ 0 , δ)
n
T(n) = 2T + O(n log n)
2
40
Closest pair of points - solving the recurrence
n

2
41
n

2
a b f(n)
Recall the formula from the recursion tree:

Plog n−1
T(n) = nlogb a + i=0 ai f(n/bi )
41
n

2
a b f(n)

Plog n−1
log n−1
n n
2i
X
T(n) = n + log
2i 2i
i=0
log n−1
! log n
Y n n
= n + n log i
= n + n log Plog n−1
2 2 i=0 i
i=0
nlog n

= n + n log = O(n log2 n)
2log n(log n−1)/2
41
Can we avoid sorting
n

2 at each step?
a b f(n)

Plog n−1
log n−1
n n
2i
X
T(n) = n + log
2i 2i
i=0
log n−1
! log n
Y n n
= n + n log i
= n + n log Plog n−1
2 2 i=0 i
i=0
nlog n

= n + n log = O(n log2 n)
2log n(log n−1)/2
41
Closest pair of points - preprocessing the input
n

2 at each step?
42
n

2 at each step?
• Sort the list of points P according to both the x and y

coordinates and maintain two lists Px and Py
42
n

2 at each step?

• At each recursive call, divide the the list into two lists
maintaining the sorted order
42
n

2 at each step?

maintaining the sorted order
• Scan the sorted list to obtain the list of points in the 2δ-strip
sorted according to y coordinates
42
n

2 at each step?

maintaining the sorted order O(n)
sorted according to y coordinates O(n)
42
n

2 at each step?

maintaining the sorted order O(n)
sorted according to y coordinates O(n)
n
T(n) = 2T + O(n) + one-time cost of sorting

2
42

Recursion and Divide-and-Conquer: CS2800: Design and Analysis of Algorithms

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Recursion and Divide-and-Conquer: CS2800: Design and Analysis of Algorithms

Uploaded by

Copyright:

Available Formats

Recursion and Divide-and-Conquer

CS2800: Design and Analysis of Algorithms

2. Integer Multiplication Redux

4. Fast Fourier Transform

5. Closest pair of points

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• Often, we are only interested in the the rate of growth of a

• O is like ≤, o is like <, Θ is like =, but...

• O is like ≤, o is like <, Θ is like =, but...

Consider a hypothetical algorithm for checking primality of a

• If n is even (check the LSB), answer ”no”

• O is like ≤, o is like <, Θ is like =, but...

Consider a hypothetical algorithm for checking primality of a

• If n is even (check the LSB), answer ”no”

• O is like ≤, o is like <, Θ is like =, but...

Consider a hypothetical algorithm for checking primality of a

• If n is even (check the LSB), answer ”no”

• f(n) = O(log n), but not Θ(log n)

Input: Integers a and b

Input: Integers a and b

• The main while loop runs for log a many iterations

Input: Integers a and b

• The main while loop runs for log a many iterations

Input: Integers a and b

• The main while loop runs for log a many iterations

(10n/2 a + b )(10n/2 c + d ) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )

(10n/2 a + b )(10n/2 c + d ) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )

Numbers with n/2 digits

(10n/2 a + b )(10n/2 c + d ) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )

Numbers with n/2 digits Recursive calls

(10n/2 a + b )(10n/2 c + d ) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )

Numbers with n/2 digits Recursive calls

Correctness: From the definition of multiplication

(10n/2 a + b )(10n/2 c + d ) = 10n (ac ) + 10n/2 ( ad + bc ) + ( bd )

Numbers with n/2 digits Recursive calls

Correctness: From the definition of multiplication

(10n/2 a + b)(10n/2 c + d) = 10n (ac) + 10n/2 (ad + bc) + (bd)

(10n/2 a + b)(10n/2 c + d) = 10n (ac) + 10n/2 (ad + bc) + (bd)

(10n/2 a + b)(10n/2 c + d) = 10n (ac) + 10n/2 (ad + bc) + (bd)

# recursive calls adding the sub-products

running time on inputs of size n

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

logb n−1 n

Solving recurrence relations of the form T(n) = aT( bn ) + f(n)?

logb n−1 n

• If f(n) = O(nlogb a− ), then T(n) = Θ(nlogb a )

• If f(n) = O(nlogb a− ), then T(n) = Θ(nlogb a )

Karatsuba’s observation: bc + ad = ac + bd − (a − b)(c − d)

Karatsuba’s observation: bc + ad = ac + bd − (a − b)(c − d)

logb n−1 n

logb n−1 n

• If f(n) = O(nlogb a− ), then T(n) = Θ(nlogb a )

• If f(n) = O(nlogb a− ), then T(n) = Θ(nlogb a )