Algorithms Analysis & Design

Divide and Conquer II

(More Examples)
 Announcements
• QUIZ#1: Analysis
– Assess: analysis part (normal & recursive codes)
– Location: @LMS under “Tests/Quizzes”
– Date: SUN 12 MAR 22:00 (10:00 PM) isA
– Duration: 15 min.
– Evaluation: 3 Marks
 Announcements
• Midterm
– till Lec05 (inclusive)
– Midterm covers 2 parts, as follows:

1. Analysis 50%
2. D&C 50%
– Contains new design problem
 Announcements
Competition TOP5
SC & CSYS Compensation Quiz
1) You are running a library catalog. You
know that the books in your collection are
almost in sorted ascending order by title,
with the exception of one book which is in
the wrong place. You want the catalog to
be completely sorted in ascending order?
What’s the most suitable sort?
2) In binary insertion sorting
(insertion sort that uses
binary search to find each
insertion point)... How many
total operations are required
in its expected case?
3) Given N points in 2-dimensional space, it's required
to find the two points that are closest to each
other? What's the complexity of BRUTE-FORCE Sol?
 GENERAL NOTE
Hidden Slides are Extra Knowledge
Not Included in Exams
 Roadmap

Algorithm Analysis & Design

Think…Design…Analyze…

Analysis Design

Statements, Recursion Brute- Divide & Dynamic Greedy Increm.

Conditions force Conquer Program. Method Improv.
& Loops (3 methods)
 From Last Time…
• D&C Paradigm Why?
1. Divide the problem into small distinct sub-problem(s) efficient solutions.
2. Conquer the sub-problems by solving them recursively in
same manner RECURSIVE by nature
3. Combine the solution of sub-problems to get the final solution
Fun(…)
1) Divide
….
2) Conquer . . . …. 1) Divide
. . . . . . Fun(…) … 2) Conquer
3) Combine
. . . …. 3) Combine
. . . ….
. . .
. . . . . . . . .
 PRE-TEST

• How can we find Kth smallest element in array?

• How can we find the sub-array with max sum?
• Can we represent/operate on values greater than long?
• What’s the complexity of multiplying TWO square matrices?

Can we do better?
 Agenda
1. Select Kth Order Element

ALG1: Expected Linear Time PART I

2. Maximal Subarray
Description
3. Integer Multiplication (Karatsuba)

4. Matrix Multiplication PART II Naïve Solution

[Optional] D&C Solution

5. Van Emde Boas Tree Analysis

6. Closest Pair

7. Select Kth Order Element: ALG2: Worst Case Linear Time

•
 Learning Outcomes

1. Use a divide-and-conquer algorithm to solve an appropriate problem.

2. Demonstrate the ability to evaluate algorithms, to select from a range of
possible options, to provide justification for that selection.
 References
• Book Chapters
– Section 4.1 (Max Subarray)
– Section 4.2 (Matrix Mul)
– Section 9.2 (Selection ALG1)
– Section 9.3 (Selection ALG2)
– Chapter 20 (van Emde Boas Tree) MIT Video Lecture
• Stanford D&C Lecture
– Integer Multiplication (Karatsuba) [Video1, Video2]
– Closest Pair
 Select K Order Element
th

• Given: a set of "n" unordered numbers

• Assumption: numbers are distinct
• Required: find "kth" smallest num (k is an integer between 1 & n).

K-1 elements Kth N - K elements

< ? >
 Select K Order Element
th

• Naïve Solution
1. Sort the elements O(Nlog(N))
2. Select Kth element O(1)

Total O(Nlog(N))
Desired O(N)
 Select K Order Element
th

• ALG1: D&C with EXPECTED Linear Time

1. Divide:
1. select a pivot
2. place it in its correct position [linear time]
2. Conquer: Recursively select desired element in ONE subarray
• Select Kth in left subarray, OR
• Select (K-J)th in right subarray
3. Combine: Trivial K < J
• Analysis:
Select Kth P ? < > N
• T(N) = T(L)J + Θ(N) < K
Select Kth• L: length
<P of one
> of the
? two subarrays
• Depend on the inputK - Jarray and the pivot selection
Select (K-J)th ?
 Select Kth Order Element – ALG1
• What would the best split be?
o Each call to Partition splits the array into one half

1. T(N) = T(N/2) + Θ(N)

2. Master Case#3: Θ(N)
Analysis and Design of Algorithms 19
 Select Kth Order Element – ALG1
• What would the worst split be?
o Each call to Partition splits the array into ONE part with N-1

Partition ?

N–1 ?

N–2 ?

N–3 ?

1. T(N) = T(N – 1) + Θ(N)

2. Iteration method  Θ(N2)
Analysis and Design of Algorithms 20
 Select Kth Order Element – ALG1
• What about splits between best and worst?
o Ex.: Each call to Partition splits array into factor from array size

1. T(N) = T(N/P) + Θ(N) (Regardless the value of P)

2. Master Case#3: Θ(N)

Analysis and Design of Algorithms 21

 Select Kth Order Element – ALG1
• How to Avoid N2 Worst Case?
– Randomly chose the pivot
Avoids the worst case on a completely sorted list.
Expected time is Θ(N). Assuming elements are distinct.
See Textbook P.217-219 for detailed analysis.
 Select Kth Order Element – ALG1
J = 7-3+1=5
• Pseudo-code L Q R
1 2 3 4 5 6 7 8

RANDOMIZED-SELECT (A, L, R, K) 2 5 6 1 4 3 7 8
1 if L == R
Where’s each
2 return A[L]
step of D&C
3 Q = RANDOMIZED-PARTITION (A, L, R)
???
4J=Q–L+1
5 if K == J //the pivot value is the answer
6 return A[Q]
7 elseif K < J
8 return RANDOMIZED-SELECT (A, L, Q – 1, K)
9 else return RANDOMIZED-SELECT (A, Q + 1, R, K – J)
 Select Kth Order Element – ALG1
• Pseudo-code
RANDOMIZED-PARTITION (A, L, R)
1 i = RANDOM (L, R)
2 exchange A[L] with A[i]
3 return PARTITION(A, L, R) From QuickSort
 Agenda
1. Select Kth Order Element

ALG1: Expected Linear Time PART I

2. Maximal Subarray
Description
3. Integer Multiplication (Karatsuba)

4. Matrix Multiplication PART II Naïve Solution

[Optional] D&C Solution

5. Van Emde Boas Tree Analysis

6. Closest Pair

7. Select Kth Order Element: ALG2: Worst Case Linear Time

•
 Maximum Sequence Sum
• Given: array of N numbers
• Required: find a sequence of contiguous numbers with max sum

31 -41 59 26 -53 58 97 -93 -23 84

• How?
1. Find all possible sequences
2. Chose one with max sum
• Complexity?
Θ(N3) or Θ(N2)
 Maximum Sequence Sum
• Given: array of N numbers
• Required: find a sequence of contiguous numbers with max sum

31 -41 59 26 -53 58 97 -93 -23 84

• Applications?

1. Stock prices over n days (in the past)

2. Maximize profits
 Maximum Sequence Sum (D&C)
• D&C Solution
1. Divide: Trivial (divide the given array in two halves)
2. Conquer: Recursively calculate max. sequence sum in each half
3. Combine: Find maximum subarray sum from left (ML), right (MR) &
subarray crosses the midpoint (MC)

Find MaxSeqSum found N

Find MaxSeqSum found N/2 N/2 Find MaxSeqSum found

MAX: ML MC MR
 Maximum Sequence Sum (D&C)
• D&C Solution
1. Divide: Trivial (divide the given array in two halves)
2. Conquer: Recursively calculate max. sequence sum in each half
3. Combine: Find maximum subarray sum from left (ML), right (MR) &
subarray crosses the midpoint (MC) HOW?
a) Find max sum starting from mid point and ending at some point on left of mid
b) Find max sum starting from mid+1 & ending with some point on right of mid+1
c) Add the two sums together to form Mc

Max Seq Sum + Max Seq Sum

N/2 M N/2
 Maximum Sequence Sum (D&C)
• D&C Solution
– Analysis:
• Divide (Split the array into 2 Halves) Θ(1)
• Conquer (Recurse on 2 Subproblems) 2 T(N/2)
• Combine (Calculate the maximum subarray crosses the midpoint)
Θ(N)

T(N) = 2 T(N/2) + Θ(N)

Master Case#2  Θ(N log N)
BETTER Than NAÏVE !!!
Can we do BETTER? (later in the course )
 Maximum Sequence Sum (D&C)
2 -1 4 -6 7
 Maximum Sequence Sum (D&C)
• Pseudo-code
maxsub(int[] S; low, high: int) return (lowIndex, highIndex, sum)
if low = high then
return (low, high, S(low))
else
mid = (low + high) / 2
(llow, lhigh, lsum) = maxsub(S, low, mid)
(rlow, rhigh, rsum) = maxsub(S, mid+1, high)
(mlow, mhigh, msum) = middlemaxsub(S, low, mid, high)
end if;
return triple with highest sum
end maxsub

middlemaxsub(int[] S; low, high, int) return (lowIndex, highIndex,

sum)
start at mid and find bestleft and leftsum
start at mid and find bestright and rightsum

return (bestleft, bestright, rightsum+leftsum)

end middlemaxsub
Try Yourself…#1
For the Select Kth Element with expected linear time and first element as a pivot,
answer the following questions on the array {29, 6, -2, -3, -8, -11, 50, 0}:
1) How many pivots are selected until finding the 2nd largest element?
2) How many times you conquer on the left sub-array?
3) How many times you conquer on the right sub-array?
Try Yourself…#2
For the Select Kth Element with expected linear time and first element as a pivot,
answer the following questions on the array { 0, 6, -2, -3, -8, -11, 50, 29}:
1) How many pivots are selected until finding the smallest element?
2) How many times you conquer on the left sub-array?
3) How many times you conquer on the right sub-array?
Try Yourself…#3
For the Select Kth Element with expected linear time and first element as a pivot,
answer the following questions on the array {5, -6, 7, -8, 9, -10, 11, -12}:
1) How many pivots are selected until finding the 5th smallest element?
2) How many times you conquer on the left sub-array?
3) How many times you conquer on the right sub-array?
 Agenda
1. Select Kth Order Element

ALG1: Expected Linear Time PART I

2. Maximal Subarray
Description
3. Integer Multiplication (Karatsuba)

4. Matrix Multiplication PART II Naïve Solution

[Optional] D&C Solution

5. Van Emde Boas Tree Analysis

6. Closest Pair

7. Select Kth Order Element: ALG2: Worst Case Linear Time

•
 Integer Multiplication
• Given: two N-digit large integers, X & Y
• Required: Multiply X & Y

N N-1 N-2 3 2 1
X . . .
×
Y . . .

2N N N-1 N-2 3 2 1
. . . . . .
 Integer Multiplication (Naïve)
– Multiply each digit from Y by X Θ(N2)
– Add the results COMPLEXITY?
Θ(N2)
N N-1 N-2 3 2 1
X . . .
×
Y . . .

. . .
. . .
. . . +
. . .
. . .
2N N N-1 N-2 3 2 1
. . . . . .
 Integer Multiplication (D&C)
• D&C Solution – Trial#1
1. Divide: split X & Y into two halves: A & B, C & D.
2. Conquer: Recursively multiply 4 subproblems
3. Combine: X = 10N/2
×B+A X B A
Y = 10N/2 × D + C Y D C
×
a) Pad B×D with N zeros,
A
b) Pad B×C, A×D with N/2 zeros
C
×
c) Add all together
B
B×D 000000000 ………. 00000000000 ×
B×C 000 ………. 000
+ C

A×D 000 ………. 000

+ A
×
A×C
+ D

X × Y = 10 × B×D + 10
N N/2
× (B×C + A×D) + A×C
B
D
×
 Integer Multiplication (D&C)
• D&C Solution – Trial#1
– Analysis:
• Divide (Split X & Y into 2 Halves) Θ(N)
• Conquer (Recurs on 4 Subproblems) 4 T(N/2)
• Combine (Pad by 0’s & Add 4 Values) Θ(N)

T(N) = 4 T(N/2) + Θ(N)

Master Case#1  Θ(N2)

SAME AS NAÏVE !!!

Can we do BETTER?
 Integer Multiplication (D&C: Karatsuba)
• D&C Solution – Trial#2
– Gauss Trick:
Res = 10N × B×D + 10N/2 × (B×C + A×D) + A×C
• Can calculate B×C + A×D using single multiplication
1. Calculate Z = (A + B) × (C + D)
= A×C + A×D + B×C + B×D
2. Subtract B×D & A×C from Z:
= A×C + A×D + B×C + B×D - B×D - A×C = B×C + A×D
• Now, we can calculate Res using 3 MULTIPLICATION
M1 = A×C M2 = B×D Z = (A + B)×(C + D)

Res = 10N × M2 + 10N/2 × (Z – M1 – M2) + M1

 Integer Multiplication (D&C: Karatsuba)
• D&C Solution – Trial#2
1. Divide:
X B A
Θ(N)
1. split X & Y into two halves: A & B, C & D. D C
Y
2. Calculate A+B & C+D Θ(N)
2. Conquer: Only enhance by 0.42…
• Recursively multiply 3 subprob. (M1 (A×C),M2 (B×D), Z (A+B)(C+D)) 3 T(N/2)
3. Combine:
1. Is It WORTH??
Subtract M1 & M2 from Z = A×C + A×D + B×C + B×D - B×D - A×C = B×C + A×D Θ(N)
2. Pad M2 (B×D) with N zeros B×D 000000 ………. 0000000 Θ(N)
+
3. Pad result of (1) with N/2 zeros B×C + A×D 000 …. 000 Θ(N)
+
4. Add 2 & 3 to M1 (A×C) A×C Θ(N)
T(N) = 3 T(N/2) + Θ(N) Master Case#1  Θ(N1.58)
 Integer Multiplication (D&C: Karatsuba)
• D&C Solution – Trial#2

N Naïve Solution Θ(N2) D&C Solution Θ(N1.58) Speedup factor

100 10,000 1,446 7x
1000 1,000,000 54,954 18x
10,000 100,000,000 2,089,296 48x
 Integer Multiplication (Karatsuba)
• Application
– RSA Cryptosystem Algorithm

Enc(M) = ME mod N

M = Enc(M)D mod N
• Public Key: two BIG integers N, E
• Private Key: two BIG integers N, D
 Integer Multiplication (Karatsuba)
• Application
– RSA Cryptosystem Algorithm (Ex) DEMO
Enc(M) = ME mod N

M: Algorithms A’D

M = Enc(M)D mod N
N:475949804756254177244082678231127644638635769186852263630327872399101187400048606241668
59668486833021538759738968887527
E:2239763787088254951736859662264130092417109502055775358260344690284619737765819697471457
5577237681892436409853219169457
D:17
 Agenda
1. Select Kth Order Element

ALG1: Expected Linear Time PART I

2. Maximal Subarray
Description
3. Integer Multiplication (Karatsuba)

4. Matrix Multiplication PART II Naïve Solution

[Optional] D&C Solution

5. Van Emde Boas Tree Analysis

6. Closest Pair

7. Select Kth Order Element: ALG2: Worst Case Linear Time

•
 Matrix multiplication
Input: A = [aij], B = [bij].
i, j = 1, 2,… , n.
Output: C = [cij] = A B.

n
cij   a ik  bkj
k 1
September 14, 2005 Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.27
 Standard algorithm
for i  1 to n
do for j  1 to n
do cij  0
for k  1 to n
do cij  cij + aik bkj

Running time = (n3)

September 14, 2005 Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.28
 Matrix Mul: D&C ALG1
IDEA:
nn matrix = 22 matrix of (n/2)n/2) submatrices:

r = ae + recursive
bg s = af 8 mults of (n/2)n/2) submatrices
^
+ bh t = 4 adds of (n/2)n/2) submatrices
ce + dg u
= cf + dh Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.30
 Matrix Mul: D&C ALG1
1. Divide: each matrix into 4 quadrants
2. Conquer: Recursively multiply 8 problems.
3. Combine: add mul results & form final matrix.
T(N) = 8 T(N/2)+ (N2)

# subproblems work dividing

subproblem size and combining

 CASE 1  T(n) =
SAME AS NAÏVE !!!
nlogba = nlog28 = n3
September 14, 2005 (n
Copyright © 2001-5 ).Demaine and Charles E. Leiserson
Erik3D. L2.4 Can we do BETTER?
 Strassen’s idea [D&C ALG2]
• Multiply 22 matrices with only 7 recursive mults.
P1 = a  ( f – h) 7 mults, 18 adds/subs.
P2 = (a + b)  h Note: No reliance on
commutativity of mult!
P3 = (c + d)  e
P4 = d  (g – e)
P5 = (a + d)  (e + h) r = P5 + P4 – P2 + P6
P6 = (b – d)  (g + h) s = P1 + P2
P7 = (a – c)  (e + f ) t = P3 + P4
u = P5 + P1 – P3 – P7
September 14, 2005 Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.36
 Strassen’s idea [D&C ALG2]
• Multiply 22 matrices with only 7 recursive mults.
P1 = a  ( f – h) r = P5 + P4 – P2 + P6
P2 = (a + b)  = (a + d) (e + h)
h P3 = (c + d) + d (g – e) – (a
 e P4 = d  (g + b) h
– e) + (b – d) (g + h)
P5 = (a + d)  (e + h) = ae + ah + de +
P6 = (b – d)  (g + dh
h) + dg –de – ah –
P =
September
7 (a
14, –
2005 c)  (e
Copyright+© f )
2001-5 Erik D. Demaine
bh
and Charles E. Leiserson L2.39
 Strassen’s Alg. [D&C ALG2]
P1 = a  ( f – h)
1. Divide: P2 = (a + b)  h
1. Partition A and B into (n/2)(n/2) submatrices. P3 = (c + d)  e
P4 = d  (g – e)
2. Form 10 terms to be multiplied using + and – . P5 = (a + d)  (e + h)
P6 = (b – d)  (g + h)
2. Conquer: P7 = (a – c)  (e + f )
• Perform 7 multiplications of (n/2)(n/2) submatrices recursively.
3. Combine:
• Form C using + and – on (n/2)(n/2) submatrices.

T(n) = 7 T(n/2) + (n2)

September 14, 2005 Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.40
 Analysis of Strassen
T(n) = 7 T(n/2) +
(n2)
nlogba = nlog27  n2.81
 C ASE 1  T(n) = (n lg 7).

The number 2.81 may not seem much smaller than

3, but because the difference is in the exponent,
the impact on running time is significant. In
fact, Strassen’s algorithm beats the ordinary
algorithm on today’s machines for n  32 or so.
Best to date (of theoretical interest only): (n2.376L ).
September 14, 2005 Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.42
 Analysis of Strassen
T(n) = 7 T(n/2) +
(n2)
nlogba = nlog27  n2.81
 C ASE 1  T(n) = (n lg 7).

Speedup
Naïve Solution N3 D&C Solution N2.81 factor
100 1,000,000 416,869 2.4x
1000 1,000,000,000 269,153,480 3.7x
10,000 1,000,000,000,000 173,780,082,874 5.6x
 Agenda
1. Select Kth Order Element

ALG1: Expected Linear Time PART I

2. Maximal Subarray
Description
3. Integer Multiplication (Karatsuba)

4. Matrix Multiplication PART II Naïve Solution

[Optional] D&C Solution

5. Van Emde Boas Tree Analysis

6. Closest Pair

7. Select Kth Order Element: ALG2: Worst Case Linear Time

•
 Agenda
1. Select Kth Order Element

ALG1: Expected Linear Time PART I

2. Maximal Subarray
Description
3. Integer Multiplication (Karatsuba)

4. Matrix Multiplication PART II Naïve Solution

[Optional] D&C Solution

5. Van Emde Boas Tree Analysis

6. Closest Pair

7. Select Kth Order Element: ALG2: Worst Case Linear Time

•
 Summary
• D&C paradigm has 3 steps:
1. Divide
2. Conquer
3. Combine
• It significantly reduce complexity of some brute-force solutions
N = 106 Speedup
1. Search: N  Log(N) 106 vs. 20 x50k
2. Sort: N2  N Log(N) 1012 vs. 2x107 x50k
3. Select Kth order: N Log(N)  N 2x107 vs. 106 x20
4. Max Seq. Sum: N2  N Log(N) 1012 vs. 2x107 x50k
5. Big Integer Mul: N2  N1.58 1012 vs. 3x109 x330
3 2.8 18 15
 Questions
• FIRST: D&C SHEET with Solutions
1. General Questions (MCQ, T/F, Trace…)
2. 12 Design Problems [with solution]
3. 6 Design Problems [without solution]
 Questions
• SECOND: MIT Examples
1. Fibonacci Number
2. VLSI Tree Layout