Design and Analysis of Algorithm

Divide and Conquer

 Acknowledgement
 This lecture note has been summarized from lecture note
on Data Structure and Algorithm, Design and Analysis of
Computer Algorithm all over the world. I can’t remember
where those slide come from. However, I’d like to thank
all professors who create such a good work on those
lecture notes. Without those lectures, this slide can’t be
finished.

Divide and Conquer Technique 2

 Review: What is an algorithm?
 A step by step procedure for performing some
task in a finite amount of time

 A precise set of instructions for achieving a

particular goal

Divide and Conquer Technique 3

 Review: General Concepts
 Algorithm strategy
 Approach to solving a problem
 May combine several approaches

 Algorithm structure
 Iterative ⇒ execute action in loop
 Recursive ⇒ reapply action to subproblem(s)
 Problem type
 Satisfying ⇒ find any satisfactory solution
 Optimization ⇒ find best solutions (vs. cost
metric)

Divide and Conquer Technique 4

 Review: What you need to know about
algorithms?
 Running time
 Memory requirements
 Understand how the complexity of an algorithm
is measured
 Develop techniques to measure these
complexities experimentally
 In other words, measure the efficiency of an
algorithm!

Divide and Conquer Technique 5

 Review: Measures of efficiency
 Three main measures
 Best
 Worst
 Average

 E.g. consider a list

 Best: first item in the list
 Worst: last in the list, or not in the list
 Average: in the middle of the list

Divide and Conquer Technique 6

 Review: An example of an algorithm
(pseudocode)
BEGIN
Get WAGES
IF WAGES <=100
Tax = wages*0.25
ELSE
Tax = 100*0.25 + (wages-100)*0.4
END IF
Display Tax
END
It calculates tax at 25% on the first £100 of
earnings then at 40% on subsequent earnings.
Divide and Conquer Technique 7
 Some Algorithm Strategies
 Recursive algorithms
 Backtracking algorithms
 Divide and conquer algorithms
 Dynamic programming algorithms
 Greedy algorithms
 Brute force algorithms
 Branch and bound algorithms
 Heuristic algorithms

Divide and Conquer Technique 8

 Divide-and-Conquer
 A general methodology for using recursion to
design efficient algorithms
 It solves a problem by:
 Diving the data into parts
 Finding sub solutions for each of the parts
 Constructing the final answer from the sub
solutions

Divide and Conquer Technique 9

 Divide and Conquer
 Based on dividing problem into subproblems
 Approach
1. Divide problem into smaller subproblems
Subproblems must be of same type
Subproblems do not need to overlap
2. Solve each subproblem recursively
3. Combine solutions to solve original problem
 Usually contains two or more recursive calls

Divide and Conquer Technique 10

 Divide-and-conquer technique
a problem of size n

subproblem 1 subproblem 2
of size n/2 of size n/2

a solution to a solution to
subproblem 1 subproblem 2

a solution to
the original problem

Divide and Conquer Technique 11

 Divide and Conquer Algorithms
 Based on dividing problem into subproblems
 Divide problem into sub-problems
 Subproblems must be of same type

 Subproblems do not need to overlap

 Conquer by solving sub-problems recursively.

If the sub-problems are small enough, solve
them in brute force fashion
 Combine the solutions of sub-problems into a
solution of the original problem (tricky part)

Divide and Conquer Technique 12

 D-A-C
 For Divide-and-Conquer algorithms the running
time is mainly affected by 3 criteria:
 The number of sub-instances into which a
problem is split.
 The ratio of initial problem size to sub-
problem size.
 The number of steps required to divide the
initial instance and to combine sub-solutions.

Divide and Conquer Technique 13

 An example of D-A-C
 Given an array of n values find, in order, the lowest two
values efficiently.
 To make life easier we assume n is at least 2 and that it
is always divisible by 2
 Have you understood the problem??

2 10 9 6 4 5
 The solution should be 2 followed by 4

What if?? 2 8 2 7 9 3

Divide and Conquer Technique 14

 Possible Solutions
 Sort the elements and take the two first
 We get more information than we require (not
very efficient!!)
 Find the lowest, then find the second lowest
 Better than the first but still we have to go through
the array two times
 Be careful not to count the lowest value twice

 Search through the array keeping track of the

lowest two found so far
 Lets see how the divide and conquer technique
works!
Divide and Conquer Technique 15
 Divide and conquer approach
 There are many ways to divide the array
 It seems reasonable that the division should be
related to the problem
 Division based on 2!!
 Divide the array into two parts?
 Divide the array into pairs?

 Go ahead with the second option!

Divide and Conquer Technique 16

 The algorithm

 Lets assume the following array

2 6 7 3 5 6 9 2 4 1
 We divide the values into pairs

2 6 7 3 5 6 9 2 4 1
 We sort each pair

2 6 3 7 5 6 2 9 1 4

 Get the first pair (both lowest values!)

Divide and Conquer Technique 17

 The algorithm (2)
 We compare these values (2 and 6) with the values of
the next pair (3 and 7)

2 6 3 7 5 6 2 9 1 4
 Lowest 2,3
 The next one (5 and 6)
 Lowest 2,3
 The next one (2 and 9)
 Lowest 2,2
 The next one (1 and 4)
 Lowest 1,2

Divide and Conquer Technique 18

 Example: Divide and Conquer
 Binary Search
 Heap Construction
 Tower of Hanoi
 Exponentiation
Fibonnacci Sequence

 Quick Sort
 Merge Sort
 Multiplying large Integers
 Matrix Multiplications
 Closest Pairs

Divide and Conquer Technique 19

 Exponentiation
Power can be easily computed in O(logn) time as follows:

long Power (int x, int n) {

if (n==0) return 1;
else {
y = Power (x, n/2);
if (n % 2 == 0) return(y*y) // n even
else return (y*y*x);
}

Divide and Conquer Technique 20

 Time Complexity Example 1:
Fibonnacci Sequence
Fibonnacci sequence F(n) = F(n-1) + F(n-2)
Approach 1: Approach 2:

int Fib(n)
{ int Fib(n) {
if (n < 2) return (n); if (n < 2) return (n);
else { else {
return (Fib (n-1) + Fib (n-2)); int F1 = 1; F2 = 0;
} for (i = 2; i <= n; i++)
} { F = F1 + F2;
F2 = F1;
F1 = F;
}
return (F)
}

Divide and Conquer Technique 21

 Fibonnacci Sequence
Approach 3: the complexity is O(logn)
Note that we can rewrite
| F(n) | | 1 1 | | F(n-1) |
| | = | | * | |
| F(n-1)| | 1 0 | | F(n-2) |
Thus,
2
| F(n)
| | 1 1 | | F(n-1) | | 1 1 | | F(n-2) |
| | = | | * | | = | | * | |
| F(n-1)| | 1 0 | | F(n-2) | | 1 0 | | F(n-3) |

(n-1)
| 1 1 | | F(1) |
= | | * | |
| 1 0 | | F(0) |

Therefore, computing F(n) is reduced to computing

(n-1)
| 1 1 |
| |
| 1 0 |

Divide and Conquer Technique 22

 Heap Construction
Construct heap of T =>
Construct heap T1,
Construct heap T2
Adjust heap with root T
T
T1 T2

Divide and Conquer Technique 23

 Divide and Conquer – Examples
 Quicksort
 Partition array into two parts around pivot
 Recursively quicksort each part of array
 Concatenate solutions

 Mergesort
 Partition array into two parts
 Recursively mergesort each half
 Merge two sorted arrays into single sorted array

Divide and Conquer Technique 24

Quick Sort
 Sorting Problem Revisited
Given: an unsorted array
5 2 4 7 1 3 2 6

 Goal: sort it

1 2 2 3 4 5 6 7

Divide and Conquer Technique 26

 Quick Sort
 Approach
 Select pivot value (near median of list)
 Partition elements (into 2 lists) using pivot value
 Recursively sort both resulting lists
 Concatenate resulting lists
 For efficiency pivot needs to partition list evenly
 Performance
 O( n log(n) ) average case
 O( n2 ) worst case

Divide and Conquer Technique 27

 Quick Sort Algorithm
1. If list below size K
 Sort w/ other algorithm
x
 Else pick pivot x and
partition S into
 L elements < x
x
 E elements = x
 G elements > x L E G
3. Quicksort L & G
4. Concatenate L, E & G x
 If not sorting in place
Divide and Conquer Technique 28
 Quick Sort Code
void quickSort(int[] a, int x, int y) {
int pivotIndex;
if ((y – x) > 0) { Lower Upper
pivotIndex = partionList(a, x, y); end of end of
quickSort(a, x, pivotIndex – 1); array array
quickSort(a, pivotIndex+1, y); region region
to be to be
} sorted sorted
}

int partionList(int[] a, int x, int y) {

… // partitions list and returns index of pivot
}
Divide and Conquer Technique 29
 Quick Sort Example

7 2 8 5 4 2 4 5 7 8

2 5 4 7 8 2 4 5 7 8

2 5 4 2 4 5

4 5 4 5

Partition & Sort Result

Divide and Conquer Technique 30

 Quick Sort Code
int partitionList(int[] a, int x, int y) {
int pivot = a[x]; Use first
int left = x; element
int right = y; as pivot
while (left < right) {
while ((a[left] < pivot) && (left < right))
left++;
while (a[right] > pivot) Partition elements
in array relative to
right--; value of pivot
if (left < right)
swap(a, left, right);
} Place pivot in middle
swap(a, x, right); of partitioned array,
return right; return index of pivot
Divide and Conquer Technique 31
Merge Sort
 Merge Sort
 Approach
 Partition list of elements into 2 lists
 Recursively sort both lists
 Given 2 sorted lists, merge into 1 sorted list
a) Examine head of both lists
b) Move smaller to end of new list
 Performance
 O( n log(n) ) average / worst case

Divide and Conquer Technique 33

 Merge Example
2 4 5

2 7 4 5 8 7 8

2 2 4 5 7

7 4 5 8 8

2 4 2 4 5 7 8

7 5 8
Divide and Conquer Technique 34
 Merge Sort Example

7 2 8 5 4 2 4 5 7 8

7 2 8 5 4 2 7 4 5 8

7 2 8 5 4 7 2 8 4 5

5 4 5 4

Split Merge

Divide and Conquer Technique 35

 Merge Sort Code
void mergeSort(int[] a, int x, int y) {
int mid = (x + y) / 2; Upper
if (y == x) return; Lower end of
end of array
mergeSort(a, x, mid);
array region
mergeSort(a, mid+1, y); region
merge(a, x, y, mid); to be
to be sorted
} sorted
void merge(int[] a, int x, int y, int mid) {
… // merges 2 adjacent sorted lists in array
}

Divide and Conquer Technique 36

 Upper
Merge Sort Code end of
1st array
void merge (int[] a, int x, int y, int mid) { region
int size = y – x; Upper
int left = x; Lower end of
int right = mid+1; end of 2nd array
int[] tmp; int j; 1st
array region
region
for (j = 0; j < size; j++) {
if (left > mid) tmp[j] = a[right++];
else if (right > y) || (a[left] < a[right])
tmp[j] = a[left++];
Copy smaller of two
else tmp[j] = a[right++]; elements at head of 2
} array regions to tmp
for (j = 0; j < size; j++) buffer, then move on
a[x+j] = tmp[j];
} Copy merged
array back
Divide and Conquer Technique 37
 Mergesort: Divide Step

Step 1 – Divide
5 2 4 7 1 3 2 6

5 2 4 7 1 3 2 6

5 2 4 7 1 3 2 6

5 2 4 7 1 3 2 6

log(n) divisions to split an array of size n into single elements

Divide and Conquer Technique 38

 Mergesort: Conquer Step

Step 2 – Conquer
5 2 4 7 1 3 2 6 O(n)

2 5 4 7 1 3 2 6 O(n)

2 4 5 7 1 2 3 6 O(n)

1 2 2 3 4 5 6 7 O(n)

logn iterations, each iteration takes O(n) time. Total Time: O(n logn)

Divide and Conquer Technique 39

 Mergesort: Combine Step

Step 3 – Combine
5 2 2 5

• 2 arrays of size 1 can be easily merged to

form a sorted array of size 2
• 2 sorted arrays of size n and m can be
merged in O(n+m) time to form a sorted
array of size n+m

Divide and Conquer Technique 40

 Mergesort: Combine Step

Combining 2 arrays of size 4

2 4 5 7 2 4 5 7
1 1 2
1 2 3 6 2 3 6

4 5 7 4 5 7
1 2 2 1 2 2 3
2 3 6 3 6

4 5 7 Etcetera…
1 2 2 3 4
6 1 2 2 3 4 5 6 7

Divide and Conquer Technique 41

 Mergesort: Example

20 4 7 6 1 3 9 5

Divide 20 4 7 6 1 3 9 5

20 4 7 6 1 3 9 5

20 4 7 6 1 3 9 5

4 20 6 7 1 3 5 9

Conquer
4 6 7 20 1 3 5 9

1 3 4 5 6 7 9 20

Divide and Conquer Technique 42

 MergeSort: Running Time
 The problem is simplified to baby steps
 forthe i’th merging iteration, the
complexity of the problem is O(n)
 number of iterations is O(log n)
 running time: O(n logn)

Divide and Conquer Technique 43

Integer Multiplication
 Integer Arithmetic
 Add. Given two n-digit integers a and b, compute a + b.
O(n) bit operations.

 Multiply. Given two n-digit integers a and b, compute a × b.
 Brute force solution: Θ(n2) bit operations.
11010101
* 01111101
1 1 1 1 1 1 0 1
110101010
1 1 0 1 0 1 0 1
+ 0 1 1 1 1 1 0 1 Multiply 0 0 0 0 0 0 0 0 0
110101010
1 0 1 0 1 0 0 1 0
110101010
Add 110101010
110101010
110101010
000000000
01101000000000010
Divide and Conquer Technique 45
 Divide-and-Conquer Multiplication
 To multiply two n-digit integers:
 Multiply four ½n-digit integers.
 Add two ½n-digit integers, and shift to obtain
result.

assumes n is a power of 2
Divide and Conquer Technique 46
 Karatsuba Multiplication
 To multiply two n-digit integers:
 Add two ½n digit integers.
 Multiply three ½n-digit integers.
 Add, subtract, and shift ½n-digit integers to
obtain result.

A B A C C

Divide and Conquer Technique 47

 Karatsuba Multiplication (cont’)
 Theorem. [Karatsuba-Ofman, 1962] Can
multiply two n-digit integers in O(n1.585) bit
operations.

Divide and Conquer Technique 48

 Karatsuba: Recursion Tree

T(n) n

T(n/2) T(n/2) T(n/2) 3(n/2)

T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) T(n/4) 9(n/4)

... ...

T(n / 2k) 3k (n / 2k)

... ...

T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2) 3 lg n (2)

Divide and Conquer Technique 49

 Multiplying Large Integers

High school multiplications eg)

110011
×101001
A = an-1 an-2 … a1 a0
110011
× B = bn-1 bn-2 … b1 b0 000000
000000
110011
Time O(N2) 000000
110011

Divide and Conquer Technique 50

 Multiplying Large Integers (cont’)

A1 = an-1 an-2 …an/2 A0 = an/2-1 … a1 …a0

B1 = bn-1 bn-2 …bn/2 B0 = bn/2-1 …b1 …b0
Define X = A1 + A0
Y = B1 + B0

AB = A1B12n + (XY – A1B1 – A0B0) 2n/2 + A0B0

Complexity? An integer multiplication of two n bit integer is
reduced to
=> three integer multiplications of n/2 bits, and 6 additions of n bits

T(n) = 3T(n/2) + O(n) = O(nlog23) = O(n1.59)

Divide and Conquer Technique 51

 Multiplying integers
(Decimal numbers)
A = 2 3 6 7 8 1 8 3 A1 = 2 3 6 7 A0 = 8 1 8 3
B = 1 3 0 4 6 4 9 1 B1 = 1 3 0 4 B0 = 6 4 9 1

AB = ( A1104 + A0 ) ( B1104 + B0 )
= A1B1108 + (A1B0+A0B1)104 + A0B0

4 multiplications T(n) = 4T(n/2) + O(n)

3 additions T(1) = O(1)
2 shiftings
T(n) = O(n2)

Divide and Conquer Technique 52

 Multiplying integers(Decimal numbers)
D-A-C Approach
Let X = ( A1 + A0 ) , Y = ( B1 + B0 )
XY =A1B1 + A1B0 + A0B1 + A0B0

XY - A1B1 - A0B0
= A1B0+A0B1

AB = ( A1104 + A0 ) ( B1104 + B0 )
= A1B1108 + (A1B0+A0B1)104 +A0B0
= A1B1108 + (XY - A1B1 - A0B0)104 + A0B0
T(n) = 3T(n/2) + O(n)
T(1) = O(1)
Recursively compute XY, A1B1 , A0B0,
T(n) = nlog23 = O(n1.59)
3 multiplications
6 additions
53
2 Conquer
Divide and shiftings
Technique
Matrix Multiplication
 Matrix Multiplication
 Matrix multiplication. Given two n-by-n matrices A and B,
compute C = AB.

 Brute force. Θ(n3) arithmetic operations.

 Fundamental question. Can we improve upon brute
force?
Divide and Conquer Technique 55
 Matrix Multiplication: Warmup
 Divide-and-conquer.
 Divide: partition A and B into ½n-by-½n blocks.
 Conquer: multiply 8 ½n-by-½n recursively.
 Combine: add appropriate products using 4
matrix additions.

Divide and Conquer Technique 56

 Matrix Multiplication: Key Idea
 Key idea. multiply 2-by-2 block matrices with
only 7 multiplications.

 7 multiplications.
 18 = 10 + 8 additions (or subtractions).

Divide and Conquer Technique 57

 Fast Matrix Multiplication
 Fast matrix multiplication. (Strassen, 1969)
 Divide: partition A and B into ½n-by-½n blocks.
 Compute: 14 ½n-by-½n matrices via 10 matrix
additions.
 Conquer: multiply 7 ½n-by-½n matrices recursively.
 Combine: 7 products into 4 terms using 8 matrix
additions.

 Analysis.
 Assume n is a power of 2.
 T(n) = # arithmetic operations.
Divide and Conquer Technique 58
 Fast Matrix Multiplication in Practice
 Implementation issues.
 Sparsity.
 Caching effects.
 Numerical stability.
 Odd matrix dimensions.
 Crossover to classical algorithm around n = 128.

 Common misperception: "Strassen is only a theoretical

curiosity."
 Advanced Computation Group at Apple Computer reports
8x speedup on G4 Velocity Engine when n ~ 2,500.
 Range of instances where it's useful is a subject of
controversy.
Divide and Conquer Technique 59
 Fast Matrix Multiplication in Theory
 Q. Multiply two 2-by-2 matrices with only 7 scalar multiplications?
 A. Yes! [Strassen, 1969] Θ (n log 2 7 ) =O(n 2.81 )

 Q. Multiply two 2-by-2 matrices with only 6 scalar multiplications?

 A. Impossible. [Hopcroft and Kerr, 1971] Θ (n log2 6 ) =O(n 2.59 )

 Q. Two 3-by-3 matrices with only 21 scalar multiplications?

 A. Also impossible. Θ(n log3 21 ) =O(n 2.77 )

 Q. Two 70-by-70 matrices with only 143,640 scalar multiplications?

 A. Yes! [Pan, 1980] Θ (n log70 143640 ) =
O(n 2.80 )

Divide and Conquer Technique 60

 Fast Matrix Multiplication in Theory
 Decimal wars.
 December, 1979: O(n2.521813).
 January, 1980: O(n2.521801).

 Best known. O(n2.376) [Coppersmith-Winograd, 1987.]

 Conjecture. O(n2+ε) for any ε > 0.

 Caveat. Theoretical improvements to Strassen are

progressively less practical.

Divide and Conquer Technique 61

 Observations
 Comparison: n= 70
 Direct multiplication: 703 = 343,000
 Strassen: 70lg 7 is approximately 150,000
 Crossover point typically around n = 20
 Hopcroft and Kerr have shown 7 multiplications are
necessary to multiply 2 by 2 matrices
 But we can do better with larger matrices
 Best lower bound is Ω(n2) (since there are n2 entries)
 Matrix multiplication can be used in some graph
algorithms as a fundamental step, so theoretical
improvements in the efficiency of this operation can lead
to theoretical improvements in those algorithms

Divide and Conquer Technique 62

Closest Pair of Points
 Closest Pair of Points
 Closest pair. Given n points in the plane, find a pair with smallest
Euclidean distance between them.
 Fundamental geometric primitive.
 Graphics, computer vision, geographic information systems,
molecular modeling, air traffic control.
 Special case of nearest neighbor, Euclidean MST, Voronoi.

fast closest pair inspired fast algorithms for these problems

 Brute force. Check all pairs of points p and q with Θ(n2)

comparisons.
 1-D version. O(n log n) easy if points are on a line.
 Assumption. No two points have same x coordinate.

to make presentation cleaner

Divide and Conquer Technique 64

 Closest Pair of Points: First Attempt
 Divide. Sub-divide region into 4 quadrants.

Divide and Conquer Technique 65

 Closest Pair of Points: First Attempt
 Divide. Sub-divide region into 4 quadrants.
 Obstacle. Impossible to ensure n/4 points in
each piece.

Divide and Conquer Technique 66

 Closest Pair of Points
 Algorithm.
 Divide: draw vertical line L so that roughly ½n points on each side.

Divide and Conquer Technique 67

 Closest Pair of Points
 Algorithm.
 Divide: draw vertical line L so that roughly ½n points on each side.
 Conquer: find closest pair in each side recursively.

21

12

Divide and Conquer Technique 68

 Closest Pair of Points
 Algorithm.
 Divide: draw vertical line L so that roughly ½n points on each side.
 Conquer: find closest pair in each side recursively. seems like Θ(n )
2

 Combine: find closest pair with one point in each side.

 Return best of 3 solutions.

8
21

12

Divide and Conquer Technique 69

 Closest Pair of Points
 Find closest pair with one point in each side, assuming that distance < δ.

21

δ = min(12, 21)
12

Divide and Conquer Technique 70

 Closest Pair of Points
 Find closest pair with one point in each side, assuming that distance < δ.
 Observation: only need to consider points within δ of line L.

21

δ = min(12, 21)
12

Divide and Conquer Technique δ 71

 Closest Pair of Points
 Find closest pair with one point in each side, assuming that distance < δ.
 Observation: only need to consider points within δ of line L.
 Sort points in 2δ-strip by their y coordinate.

L
7

5
4 21

δ = min(12, 21)
12 3

Divide and Conquer Technique δ 72

 Closest Pair of Points
 Find closest pair with one point in each side, assuming that distance < δ.
 Observation: only need to consider points within δ of line L.
 Sort points in 2δ-strip by their y coordinate.
 Only check distances of those within 11 positions in sorted list!

L
7

5
4 21

δ = min(12, 21)
12 3

Divide and Conquer Technique δ 73

 Closest Pair of Points
 Def. Let si be the point in the 2δ-strip, with
the ith smallest y-coordinate.
39 j
31
 Claim. If |i – j| ≥ 12, then the distance between
si and sj is at least δ.
 Pf. ½δ
 2 rows
No two points lie in same ½δ-by-½δ box.
29
30
½δ
 Two points at least 2 rows apart
have distance ≥ 2(½δ). ▪ i 27
28 ½δ

26

 Fact. Still true if we replace 12 with 7. 25

Divide and Conquer Technique δ δ 74

 Closest Pair Algorithm
Closest-Pair(p1, …, pn) {
Compute separation line L such that half the points O(n log n)
are on one side and half on the other side.

δ 1 = Closest-Pair(left half) 2T(n / 2)

δ 2 = Closest-Pair(right half)
δ = min(δ 1, δ 2)
O(n)
Delete all points further than δ from separation line L
O(n log n)
Sort remaining points by y-coordinate.

Scan points in y-order and compare distance between O(n)

each point and next 11 neighbors. If any of these
distances is less than δ, update δ.

return δ.
}

Divide and Conquer Technique 75

 Closest Pair of Points: Analysis
 Running time.

 Q. Can we achieve O(n log n)?

 A. Yes. Don't sort points in strip from scratch each time.

 Each recursive returns two lists: all points sorted by y
coordinate, and all points sorted by x coordinate.
 Sort by merging two pre-sorted lists.

Divide and Conquer Technique 76

 Other Applications
 Find median (Probabilistic)
 Convex Hull
 Reversing array
 Maximum consecutive subsequence
Etc…

Divide and Conquer Technique 77