Binder1 PDF

Introduction: Analysis of
Algorithms, Insertion Sort,

Merge Sort
Lecture 1
Analysis of algorithms
The theoretical study of computer-program
performance and resource usage.
Whats more important than performance?
modularity user-friendliness
correctness programmer time
maintainability simplicity
functionality extensibility
robustness reliability
L1.2
Why study algorithms and
performance?
Algorithms help us to understand scalability.
Performance often draws the line between what
is feasible and what is impossible.
Algorithmic mathematics provides a language
for talking about program behavior.
The lessons of program performance generalize
to other computing resources.
Speed is fun!
L1.3
The problem of sorting
Input: sequence a1, a2, , an of numbers.
Output: permutation a'1, a'2, , a'n such

that a'1 a'2 a'n .
Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
L1.4
Insertion Sort
INSERTION-SORT (A, n) A[1 . . n]
for j 1 to n
do key A[ j]
ij1
pseudocode while i > 0 and A[i] > key
do A[i+1] A[i]
ii1
A[i+1] = key
i j
key
sorted
L1.5
Example of Insertion Sort
8 2 4 9 3 6
L1.6
8 2 4 9 3 6
L1.7
8 2 4 9 3 6
2 8 4 9 3 6
L1.8
8 2 4 9 3 6
2 8 4 9 3 6
L1.9
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
L1.10
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
L1.11
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
L1.12
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
L1.13
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
L1.14
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
L1.15
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
2 3 4 6 8 9 done
L1.16
Running time
The running time depends on the input: an

already sorted sequence is easier to sort.
Parameterize the running time by the size of
the input, since short sequences are easier to
sort than long ones.
Generally, we seek upper bounds on the
running time, because everybody likes a
guarantee.
L1.17
Kinds of analyses
Worst-case: (usually)
T(n) = maximum time of algorithm
on any input of size n.
Average-case: (sometimes)
T(n) = expected time of algorithm
on any input of size n.
Best-case: (bogus)
Cheat with a slow algorithm that
works fast on some input.
L1.18
Machine-independent time
What is insertion sorts worst-case time?

It depends on the speed of our computer:
relative speed (on the same machine),
absolute speed (on different machines).
BIG IDEA:
Ignore machine-dependent constants.
Look at growth of T(n) as n .
Asymptotic Analysis
L1.19
Q-notation
Math:
Q(g(n)) = { f (n) : there exist positive constants c1, c2, and
n0 such that 0 c1 g(n) f (n) c2 g(n)
for all n n0 }
Engineering:
Drop low-order terms; ignore leading constants.
Example: 3n3 + 90n2 5n + 6046 = Q(n3)
L1.20
Asymptotic performance
When n gets large enough, a Q(n2) algorithm
always beats a Q(n3) algorithm.
We shouldnt ignore
asymptotically slower
algorithms, however.
Real-world design
situations often call for a
T(n) careful balancing of
engineering objectives.
Asymptotic analysis is a
useful tool to help to
n n0 structure our thinking.
L1.21
Insertion sort analysis
Worst case: Input reverse sorted.
n
T ( n) Q ( j ) Qn 2 [arithmetic series]
j 2
Average case: All permutations equally likely.
n
T ( n) Q( j / 2) Qn 2
j 2
Is insertion sort a fast sorting algorithm?
Moderately so, for small n.
Not at all, for large n.
L1.22
Merge sort
MERGE-SORT A[1 . . n]
To sort n numbers:
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ] and
A[ n/2+1 . . n ] .
3. Merge the 2 sorted lists.
Key subroutine: MERGE
L1.23
Merging two sorted arrays
20 12
13 11
7 9
2 1
L1.24
20 12
13 11
7 9
2 1
L1.25
20 12 20 12
13 11 13 11
7 9 7 9
2 1 2
L1.26
20 12 20 12
13 11 13 11
7 9 7 9
2 1 2
1 2
L1.27
20 12 20 12 20 12
13 11 13 11 13 11
7 9 7 9 7 9
2 1 2
1 2
L1.28
20 12 20 12 20 12
13 11 13 11 13 11
7 9 7 9 7 9
2 1 2
1 2 7
L1.29
20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7
L1.30
20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7 9
L1.31
20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7 9
L1.32
20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11
L1.33
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11
L1.34
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11 12
L1.35
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11 12
Time = Q(n) to merge a total

of n elements (linear time).
L1.36
Analyzing merge sort
MERGE-SORT (A, n) A[1 . . n]
T(n) To sort n numbers:
Q(1) 1. If n = 1, done.
2T(n/2) 2. Recursively sort A[ 1 . . n/2 ]
Abuse and A[ n/2+1 . . n ] .
Q(n) 3. Merge the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) ,
but it turns out not to matter asymptotically.
L1.37
Recurrence for merge sort
Q(1) if n = 1;
T(n) =
2T(n/2) + Q(n) if n > 1.
We shall usually omit stating the base
case when T(n) = Q(1) for sufficiently
small n (and when it has no effect on the
solution to the recurrence.
Lecture 2 provide several ways to find a
good upper bound on T(n).
L1.38
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
L1.39
Recursion tree
T(n)
L1.40
Recursion tree
cn
T(n/2) T(n/2)
L1.41
Recursion tree
cn
cn/2 cn/2
T(n/4) T(n/4) T(n/4) T(n/4)
L1.42
Recursion tree
cn
cn/2 cn/2
cn/4 cn/4 cn/4 cn/4
Q(1)
L1.43
Recursion tree
cn
cn/2 cn/2
h = lg n cn/4 cn/4 cn/4 cn/4
Q(1)
L1.44
Recursion tree
cn cn
cn/2 cn/2
Q(1)
L1.45
Recursion tree
cn cn
cn/2 cn/2 cn
Q(1)
L1.46
Recursion tree
cn cn
cn/2 cn/2 cn
h = lg n cn/4 cn/4 cn/4 cn/4 cn

Q(1)
L1.47
Recursion tree
cn cn
cn/2 cn/2 cn

Q(1) #leaves = n Q(n)
L1.48
Recursion tree
cn cn
cn/2 cn/2 cn

Q(1) #leaves = n Q(n)
TotalQ(n lg n)
L1.49
Conclusions
Q(n lg n) grows more slowly than Q(n2).

Therefore, merge sort asymptotically
beats insertion sort in the worst case.
In practice, merge sort beats insertion
sort for n > 30 or so.
Go test it out for yourself!
L1.50
Asymptotic Notation,
Recurrences: Substitution,
Iteration, Master Method
Lecture 2
Solving recurrences
The analysis of merge sort from
Lecture 1 required us to solve a
recurrence.
Recurrences are like solving integrals,
differential equations, etc.
Learn a few tricks.
Lecture 3: Applications of recurrences.
L2.2
Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
Example: T(n) = 4T(n/2) + n
[Assume that T(1) = Q(1).]
Guess O(n3) . (Prove O and W separately.)
Assume that T(k) ck3 for k < n .
Prove T(n) cn3 by induction.
L2.3
Example of substitution
T (n) 4T (n / 2) n
4c(n / 2)3 n
(c / 2)n3 n
cn3 ((c / 2)n3 n) desired residual
cn3 desired
whenever (c/2)n3 n 0, for
example, if c 2 and n 1.
residual
L2.4
Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = Q(1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have Q(1) cn3, if we
pick c big enough.
This bound is not tight!

L2.5
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ck2 for k < n:
T (n) 4T (n / 2) n
4cn 2 n
O(n) Wrong! We must prove the I.H.
cn 2 (n) [ desired residual ]
cn 2
for no choice of c > 0. Lose!
L2.6
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
Subtract a low-order term.
Inductive hypothesis: T(k) c1k2 c2k for k < n.
T (n) 4T (n / 2) n
4(c1 (n / 2) 2 c2 (n / 2) n
c1n 2 2c2 n n
c1n 2 c2 n (c2 n n)
c1n 2 c2 n if c2 > 1.
Pick c1 big enough to handle the initial conditions.
L2.7
Recursion-tree method
A recursion tree models the costs (time) of a
recursive execution of an algorithm.
The recursion tree method is good for
generating guesses for the substitution method.
The recursion-tree method can be unreliable,
just like any method that uses ellipses ().
The recursion-tree method promotes intuition,
however.
L2.8
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
L2.9
Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)
L2.10
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4) T(n/2)
L2.11
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2
T(n/16) T(n/8) T(n/8) T(n/4)
L2.12
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2
(n/16)2 (n/8)2 (n/8)2 (n/4)2
Q(1)
L2.13
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
(n/4)2 (n/2)2
(n/16)2 (n/8)2 (n/8)2 (n/4)2
Q(1)
L2.14
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4)2 (n/2)2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2
Q(1)
L2.15
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4)2 (n/2)2
16
25 n 2
(n/16)2 (n/8)2 (n/8)2 (n/4)2
256

Q(1)
L2.16
Solve T(n) = T(n/4) + T(n/2) + n2:
n2 n2
5 n2
(n/4)2 (n/2)2
16
25 n 2
(n/16)2 (n/8)2 (n/8)2 (n/4)2
256

Q(1) Total = n2

1 16 16
5 5 2 5 3
16

= Q(n2) geometric series
L2.17
The master method
The master method applies to recurrences of

the form
T(n) = a T(n/b) + f (n) ,
where a 1, b > 1, and f is asymptotically
positive.
L2.18
Three common cases
Compare f (n) with nlogba:
1. f (n) = O(nlogba e) for some constant e > 0.
f (n) grows polynomially slower than nlogba
(by an ne factor).
Solution: T(n) = Q(nlogba) .
2. f (n) = Q(nlogba lgkn) for some constant k 0.
f (n) and nlogba grow at similar rates.
Solution: T(n) = Q(nlogba lgk+1n) .
L2.19
Three common cases (cont.)
Compare f (n) with nlogba:
3. f (n) = W(nlogba + e) for some constant e > 0.
f (n) grows polynomially faster than nlogba (by
an ne factor),
and f (n) satisfies the regularity condition that
a f (n/b) c f (n) for some constant c < 1.
Solution: T(n) = Q( f (n) ) .
L2.20
Examples
Ex. T(n) = 4T(n/2) + n

a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 e) for e = 1.
T(n) = Q(n2).
Ex. T(n) = 4T(n/2) + n2

a = 4, b = 2 nlogba = n2; f (n) = n2.
CASE 2: f (n) = Q(n2lg0n), that is, k = 0.
T(n) = Q(n2lg n).
L2.21
Examples
Ex. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = W(n2 + e) for e = 1
and 4(cn/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = Q(n3).
Ex. T(n) = 4T(n/2) + n2/lg n

a = 4, b = 2 nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant e > 0, we have ne w(lg n).
L2.22
General method (Akra-Bazzi)
k
T (n) aiT (n / bi ) f (n)
i 1
Let p be the unique solution to
i i 1.

k
p
a /b
i 1
Then, the answers are the same as for the
master method, but with np instead of nlogba.
(Akra and Bazzi also prove an even more
general result.)
L2.23
Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)

#leaves = ah
= alogbn nlogbaT (1)
T (1)
= nlogba
L2.24
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)

CASE 1: The weight increases
geometrically from the root to the nlogbaT (1)
T (1) leaves. The leaves hold a constant
fraction of the total weight. Q(nlogba)
L2.25
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)

CASE 2: (k = 0) The weight
nlogbaT (1)
T (1) is approximately the same on
each of the logbn levels.
Q(nlogbalg n)
L2.26
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)

CASE 3: The weight decreases
geometrically from the root to the nlogbaT (1)
T (1) leaves. The root holds a constant
fraction of the total weight. Q( f (n))
L2.27
Conclusion
Next time: applying the master method.

For proof of master theorem, see text book.
L2.28
Divide and Conquer:
Strassen's Algorithm,
Fibonacci Numbers
Lecture 3
The divide-and-conquer
design paradigm
1. Divide the problem (instance)
into subproblems.
2. Conquer the subproblems by
solving them recursively.
3. Combine subproblem solutions.
L3.2
Example: merge sort
1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Linear-time merge.
T(n) = 2 T(n/2) + O(n)
# subproblems work dividing

and combining
subproblem size
L3.3
Master theorem (reprise)
T(n) = a T(n/b) + f (n)
CASE 1: f (n) = O(nlogba e)
T(n) = Q(nlogba) .
CASE 2: f (n) = Q(nlogba lgkn)
T(n) = Q(nlogba lgk+1n) .
CASE 3: f (n) = W(nlogba + e) and a f (n/b) c f (n)
T(n) = Q( f (n)) .
Merge sort: a = 2, b = 2 nlogba = n
CASE 2 (k = 0) T(n) = Q(n lg n) .
L3.4
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15
L3.5
Binary search
Example: Find 9
3 5 7 8 9 12 15
L3.6
Binary search
Example: Find 9
3 5 7 8 9 12 15
L3.7
Binary search
Example: Find 9
3 5 7 8 9 12 15
L3.8
Binary search
Example: Find 9
3 5 7 8 9 12 15
L3.9
Binary search
Example: Find 9
3 5 7 8 9 12 15
L3.10
Recurrence for binary search
T(n) = 1 T(n/2) + Q(1)
# subproblems work dividing

and combining
subproblem size
nlogba = nlog21 = n0 = 1 CASE 2 (k = 0)

T(n) = Q(lg n) .
L3.11
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: Q(n).
Divide-and-conquer algorithm:
a n/2 a n/2 if n is even;
an =
a (n1)/2 a (n1)/2 a if n is odd.
T(n) = T(n/2) + Q(1) T(n) = Q(lg n) .
L3.12
Fibonacci numbers
Recursive definition:
0 if n = 0;
Fn = 1 if n = 1;
Fn1 + Fn2 if n 2.
0 1 1 2 3 5 8 13 21 34 L
Naive recursive algorithm: W(fn)
(exponential time), where f = (1 5) / 2
is the golden ratio.
L3.13
Computing Fibonacci
numbers
Naive recursive squaring:
Fn = fn/ 5 rounded to the nearest integer.
Recursive squaring: Q(lg n) time.
This method is unreliable, since floating-point
arithmetic is prone to round-off errors.
Bottom-up:
Compute F0, F1, F2, , Fn in order, forming
each number by summing the two previous.
Running time: Q(n).
L3.14
Recursive squaring
n
Fn 1 Fn 1 1
Theorem: .
Fn Fn 1 1 0
Algorithm: Recursive squaring.
Time = Q(lg n) .
Proof of theorem. (Induction on n.)
1
2
F F1 1 1
Base (n = 1): .
1
F F0 1 0
L3.15
Recursive squaring
Inductive step (n 2):
.
Fn 1 Fn Fn Fn 1 1 1
F
n Fn 1 Fn 1 Fn 2 1 0
n1
1 1 1 1
.
1 0 1 0
n
1 1

1 0
L3.16
Matrix multiplication
Input: A = [aij], B = [bij].
i, j = 1, 2, , n.
Output: C = [cij] = AB.
c11 c12 L c1n a11 a12 L a1n b11 b12 L b1n
c c L c2n a21 a22 L a2 n b21 b22 L b2n
21 22

c c L cnn an1 an 2 L ann bn1 bn 2 L bnn
n1 n 2
n
cij aik bkj
k 1
L3.17
Standard algorithm
for i 1 to n
do for j 1 to n
do cij 0
for k 1 to n
do cij cij + aik bkj
Running time = Q(n3)
L3.18
Divide-and-conquer algorithm
IDEA:
nn matrix = 22 matrix of (n/2)(n/2) submatrices:
r s a b e f

t u c d g h

C = A B
r = ae + bg
s = af + bh 8 mults of (n/2)(n/2) submatrices
t = ce + dh 4 adds of (n/2)(n/2) submatrices
u = cf + dh
L3.19
Analysis of D&C algorithm
T(n) = 8 T(n/2) + Q(n2)
# submatrices work adding

submatrices
submatrix size
nlogba = nlog28 = n3 CASE 1 T(n) = Q(n3).
No better than the ordinary algorithm.
L3.20
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h) r = P 5 + P 4 P2 + P 6
P2 = (a + b) h s = P1 + P2
P3 = (c + d) e t = P3 + P4
P4 = d (g e) u = P5 + P1 P3 P7
P5 = (a + d) (e + h)
P6 = (b d) (g + h) 7 mults, 18 adds/subs.
P7 = (a c) (e + f ) Note: No reliance on
commutativity of mult!
L3.21
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h) r = P 5 + P 4 P2 + P 6
P2 = (a + b) h = (a + d) (e + h)
P3 = (c + d) e + d (g e) (a + b) h
P4 = d (g e) + (b d) (g + h)
P5 = (a + d) (e + h) = ae + ah + de + dh
P6 = (b d) (g + h) + dg de ah bh
P7 = (a c) (e + f ) + bg + bh dg dh
= ae + bg
L3.22
Strassens algorithm
1. Divide: Partition A and B into
(n/2)(n/2) submatrices. Form terms
to be multiplied using + and .
2. Conquer: Perform 7 multiplications of
(n/2)(n/2) submatrices recursively.
3. Combine: Form C using + and on
(n/2)(n/2) submatrices.
T(n) = 7 T(n/2) + Q(n2)
L3.23
Analysis of Strassen
T(n) = 7 T(n/2) + Q(n2)
nlogba = nlog27 n2.81 CASE 1 T(n) = Q(nlg 7).
The number 2.81 may not seem much smaller than
3, but because the difference is in the exponent, the
impact on running time is significant. In fact,
Strassens algorithm beats the ordinary algorithm
on todays machines for n 30 or so.
Best to date (of theoretical interest only): Q(n2.376L).
L3.24
Conclusion
Divide and conquer is just one of several

powerful techniques for algorithm design.
Divide-and-conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
Can lead to more efficient algorithms
L3.25
Quicksort, Randomized
Algorithms
Lecture 4
Quicksort
Proposed by C.A.R. Hoare in 1962.
Divide-and-conquer algorithm.
Sorts in place (like insertion sort, but not
like merge sort).
Very practical (with tuning).
L4.2
Divide and conquer
Quicksort an n-element array:
1. Divide: Partition the array into two subarrays
around a pivot x such that elements in lower
subarray x elements in upper subarray.
x x x
2. Conquer: Recursively sort the two subarrays.
Key: Linear-time partitioning subroutine.
L4.3
Partitioning subroutine
PARTITION(A, p, q) A[ p . . q]
x A[ p] pivot = A[ p] Running time
ip = O(n) for n
for j p + 1 to q elements.
do if A[ j] x
then i i + 1
exchange A[i] A[ j]
exchange A[ p] A[i]
return i
Invariant: x x x ?
p i j q
L4.4
Example of partitioning
6 10 13 5 8 3 2 11
i j
L4.5
6 10 13 5 8 3 2 11
i j
L4.6
6 10 13 5 8 3 2 11
i j
L4.7
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
i j
L4.8
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
i j
L4.9
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
i j
L4.10
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
6 5 3 10 8 13 2 11
i j
L4.11
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
6 5 3 10 8 13 2 11
i j
L4.12
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
6 5 3 10 8 13 2 11
6 5 3 2 8 13 10 11
i j
L4.13
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
6 5 3 10 8 13 2 11
6 5 3 2 8 13 10 11
i j
L4.14
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
6 5 3 10 8 13 2 11
6 5 3 2 8 13 10 11
i j
L4.15
6 10 13 5 8 3 2 11
6 5 13 10 8 3 2 11
6 5 3 10 8 13 2 11
6 5 3 2 8 13 10 11
2 5 3 6 8 13 10 11
i
L4.16
Pseudocode for quicksort
QUICKSORT(A, p, r)
if p < r
then q PARTITION(A, p, r)
QUICKSORT(A, p, q)
QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)
L4.17
Analysis of quicksort
Assume all input elements are distinct.

In practice, there are better partitioning
algorithms for when duplicate input
elements may exist.
Let T(n) = worst-case running time on
an array of n elements.
L4.18
Worst-case of quicksort
Input sorted or reverse sorted.
Partition around min or max element.
One side of partition always has no elements.
T (n) T (0) T (n 1) (n)
(1) T (n 1) (n)
T (n 1) (n)
( n 2 ) (arithmetic series)
L4.19
Worst-case recursion tree
T(n) = T(0) + T(n1) + cn
L4.20
T(n) = T(0) + T(n1) + cn
T(n)
L4.21
T(n) = T(0) + T(n1) + cn
cn
T(0) T(n1)
L4.22
T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) T(n2)
L4.23
T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) c(n2)
T(0)
(1)
L4.24
T(n) = T(0) + T(n1) + cn
cn n
T(0) c(n1) k n 2
k 1
T(0) c(n2)
T(0)
(1)
L4.25
T(n) = T(0) + T(n1) + cn
cn n
(1) c(n1) k n 2
k 1
(1) c(n2)
h=n T(n) = (n) + (n2)
(1) = (n2)
(1)
L4.26
Best-case analysis
(For intuition only!)
If were lucky, PARTITION splits the array evenly:
T(n) = 2T(n/2) + (n)
= (n lg n) (same as merge sort)
1 9
What if the split is always 10 : 10 ?
T (n) T 101 n T 109 n (n)
What is the solution to this recurrence?
L4.27
Analysis of almost-best case
T (n)
L4.28
cn
T 101 n T 109 n
L4.29
cn
1
10
cn 9
10
cn
T 100
1
n T 100
9
n T 100
9
n T 100
81
n
L4.30
cn cn
1
10
cn 9
10
cn cn
log10/9n
1 9 9 81
100
cn 100
cn 100
cn 100
cn cn

(1) O(n) leaves
(1)
L4.31
cn cn
1
10
cn 9
10
cn cn
log10 log10/9n
n 1 cn 9
cn 9
cn 81
cn cn
100 100 100 100

(1) O(n) leaves
(n lg n) (1)
Lucky! cn log10n T(n) cn log10/9n + O(n)
L4.32
More intuition
Suppose we alternate lucky, unlucky,
lucky, unlucky, lucky, .
L(n) = 2U(n/2) + (n) lucky
U(n) = L(n 1) + (n) unlucky
Solving:
L(n) = 2(L(n/2 1) + (n/2)) + (n)
= 2L(n/2 1) + (n)
= (n lg n) Lucky!
How can we make sure we are usually lucky?
L4.33
Randomized quicksort
IDEA: Partition around a random element.
Running time is independent of the input
order.
No assumptions need to be made about
the input distribution.
No specific input elicits the worst-case
behavior.
The worst case is determined only by the
output of a random-number generator.
L4.34
Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
1 if PARTITION generates a k : nk1 split,
Xk =
0 otherwise.
E[Xk] = Pr{Xk = 1} = 1/n, since all splits are
equally likely, assuming elements are distinct.
L4.35
Analysis (continued)
T(0) + T(n1) + (n) if 0 : n1 split,
T(1) + T(n2) + (n) if 1 : n2 split,
T(n) = M
T(n1) + T(0) + (n) if n1 : 0 split,
n 1
X k T (k ) T (n k 1) (n) .
k 0
L4.36
Calculating expectation
n 1
E[T (n)] E X k T (k ) T (n k 1) (n)
k 0
Take expectations of both sides.
L4.37
n 1
E[T (n)] E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k T (k ) T (n k 1) (n)
k 0
Linearity of expectation.
L4.38
n 1
E[T (n)] E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k ET (k ) T (n k 1) (n)
k 0
Independence of Xk from other random

choices.
L4.39
n 1
E[T (n)] E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k ET (k ) T (n k 1) (n)
k 0
n 1 n 1 n 1
1 E T (k ) 1 E T (n k 1) 1 (n)
n k 0 n k 0 n k 0
Linearity of expectation; E[Xk] = 1/n .
L4.40
n 1
E[T (n)] E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k T (k ) T (n k 1) (n)
k 0
n 1
E X k ET (k ) T (n k 1) (n)
k 0
n 1 n 1 n 1
1 E T (k ) 1 E T (n k 1) 1 (n)
n k 0 n k 0 n k 0
n 1
2 E T (k ) (n) Summations have
n k 1
identical terms.
L4.41
Hairy recurrence
n 1
E[T (n)] 2 ET (k ) (n)
n k 2
(The k = 0, 1 terms can be absorbed in the (n).)
Prove: E[T(n)] a n lg n for constant a > 0 .
Choose a large enough so that a n lg n
dominates E[T(n)] for sufficiently small n 2.
n 1
Use fact: k lg k 1 n 2 lg n 1n 2
2 8 (exercise).
k 2
L4.42
Substitution method
n 1
ET (n) 2 ak lg k (n)
n k 2
Substitute inductive hypothesis.
L4.43
Substitution method
n 1
ET (n) 2 ak lg k (n)
n k 2
2a 1 n 2 lg n 1 n 2 (n)
n 2 8
Use fact.
L4.44
Substitution method
n 1
E T (n) 2 ak lg k (n)
n k 2
2a 1 n 2 lg n 1 n 2 (n)
n 2 8
an lg n an (n)
4
Express as desired residual.
L4.45
Substitution method
n 1
E T (n) 2 ak lg k (n)
n k 2
2a 1 n 2 lg n 1 n 2 (n)
n 2 8
an lg n an (n)
4
an lg n ,
if a is chosen large enough so that
an/4 dominates the (n).
L4.46
Quicksort in practice
Quicksort is a great general-purpose

sorting algorithm.
Quicksort is typically over twice as fast
as merge sort.
Quicksort can benefit substantially from
code tuning.
Quicksort behaves well even with
caching and virtual memory.
L4.47
Heapsort
Lecture 4.1
1
Priority Queues
Heaps
Heapsort
2
Priority Queue
A data structure implementing a set S of elements, each
associated with a key, supporting the following operations:
insert(S, x) : insert element x into set S

max(S) : return element of S with largest key
extract_max(S) : return element of S with largest key and
remove it from S
increase_key(S, x, k) : increase the value of element x s key to
new value k
(assumed to be as large as current value)
3
Heap
Implementation of a priority queue
An array, visualized as a nearly complete binary tree
Max Heap Property: The key of a node is than the keys of
its children
(Min Heap defined analogously)
1
2
16 3
14 10
5 6 7 1 2 3 4 5 6 7 8 9 10
4
8 7 9 3 16 14 10 8 7 9 3 2 4 1
8 9 10
2 4 1
4
All my arrays start at index 1
Heap as a Tree
root of tree: first element in the array, corresponding to i = 1
parent(i) =i/2: returns index of node's parent
left(i)=2i: returns index of node's left child
right(i)=2i+1: returns index of node's right child
1
2
16 3
14 10
5 6 7 1 2 3 4 5 6 7 8 9 10
4
8 7 9 3 16 14 10 8 7 9 3 2 4 1
8 9 10
2 4 1
No pointers required! Height of a binary heap is O(lg n)

5
OperationsHeap
with Heaps
Operations
build_max_heap : produce a max-heap from an unordered
array
max_heapify : correct a single violation of the heap

property in a subtree at its root
insert, extract_max, heapsort
6
Max_heapify
Assume that the trees rooted at left(i) and right(i)

are max-heaps
If element A[i] violates the max-heap property, correct

violation by trickling element A[i] down the tree,
making the subtree rooted at index i a max-heap
7
Max_heapify (Example)
Node 10 is the left child of node 5 but is drawn to the right for convenience
8
9
Time=? O(log n)
10
Max_Heapify Pseudocode
l = left(i)
r = right(i)
if (l <= heap-size(A) and A[l] > A[i])
then largest = l else largest = i
if (r <= heap-size(A) and A[r] > A[largest])
then largest = r
if largest = i
then exchange A[i] and A[largest]
Max_Heapify(A, largest)
11
Build_Max_Heap(A)
Converts A[1n] to a max heap
Build_Max_Heap(A):
for i=n/2 downto 1
do Max_Heapify(A, i)
Why start at n/2?
Because elements A[n/2 + 1 n] are all leaves of the tree

2i > n, for i > n/2 + 1
Time=? O(n log n) via simple analysis

12
Build_Max_Heap(A) Analysis
Build_Max_Heap(A):
for i=n/2 downto 1
Observe however that Max_Heapify takes O(1) for

time for nodes that are one level above the leaves, and
in general, O(l) for the nodes that are l levels above the
leaves. We have n/4 nodes with level 1, n/8 with level 2,
and so on till we have one root node that is lg n levels
above the leaves.
13
Build_Max_Heap(A) Analysis
Build_Max_Heap(A):
for i=n/2 downto 1
Total amount of work in the for loop can be summed as:

n/4 (1 c) + n/8 (2 c) + n/16 (3 c) + + 1 (lg n c)
Setting n/4 = 2k and simplifying we get:
c 2k( 1/20 + 2/21 + 3/22 + (k+1)/2k )
The term is brackets is bounded by a constant!
This means that Build_Max_Heap is O(n)

14
Build-Max-Heap Demo
15
Build-Max-Heap Demo
16
Build-Max-Heap
17
Heap-Sort
Sorting Strategy:
1. Build Max Heap from unordered array;
18
Heap-Sort
Sorting Strategy:
2. Find maximum element A[1];
3. Swap elements A[n] and A[1]:

now max element is at the end of the array!
19
Heap-Sort
Sorting Strategy:

4. Discard node n from heap
(by decrementing heap-size variable)
20
Heap-Sort
Sorting Strategy:

5. New root may violate max heap property, but its
children are max heaps. Run max_heapify to fix this.
21
Heap-Sort
Sorting Strategy:

5. New root may violate max heap property, but its
children are max heaps. Run max_heapify to fix this.
6. Go to Step 2 unless heap is empty.
22
Heap-Sort Demo
Swap A[10] and A[1]
Max_heapify(A,1)
23
Heap-Sort Demo
Swap A[9] and A[1]
24
Heap-Sort Demo
25
Heap-Sort Demo
Swap A[8] and A[1]
26
Heap-Sort
Running time:
after n iterations the Heap is empty
every iteration involves a swap and a max_heapify
operation; hence it takes O(log n) time
Overall O(n log n)
27
Decision Tree, Linear-time
Sorting, Lower Bounds,
Counting Sort, Radix Sort
Lecture 5
How fast can we sort?
All the sorting algorithms we have seen so far
are comparison sorts: only use comparisons to
determine the relative order of elements.
E.g., insertion sort, merge sort, quicksort,
heapsort.
The best worst-case running time that weve
seen for comparison sorting is O(n lg n) .
Is O(n lg n) the best we can do?
Decision trees can help us answer this question.
L5.2
Decision-tree example
Sort a1, a2, , an 1:2
2:3 1:3
123 1:3 213 2:3
132 312 231 321
Each internal node is labeled i:j for i, j {1, 2,, n}.

The left subtree shows subsequent comparisons if ai aj.
The right subtree shows subsequent comparisons if ai aj.
L5.3
Sort a1, a2, a3 1:2 94
= 9, 4, 6 :
2:3 1:3
123 1:3 213 2:3
132 312 231 321

L5.4
Sort a1, a2, a3 1:2
= 9, 4, 6 :
2:3 1:3 96
123 1:3 213 2:3
132 312 231 321

L5.5
Sort a1, a2, a3 1:2
= 9, 4, 6 :
2:3 1:3
123 1:3 213

4 6 2:3
132 312 231 321

L5.6
Sort a1, a2, a3 1:2
= 9, 4, 6 :
2:3 1:3
123 1:3 213 2:3
132 312 231 321

469
Each leaf contains a permutation (1), (2),, (n) to
indicate that the ordering a(1) a(2) a(n) has been
established.
L5.7
Decision-tree model
A decision tree can model the execution of
any comparison sort:
One tree for each input size n.
View the algorithm as splitting whenever
it compares two elements.
The tree contains the comparisons along
all possible instruction traces.
The running time of the algorithm = the
length of the path taken.
Worst-case running time = height of tree.
L5.8
Lower bound for decision-
tree sorting
Theorem. Any decision tree that can sort n
elements must have height (n lg n) .
Proof. The tree must contain n! leaves, since
there are n! possible permutations. A height-h
binary tree has 2h leaves. Thus, n! 2h .
h lg(n!) (lg is mono. increasing)
lg ((n/e)n) (Stirlings formula)
= n lg n n lg e
= (n lg n) .
L5.9
Lower bound for comparison
sorting
Corollary. Heapsort and merge sort are
asymptotically optimal comparison sorting
algorithms.
L5.10
Sorting in linear time
Counting sort: No comparisons between elements.
Input: A[1 . . n], where A[ j]{1, 2, , k} .
Output: B[1 . . n], sorted.
Auxiliary storage: C[1 . . k] .
L5.11
Counting sort
for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
L5.12
Counting-sort example
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C:
B:
L5.13
Loop 1
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 0 0 0 0
B:
for i 1 to k
do C[i] 0
L5.14
Loop 2
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 0 0 0 1
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
L5.15
Loop 2
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 0 1
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
L5.16
Loop 2
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 1 1
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
L5.17
Loop 2
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 1 2
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
L5.18
Loop 2
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 2 2
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
L5.19
Loop 3
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 2 2
B: C': 1 1 2 2
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
L5.20
Loop 3
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 2 2
B: C': 1 1 3 2
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
L5.21
Loop 3
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 0 2 2
B: C': 1 1 3 5
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
L5.22
Loop 4
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 1 3 5
B: 3 C': 1 1 2 5
for j n downto 1
C[A[ j]] C[A[ j]] 1
L5.23
Loop 4
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 1 2 5
B: 3 4 C': 1 1 2 4
for j n downto 1
C[A[ j]] C[A[ j]] 1
L5.24
Loop 4
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 1 2 4
B: 3 3 4 C': 1 1 1 4
for j n downto 1
C[A[ j]] C[A[ j]] 1
L5.25
Loop 4
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 1 1 1 4
B: 1 3 3 4 C': 0 1 1 4
for j n downto 1
C[A[ j]] C[A[ j]] 1
L5.26
Loop 4
1 2 3 4 5 1 2 3 4
A: 4 1 3 4 3 C: 0 1 1 4
B: 1 3 3 4 4 C': 0 1 1 3
for j n downto 1
C[A[ j]] C[A[ j]] 1
L5.27
Analysis
for i 1 to k
(k) do C[i] 0
for j 1 to n
(n) do C[A[ j]] C[A[ j]] + 1
for i 2 to k
(k) do C[i] C[i] + C[i1]
for j n downto 1
(n) do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
(n + k)
L5.28
Running time
If k = O(n), then counting sort takes (n) time.
But, sorting takes (n lg n) time!
Wheres the fallacy?
Answer:
Comparison sorting takes (n lg n) time.
Counting sort is not a comparison sort.
In fact, not a single comparison between
elements occurs!
L5.29
Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.
A: 4 1 3 4 3
B: 1 3 3 4 4
Exercise: What other sorts have this property?
L5.30
Radix sort
Origin: Herman Holleriths card-sorting
machine for the 1890 U.S. Census.
Digit-by-digit sort.
Holleriths original (bad) idea: sort on
most-significant digit first.
Good idea: Sort on least-significant digit
first with auxiliary stable sort.
L5.31
Operation of radix sort
329 720 720 329

457 355 329 355
657 436 436 436
839 457 839 457
436 657 355 657
720 329 457 720
355 839 657 839
L5.32
Correctness of radix sort
Induction on digit position
720 329
Assume that the numbers
are sorted by their low-order 329 355
t 1 digits. 436 436
839 457
Sort on digit t
355 657
457 720
657 839
L5.33
720 329
t 1 digits. 436 436
839 457
Sort on digit t
Two numbers that differ in 355 657
digit t are correctly sorted. 457 720
657 839
L5.34
720 329
t 1 digits. 436 436
839 457
Sort on digit t
Two numbers that differ in 355 657
digit t are correctly sorted. 457 720
Two numbers equal in digit t 657 839
are put in the same order as
the input correct order.
L5.35
Analysis of radix sort
Assume counting sort is the auxiliary stable sort.
Sort n computer words of b bits each.
Each word can be viewed as having b/r base-2r
digits. 8 8 8 8
Example: 32-bit word
r = 8 b/r = 4 passes of counting sort on
base-28 digits; or r = 16 b/r = 2 passes of
counting sort on base-216 digits.
How many passes should we make?
L5.36
Analysis (continued)
Recall: Counting sort takes (n + k) time to
sort n numbers in the range from 0 to k 1.
If each b-bit word is broken into b/r equal
pieces, each pass of counting sort takes (n +
2r) time. Since there are b/r passes, we have
T (n, b) = b (n + 2 r ) .
r
Choose r to minimize T(n, b):
Increasing r means fewer passes, but as
r >> lg n, the time grows exponentially.
L5.37
Choosing r
T (n, b) = b (n + 2 r )
r
Minimize T(n, b) by differentiating and setting to 0.
Or, just observe that we dont want 2r >> n, and
theres no harm asymptotically in choosing r as
large as possible subject to this constraint.
Choosing r = lg n implies T(n, b) = (b n/lg n) .
For numbers in the range from 0 to n d 1, we
have b = d lg n radix sort runs in (d n) time.
L5.38
Conclusions
In practice, radix sort is fast for large inputs, as
well as simple to code and maintain.
Example (32-bit numbers):
At most 3 passes when sorting 2000 numbers.
Merge sort and quicksort do at least lg 2000 =
11 passes.
Downside: Unlike quicksort, radix sort displays
little locality of reference, and thus a well-tuned
quicksort fares better on modern processors,
which feature steep memory hierarchies.
L5.39

Binder1 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Binder1 PDF

Uploaded by

Copyright:

Available Formats

Introduction: Analysis of

Algorithms, Insertion Sort,

Input: sequence a1, a2, , an of numbers.

Output: permutation a'1, a'2, , a'n such

The running time depends on the input: an

What is insertion sorts worst-case time?

Key subroutine: MERGE

Time = Q(n) to merge a total

T(n/4) T(n/4) T(n/4) T(n/4)

cn/4 cn/4 cn/4 cn/4

Q(n lg n) grows more slowly than Q(n2).

This bound is not tight!

T(n/16) T(n/8) T(n/8) T(n/4)

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

The master method applies to recurrences of

Ex. T(n) = 4T(n/2) + n

Ex. T(n) = 4T(n/2) + n2

Ex. T(n) = 4T(n/2) + n2/lg n

Let p be the unique solution to

Next time: applying the master method.

# subproblems work dividing

# subproblems work dividing

nlogba = nlog21 = n0 = 1 CASE 2 (k = 0)

T(n) = T(n/2) + Q(1) T(n) = Q(lg n) .

Running time = Q(n3)

# submatrices work adding

nlogba = nlog28 = n3 CASE 1 T(n) = Q(n3).

No better than the ordinary algorithm.

T(n) = 7 T(n/2) + Q(n2)

Divide and conquer is just one of several

Initial call: QUICKSORT(A, 1, n)

Assume all input elements are distinct.

Take expectations of both sides.

Independence of Xk from other random

Linearity of expectation; E[Xk] = 1/n .

Quicksort is a great general-purpose

insert(S, x) : insert element x into set S

No pointers required! Height of a binary heap is O(lg n)

max_heapify : correct a single violation of the heap

insert, extract_max, heapsort

Assume that the trees rooted at left(i) and right(i)

If element A[i] violates the max-heap property, correct

Why start at n/2?

Because elements A[n/2 + 1 n] are all leaves of the tree

Time=? O(n log n) via simple analysis

Observe however that Max_Heapify takes O(1) for

Total amount of work in the for loop can be summed as:

This means that Build_Max_Heap is O(n)

1. Build Max Heap from unordered array;

1. Build Max Heap from unordered array;

2. Find maximum element A[1];

3. Swap elements A[n] and A[1]:

1. Build Max Heap from unordered array;

2. Find maximum element A[1];

3. Swap elements A[n] and A[1]:

1. Build Max Heap from unordered array;

2. Find maximum element A[1];

3. Swap elements A[n] and A[1]:

1. Build Max Heap from unordered array;

2. Find maximum element A[1];

3. Swap elements A[n] and A[1]:

Swap A[10] and A[1]