Topic 1: Basic tools for analysis of algorithms Assignment: please read pages: 41-90 (23-76 in the first edition)

Recurrences Recurrences are recursive definitions, i.e. definitions that specify values of a function in terms of values of the same function for smaller arguments, like: (n+1)! = (n+1)n! f(n+1) = f(n) + f(n-1) f(n) = f( n/2) + n

Recurrences naturally arise when estimating the run time of recursive algorithms. For example, recall that the MergeSort has the following pseudo-code:
1

Merge-Sort(A,p,r) 1 if p < r 2 then q (p+r)/2 3 Merge-Sort(A,p,q) 4 Merge-Sort(A,q+1,r) 5 Merge(A,p,q,r) The Merge procedure is running in linear time, i.e., in time k n for some k. Thus, as we have shown, the run time of MergeSort satisfies recurrence formula T(n)=2T(n/2)+ C n To make things a bit simpler we will assume n is a power of two; in the CLRS textbook it is shown that thats OK. Thus we get the recurrence formula: T(n)=2T(n/2)+ C n There are ways to solve some recurrences explicitly, but they all tend to be somewhat tricky.
2

However, in estimating the efficiency of an algorithm we do not need the solution of a recurrence formula, but we only need to find the growth rate of the solution i.e., its asymptotic behavior. The master method provides an estimate of the growth rate of the solution for recurrences of the form T(n) = a T(n/b) + f(n) for certain classes of a, b and overhead functions f(n), without having to solve these recurrences. This is what the Master Theorem is all about.

If T(n) is interpreted as the number of steps needed to execute an algorithm for an input of length n, this recurrence corresponds to a divide and conquer method, in which a problem of size n is divided into a problems of size n/b, where a, b are positive constants: T(n) = a T(n/b) + f(n)
number of sub-problems sub-problem price for dividing the size problem and combining the solutions (overhead)

Clearly, in order to have a recursion at all, we must have n/b < n; i.e., b > 1. Also, since a corresponds to the number of problems of smaller size, we are interested only in cases with a r 1. Function f(n) is the overhead, i.e. the cost of dividing the problem of size n into a subproblems of size n/b and then putting the results of sub-problems together.
4

The Master Theorem: Let a, b be constants such that a r 1 and b >1, let f(n) be a function and let T(n) be defined on natural numbers by the recurrence T(n)=a T(n/b) + f(n) Then T(n) can be bounded asymptotically as follows : f(n)=O(nlogb a-e) T(n)=Q(nlogb a) for some e > 0,

1) If then:

Thus, if f(n) does not grow faster than the polynomial nlogb a-e of degree strictly smaller than logba then the sollution T(n) grows as the polynomial nlogb a.

2) If f(n)=Q(nlogb a) then : T(n)=Q(nlogb a logn) Thus, if f(n) grows exactly as fast as the polynomial nlogb a then the solution T(n) grows exactly as fast as nlogb a log n.

3) If f(n) = W(nlogb a+e) for some e > 0, and if a f(n/b)b c f(n) for some constant c <1 and all n n0 for some sufficiently large n0, then T(n)=Q(f(n)).
Thus, if f(n) grows at least as fast as the polynomial nlogb a+e and if the condition a f(n/b)b c f(n) is met for some constant c <1, and all n n0 for some sufficiently large n0, then the solution T(n) grows asymptotically as fast as f(n).

The trichotomy for the growth rate of the solution T(n) of the recurrence is based on comparing the growth rate of f(n) with the growth rate of the polynomial nlogb a

Informally speaking, i) if f(n) grows substantially slower than the polynomial nlogb a then the solution T(n) grows exactly as nlogb a ; ii) if f(n) grows as fast as the polynomial nlogb a then the sollution T(n) grows exactly as nlogb a logn; iii) finally, if f(n) grows substantially faster than the polynomial nlogb a and the extra condition a f(n/b)b c f(n) is satisfied for some c <1 and all n n0 for some sufficiently large n0, then the solution T(n) grows exactly fast as f(n). iv) In all other cases no conclusion about the growth rate of T(n) can be made, but fortunately in practice this does not happen too often.

Dumb Examples (better ones later): 1. Let T(n) = 9 T(n/3) + n. Then a = 9 and b = 3, so nlog 9 =n2. Since f(n) = n = O(n2-.5) = O(n1.5) the first case applies with e = .5 and we conclude that T(n) = n2.
3

2. If T(n) = T(2/3n) + 1 then a = 1 and b = 3/ 2, so nlog 1 =n0=1. Thus, both f(n)=1 and nlog a =1 which implies f(n)=Q(nlogb a) and so the second case applies; consequently:
3/2 b

T(n)=Q(nlogb a logn)=Q(n0logn) =Q(logn) 3. Consider now T(n) = 3T(n/4)+ n. Then a=3 and b=4 and so nlog a = nlog 3 n0.79. Thus, f(n)= n = W(n.79+.1) W(n0.89). Also, a f(n/b )=3 f(n/4 ) = 3 n/4 =3/4 n=3/4 f(n)=.75n < .8 n = .8 f(n) Consequently by case 3, T(n)=Q(n).
b 4

4. Finally, consider recurrence T(n)=2T(n/2)+n log n In this case a=2, b=2, f(n) = n log n. Thus, nlogb a = nlog2 2 = n. While f(n)= n log n >n, but for no e f(n)=n log n >n1+e for sufficiently large n, so the third case does not apply and thus we cannot use the master theorem. Homework: Use basic calculus to show that for every e there is N such that for all n > N ne > log n.

10

Real examples: Analysis of Merge Sort As we saw, Merge Sort running time is given by the recurrence relation T(n)=2T(n/2)+ C n Thus, for the Master Theorem we have: a=2, b=2, f(n)=Cn Since nlogba = nlog22 = n1 = n we get that the second case holds because f(n)=Cn = (n)=(nlogba) Thus, the solution of the recurrence satisfies T(n)=(nlogbalog n)=(n log n)

11

Real examples: Analysis of the Karatsuba algorithm for integer multiplication The Karatsuba algorithm running time is given by the recurrence relation T(n)=3T(n/2)+ C n Thus, for the Master Theorem we have: a=3, b=2, f(n)=Cn Since nlogba = nlog23 = n1.58 we get that the first case holds, because f(n)=Cn =O(n1.48) =O(n1.58 - .1) = O (nlog23-) Thus, the solution of the recurrence satisfies T(n)=(nlogba)= (nlog23 ) = ( n1.58)
12

Determining the running time of an algorithm seems to be hard because it requires knowing difficult theorems. Why bother? Why not just write a correct program and not worry how fast it is? n nlogn n2 n3 2n

16 64 256 4096 65536

256 2048 65536 16777216 1077

65536 1048576 4294967296 281474976710656 1019728

The above table shows that how fast an algorithm is, often determines in practice if an algorithm will run at all or not because it would take too much time to finish!!! Why bother proving the master theorem and not just take it on faith?

13

For a practical reason : We are interested in techniques used in this proof : we want to acquire skills in how to count the total number of steps of a recursive procedure, how to do sumations of various sums which will often pop up in our analyses of algorithms etc. The methods of the proof will keep appearing on many occasions. Thus, by mastering the proof we will get used to concepts involved and learn the machinery used in estimating the run time of many algorithms. The same machinery can be used to estimate the run time of recursive algorithms which you might be designing yourselves!

14

Proof of the Master Theorem

We will prove the Master Theorem only for numbers which are powers of b. The proof for an arbitrary number follows from the properties of the floor and ceiling functions; recall 2.99 = 2 and 2.01 = 3. It does not add anything interesting from the prospective of algorithms.

Proof : (concentrate on the method, NOT on calculations !!) Step 1.(unwinding recursion): Substitute m in T(m) = aT(m/b) + f(m) by n/bj for all 1 j i where i = logb n (for simplicity we assume n is a power of b. Thus, assume that T(m) = aT(m/b) + f(m) for all m. Then by substituting m in this equality by n, n/b, n/b2 and so on up to n/bi-1, we get : T(n) = aT(n/b) + f(n) T(n/b) = aT(n/b2) + f(n/b) T(n/b2) = aT(n/b3) + f(n/b2)
. . .

T(n/bi-1) = aT(n/bi) + f(n/bi-1)

15

Step 2. Sucessive eliminations of T(n/bk) in order to replace the instances T(n/bk) of the recursively defined function T(n) by the instances f(n/bk) of the explicitely defined, simpler function f(n). By consecutive replacements of T(n/bk) by aT(n/bk+1) + f(n/bk) using equations obtained at Step 1 we get : T(n) aT(n/b) + f(n)= a( aT(n/b2) + f(n/b)) + f(n)= a2T(n/b2) +a f(n/b) + f(n)= a2 (aT(n/b3) + f(n/b2)) + a f(n/b)+ f(n)= a3T(n/b3) + a2 f(n/b2)+a f(n/b)+ f(n)= . . aiT(n/bi)+ai-1 f(n/bi-1)+ai-2 f(n/bi-2)++ f(n) Here we essentially unwind the recursion untill we hit the base case T(n/bi)= T(1), (because b i= n, i.e. i=logbn).
16

Thus, T(n)= aiT(1) + ai-1 f(n/bi-1)+ai-2 f(n/bi-2)++ f(n)=

i summands

a iT (1) + a j f ( bnj ) = a logb nT (1) +

j =0

i 1

(log b n ) 1 j =0

j n a f ( bj )

Step 3: simplifying the coefficient in front of T(1). Since alogb n=nlogb a we get:
T (n) = n log b aT (1) +
(log b n ) 1

a
j =0

f ( bnj )

Thus, to estimate the behavior of T(n) we have to estimate the behavior of the sum
S (n) =
j

log b n 1 j =0

j a f ( bnj )

Notice that in this sum a j f ( bn ) is a product of members of an increasing geometric progression aj and values of f(n/bj) with inputs n/bj that form a decreasing geometric progression. Thus, the behavior of the sum depends on whether aj grows faster than f(n/bj) decreases!! (we are interested only in f(n) that is monotonically increasing, because f(x) will stand for the overhead of divide-and-conquer algorithms. ) This is the crux of Master Theorem!. We now consider the growth rate of f(n) accordingly, i.e., we replace f(n/bj) with its asymptotic estimate.
17

Step 4: Replacing f(n/bj) by asymptotic estimates. We proceed according to our three cases. Case 1: Assume f(m)=O(mlogb a-e). Then using properties of the big-oh, O (which are easy to show) and by substituting m in the above by n/bi we get f(n/bi)=O((n/bi)logb a-e), and so
S ( n) =
log b n 1 j =0

a f ( bnj ) =
j

log b n 1 j =0

j n log b a (( ) )= a O bj

logb a logb n1 a logb n1 j n logb a = O n O log b a a (bj ) b j =0 j =0 logb a logb n1 ab logb a logb n1 ab j = O n O log b a a n b j =0 j =0 logb a logb n1 j O b n j = 0

) =
j

( )

( )

j =

( )

18

By using
q n+1 1 1 + q + q + ... + q = and b logb n = n q 1
2 n
log b n 1

we sum the geometric progression and get

(b )
j =0

log b a b log b n 1 S ( n) = O = n b 1
n logb a n logb a logb a n 1 O = O n 1 b b 1 logb a = O n

Consequently,
T ( n) = n
logb a

T (1) +

logb n 1 j =0

logb a logb a j n a f ( ) = n T ( 1 ) + O n j b

19

However, adding to nlogb a T(1) something that grows not faster than nlogb a results in something that grows as fast as nlogb a, i.e.
T (n) = n logb aT (1) + O n logb a = n logb a

This proves the first case.

20

Case 2: Assume f(n)=(nlogb a). Then as in the first case, we first estimate
S ( n) =
logb n 1 j =0

f( )=
n bj

logb n 1 j =0

a ((
j

n logb a bj

)=
j

logb a logb n1 a logb n1 j n logb a = n log b a a (bj ) b j =0 j =0 logb a logb n1 logb a logb n1 a j (a ) 1 = n = n j =0 j =0 n logb a log b n

( )

Since logbn = logb2 log2n we get

S (n) = n logb a log 2 n

Consequently, denoting log2n by lg n, we get

T ( n) = n
logb a

T (1) +

logb n 1 j =0

j a f ( bnj ) =

n logb aT (1) + n logb a lg n = n logb a lg n

) (
21

Case 3: In the assumption that f(n/b)b c/a f(n) we substitute n by n/bk-1 to get f(n/bk)b c/a f(n/bk-1) . Thus, we get: f(n/bj)b c/a f(n/bj-1) f(n/bj-1)b c/a f(n/bj-2) f(n/bj-2)b c/a f(n/bj-3) ... f(n)b c/a f(n/b) By substitution, chaining these inequalities, we get f(n/bj)b c/a f(n/bj-1)b (c/a)2 f(n/bj-2) b ...b (c/a)j f(n), i.e. we get f(n/bj)b cj/aj f(n)

22

Thus, we can estimate S(n) as follows

S (n) =
log b n 1 j log b n 1 j =0

f ( bnj )

log b n 1

j =0 j

cj a j f ( n) = a
j

f ( n) c f ( n) = f ( n) c f ( n) c = = O( f (n)) 1 c j =0 j =0 j =0
j

log b n 1

Consequently, we get:
T (n) = n log b aT (1) +
log b n 1 j =0 j n a f ( bj ) =

n log b aT (1) + O( f (n))

By assumption in the third case f(n) = W(nlogb a+e) for some e > 0, and so, since f(n) grows much faster than nlogba,
T (n) = n logb aT (1) + O( f (n)) = O( f (n))

This finishes the proof for the third case and thus we are done !

23