Introduction To Algorithms: Asymptotic Notation Recurrences

Introduction to Algorithms
6.046J/18.401J LECTURE 2
Asymptotic Notation O-, -, and -notation Recurrences Substitution method Iterating the recurrence Recursion tree Master method Prof. Erik Demaine
September 12, 2005 Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson L2.1
Asymptotic notation
O-notation (upper bounds): We We write write ff((n n)) = =O O((g g((n n)) )) if if there there exist >0 0 such such exist constants constants c c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for for all all n n n n00..
September 12, 2005
Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson
L2.2
Asymptotic notation
O-notation (upper bounds): We We write write ff((n n)) = =O O((g g((n n)) )) if if there there exist >0 0 such such exist constants constants c c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for for all all n n n n00.. EXAMPLE: 2n2 = O(n3) (c = 1, n0 = 2)
September 12, 2005
L2.3
Asymptotic notation
O-notation (upper bounds): We We write write ff((n n)) = =O O((g g((n n)) )) if if there there exist >0 0 such such exist constants constants c c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for for all all n n n n00.. EXAMPLE: 2n2 = O(n3)
functions, not values
(c = 1, n0 = 2)
Asymptotic notation
O-notation (upper bounds): We We write write ff((n n)) = =O O((g g((n n)) )) if if there there exist >0 0 such such exist constants constants c c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for for all all n n n n00.. EXAMPLE: 2n2 = O(n3)
functions, not values
September 12, 2005
(c = 1, n0 = 2)
funny, one-way equality
L2.5
Set definition of O-notation

O O((g g((n n)) )) = ={ { ff((n n)) :: there there exist exist constants constants c >0 0 such such c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for } for all all n n n n00 }
September 12, 2005
L2.6

O O((g g((n n)) )) = ={ { ff((n n)) :: there there exist exist constants constants c >0 0 such such c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for } for all all n n n n00 } EXAMPLE: 2n2 O(n3)
September 12, 2005
L2.7

O O((g g((n n)) )) = ={ { ff((n n)) :: there there exist exist constants constants c >0 0 such such c> > 0, 0, n n00 > that that 0 0 ff((n n)) cg cg((n n)) for } for all all n n n n00 } EXAMPLE: 2n2 O(n3) (Logicians: n.2n2 O(n.n3), but its convenient to be sloppy, as long as we understand whats really going on.)
Macro substitution
Convention: A set in a formula represents an anonymous function in the set.
September 12, 2005
L2.9
Macro substitution
Convention: A set in a formula represents an anonymous function in the set. EXAMPLE: f(n) = n3 + O(n2) means f(n) = n3 + h(n) for some h(n) O(n2) .
September 12, 2005
L2.10
Macro substitution
Convention: A set in a formula represents an anonymous function in the set. EXAMPLE: n2 + O(n) = O(n2) means for any f(n) O(n): n2 + f(n) = h(n) for some h(n) O(n2) .
Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson L2.11
September 12, 2005
-notation (lower bounds)

O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).
September 12, 2005
L2.12

O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2). ( (g g((n n)) )) = ={ { ff((n n)) :: there there exist exist constants constants c >0 0 such such c> > 0, 0, n n00 > that that 0 0 cg cg((n n)) ff((n n)) for } for all all n n n n00 }
September 12, 2005
L2.13

O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2). ( (g g((n n)) )) = ={ { ff((n n)) :: there there exist exist constants constants c >0 0 such such c> > 0, 0, n n00 > that that 0 0 cg cg((n n)) ff((n n)) for } for all all n n n n00 } EXAMPLE:
September 12, 2005
n = (lg n) (c = 1, n0 = 16)
Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson L2.14
-notation (tight bounds)

( (g g((n n)) )) = = O( O(g g((n n)) )) ((g g((n n)) ))
September 12, 2005
L2.15
-notation (tight bounds)

( (g g((n n)) )) = = O( O(g g((n n)) )) ((g g((n n)) )) EXAMPLE:
1 2
n 2 n = ( n )
2 2
September 12, 2005
L2.16
-notation and -notation

O-notation and -notation are like and . o-notation and -notation are like < and >.
((g g((n n)) )) = ={ { ff((n n)) :: for for any any constant constant c c> > 0, 0,
there >0 0 there is is a a constant constant n n00 > such such that that 0 0 ff((n n)) < < cg cg((n n)) for } for all all n n n n00 } (n0 = 2/c)
L2.17
EXAMPLE: 2n2 = o(n3)

September 12, 2005
-notation and -notation

O-notation and -notation are like and . o-notation and -notation are like < and >.
((g g((n n)) )) = ={ { ff((n n)) :: for for any any constant constant c c> > 0, 0,
there >0 0 there is is a a constant constant n n00 > such such that that 0 0 cg cg((n n)) < < ff((n n)) for } for all all n n n n00 } (n0 = 1+1/c)
L2.18
EXAMPLE:
September 12, 2005
n = (lg n)
Solving recurrences
The analysis of merge sort from Lecture 1 required us to solve a recurrence. Recurrences are like solving integrals, differential equations, etc. o Learn a few tricks. Lecture 3: Applications of recurrences to divide-and-conquer algorithms.
September 12, 2005
L2.19
Substitution method
The most general method: 1. Guess the form of the solution. 2. Verify by induction. 3. Solve for constants.
September 12, 2005
L2.20
Substitution method
The most general method: 1. Guess the form of the solution. 2. Verify by induction. 3. Solve for constants. EXAMPLE: T(n) = 4T(n/2) + n [Assume that T(1) = (1).] Guess O(n3) . (Prove O and separately.) Assume that T(k) ck3 for k < n . Prove T(n) cn3 by induction.
Example of substitution
T (n) = 4T (n / 2) + n 4c ( n / 2 ) 3 + n = ( c / 2) n 3 + n desired residual = cn3 ((c / 2)n3 n) cn3 desired whenever (c/2)n3 n 0, for example, if c 2 and n 1. residual
Example (continued)
We must also handle the initial conditions, that is, ground the induction with base cases. Base: T(n) = (1) for all n < n0, where n0 is a suitable constant. For 1 n < n0, we have (1) cn3, if we pick c big enough.
September 12, 2005
L2.23
Example (continued)
We must also handle the initial conditions, that is, ground the induction with base cases. Base: T(n) = (1) for all n < n0, where n0 is a suitable constant. For 1 n < n0, we have (1) cn3, if we pick c big enough.
This bound is not tight!
A tighter upper bound?

We shall prove that T(n) = O(n2).
September 12, 2005
L2.25

We shall prove that T(n) = O(n2). Assume that T(k) ck2 for k < n: T (n) = 4T (n / 2) + n 4 c ( n / 2) 2 + n = cn 2 + n = O(n 2 )
September 12, 2005
L2.26

We shall prove that T(n) = O(n2). Assume that T(k) ck2 for k < n: T (n) = 4T (n / 2) + n 4c ( n / 2) 2 + n = cn 2 + n = O(n 2 ) Wrong! We must prove the I.H.
September 12, 2005
L2.27

We shall prove that T(n) = O(n2). Assume that T(k) ck2 for k < n: T (n) = 4T (n / 2) + n 4c ( n / 2) 2 + n = cn 2 + n = O(n 2 ) Wrong! We must prove the I.H. = cn 2 ( n) [ desired residual ] cn 2 for no choice of c > 0. Lose!
A tighter upper bound!

IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c1k2 c2k for k < n.
September 12, 2005
L2.29

IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c1k2 c2k for k < n. T(n) = 4T(n/2) + n = 4(c1(n/2)2 c2(n/2)) + n = c1n2 2c2n + n = c1n2 c2n (c2n n) c1n2 c2n if c2 1.

IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c1k2 c2k for k < n. T(n) = 4T(n/2) + n = 4(c1(n/2)2 c2(n/2)) + n = c1n2 2c2n + n = c1n2 c2n (c2n n) c1n2 c2n if c2 1.
Pick c1 big enough to handle the initial conditions.

Recursion-tree method
A recursion tree models the costs (time) of a recursive execution of an algorithm. The recursion-tree method can be unreliable, just like any method that uses ellipses (). The recursion-tree method promotes intuition, however. The recursion tree method is good for generating guesses for the substitution method.
Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
September 12, 2005
L2.33

Solve T(n) = T(n/4) + T(n/2) + n2: T(n)
September 12, 2005
L2.34

Solve T(n) = T(n/4) + T(n/2) + n2: n2 T(n/4) T(n/2)
September 12, 2005
L2.35

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 T(n/16) T(n/8) (n/2)2 T(n/8) T(n/4)
September 12, 2005
L2.36

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2 (1)
(n/2)2 (n/8)2 (n/4)2
(n/8)2

n2
(n/2)2 (n/8)2 (n/4)2
(n/8)2

n2
5 n2 16
(n/2)2 (n/8)2 (n/4)2
(n/8)2

n2
5 n2 16 25 n 2 256
(n/2)2 (n/8)2 (n/4)2
(n/8)2

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2
n2
5 n2 16 25 n 2 256
5 3 16
(n/2)2 (n/8)2 (n/4)2
(n/8)2
(1)
September 12, 2005
Total = n = (n2)
2 5 5 1 + 16 + 16
+L geometric series
L2.41
( ) +( )
The master method

The master method applies to recurrences of the form T(n) = a T(n/b) + f (n) , where a 1, b > 1, and f is asymptotically positive.
September 12, 2005
L2.42
Three common cases

Compare f (n) with nlogba: 1. f (n) = O(nlogba ) for some constant > 0. f (n) grows polynomially slower than nlogba (by an n factor). Solution: T(n) = (nlogba) .
September 12, 2005
L2.43
Three common cases

Compare f (n) with nlogba: 1. f (n) = O(nlogba ) for some constant > 0. f (n) grows polynomially slower than nlogba (by an n factor). Solution: T(n) = (nlogba) . 2. f (n) = (nlogba lgkn) for some constant k 0. f (n) and nlogba grow at similar rates. Solution: T(n) = (nlogba lgk+1n) .
Three common cases (cont.)

Compare f (n) with nlogba: 3. f (n) = (nlogba + ) for some constant > 0. f (n) grows polynomially faster than nlogba (by an n factor), and f (n) satisfies the regularity condition that a f (n/b) c f (n) for some constant c < 1. Solution: T(n) = ( f (n)) .
September 12, 2005
L2.45
Examples
EX. T(n) = 4T(n/2) + n a = 4, b = 2 nlogba = n2; f (n) = n. CASE 1: f (n) = O(n2 ) for = 1. T(n) = (n2).
September 12, 2005
L2.46
Examples
EX. T(n) = 4T(n/2) + n a = 4, b = 2 nlogba = n2; f (n) = n. CASE 1: f (n) = O(n2 ) for = 1. T(n) = (n2). EX. T(n) = 4T(n/2) + n2 a = 4, b = 2 nlogba = n2; f (n) = n2. CASE 2: f (n) = (n2lg0n), that is, k = 0. T(n) = (n2lg n).
Examples
EX. T(n) = 4T(n/2) + n3 a = 4, b = 2 nlogba = n2; f (n) = n3. CASE 3: f (n) = (n2 + ) for = 1 and 4(n/2)3 cn3 (reg. cond.) for c = 1/2. T(n) = (n3).
September 12, 2005
L2.48
Examples
EX. T(n) = 4T(n/2) + n3 a = 4, b = 2 nlogba = n2; f (n) = n3. CASE 3: f (n) = (n2 + ) for = 1 and 4(n/2)3 cn3 (reg. cond.) for c = 1/2. T(n) = (n3). EX. T(n) = 4T(n/2) + n2/lg n a = 4, b = 2 nlogba = n2; f (n) = n2/lg n. Master method does not apply. In particular, for every constant > 0, we have n = (lg n).
Idea of master theorem

Recursion tree:
a f (n/b) f (n/b) f (n/b) a f (n/b2) f (n/b2) f (n/b2)
f (n)
(1)

Recursion tree:
a f (n/b) f (n/b) f (n/b) a f (n/b2) f (n/b2) f (n/b2)
f (n)
f (n) a f (n/b) a2 f (n/b2)
(1)

Recursion tree:
a f (n/b) f (n/b) f (n/b) a h = logbn f (n/b2) f (n/b2) f (n/b2)
f (n)
f (n) a f (n/b) a2 f (n/b2)
(1)

Recursion tree:
f (n)
f (n) a f (n/b) a2 f (n/b2)
(1)
September 12, 2005
#leaves = ah = alogbn = nlogba

nlogba (1)
L2.53

Recursion tree:

C ASE :: The C ASE 1 1 The weight weight increases increases geometrically geometrically from from the the root root to to the the (1) leaves. leaves. The The leaves leaves hold hold a a constant constant fraction fraction of of the the total total weight. weight.
September 12, 2005 Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson
f (n)
f (n) a f (n/b) a2 f (n/b2)
nlogba (1) (nlogba)

L2.54

Recursion tree:
f (n)
f (n) a f (n/b) a2 f (n/b2)
(1)
September 12, 2005
C ASE :: (( k C ASE 2 2 k= = 0) 0) The The weight weight is is approximately approximately the the same same on on each n each of of the the log log n levels. levels. bb
nlogba (1) (nlogbalg n)

L2.55

Recursion tree:

C ASE :: The C ASE 3 3 The weight weight decreases decreases geometrically geometrically from from the the root root to to the the (1) leaves. leaves. The The root root holds holds a a constant constant fraction fraction of of the the total total weight. weight.
September 12, 2005 Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson
f (n)
f (n) a f (n/b) a2 f (n/b2)
nlogba (1) ( f (n))

L2.56
Appendix: geometric series

n +1 x 1 2 n for x 1 1+ x + x +L+ x = 1 x
1 1+ x + x +L = for |x| < 1 1 x

2
Return to last slide viewed.
September 12, 2005
L2.57

Introduction To Algorithms: Asymptotic Notation Recurrences

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Algorithms: Asymptotic Notation Recurrences

Uploaded by

Copyright:

Available Formats

Introduction to Algorithms

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Set definition of O-notation

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Set definition of O-notation

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Set definition of O-notation

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

-notation (lower bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation (lower bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation (lower bounds)

-notation (tight bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation (tight bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation and -notation

EXAMPLE: 2n2 = o(n3)

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation and -notation

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

A tighter upper bound!

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound!

A tighter upper bound!

Pick c1 big enough to handle the initial conditions.

Example of recursion tree

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Example of recursion tree

September 12, 2005