Introduction To Algorithms: Asymptotic Notation Recurrences

Introduction to Algorithms
6.046J/18.401J
LECTURE 2
Asymptotic Notation
O-, -, and -notation
Recurrences
Substitution method
Iterating the recurrence
Recursion tree
Master method
Prof. Erik Demaine
September 12, 2005
Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson
L2.1
Asymptotic notation
O-notation (upper bounds):
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
September 12, 2005
L2.2
Asymptotic notation
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
EXAMPLE: 2n2 = O(n3)
September 12, 2005
(c = 1, n0 = 2)
L2.3
Asymptotic notation
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
(c = 1, n0 = 2)
functions,
not values
September 12, 2005
L2.4
Asymptotic notation
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
functions,
not values
September 12, 2005
(c = 1, n0 = 2)
funny, one-way
equality
L2.5
Set definition of O-notation

O(g(n))
O(g(n)) == {{ f(n)
f(n) :: there
there exist
exist constants
constants
cc >> 0,
such
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n)
for
for all
all nn nn00 }}
September 12, 2005
L2.6

O(g(n))
O(g(n)) == {{ f(n)
f(n) :: there
there exist
exist constants
constants
cc >> 0,
such
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n)
for
for all
all nn nn00 }}
EXAMPLE: 2n2 O(n3)
September 12, 2005
L2.7

O(g(n))
O(g(n)) == {{ f(n)
f(n) :: there
there exist
exist constants
constants
cc >> 0,
such
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n)
for
for all
all nn nn00 }}
EXAMPLE: 2n2 O(n3)
(Logicians: n.2n2 O(n.n3), but its
convenient to be sloppy, as long as we
understand whats really going on.)
September 12, 2005
L2.8
Macro substitution
Convention: A set in a formula represents
an anonymous function in the set.
September 12, 2005
L2.9
Macro substitution
EXAMPLE:
September 12, 2005
f(n) = n3 + O(n2)
means
f(n) = n3 + h(n)
for some h(n) O(n2) .
L2.10
Macro substitution
EXAMPLE:
September 12, 2005
n2 + O(n) = O(n2)
means
for any f(n) O(n):
n2 + f(n) = h(n)
for some h(n) O(n2) .
L2.11
-notation (lower bounds)

O-notation is an upper-bound notation. It
makes no sense to say f(n) is at least O(n2).
September 12, 2005
L2.12

(g(n))
(g(n)) == {{ f(n)
f(n) :: there
there exist
exist constants
constants
cc >> 0,
such
0, nn00 >> 00 such
that
that 00 cg(n)
cg(n) f(n)
f(n)
for
for all
all nn nn00 }}
September 12, 2005
L2.13

(g(n))
(g(n)) == {{ f(n)
f(n) :: there
there exist
exist constants
constants
cc >> 0,
such
0, nn00 >> 00 such
that
that 00 cg(n)
cg(n) f(n)
f(n)
for
for all
all nn nn00 }}
EXAMPLE:
September 12, 2005
n = (lg n) (c = 1, n0 = 16)
L2.14
-notation (tight bounds)

(g(n))
(g(n)) == O(g(n))
O(g(n))
(g(n))
(g(n))
September 12, 2005
L2.15
-notation (tight bounds)

(g(n))
(g(n)) == O(g(n))
O(g(n))
(g(n))
(g(n))
EXAMPLE:
September 12, 2005
1
2
n 2 n = ( n )
2
L2.16
-notation and -notation

O-notation and -notation are like and .
o-notation and -notation are like < and >.
(g(n))
(g(n)) == {{ f(n)
f(n) :: for
for any
any constant
constant cc >> 0,
0,
there
there is
is aa constant
constant nn00 >> 00
such
such that
that 00 f(n)
f(n) << cg(n)
cg(n)
for
for all
all nn nn00 }}
EXAMPLE: 2n2 = o(n3)

September 12, 2005
(n0 = 2/c)
L2.17
-notation and -notation

O-notation and -notation are like and .
o-notation and -notation are like < and >.
(g(n))
(g(n)) == {{ f(n)
f(n) :: for
for any
any constant
constant cc >> 0,
0,
there
there is
is aa constant
constant nn00 >> 00
such
such that
that 00 cg(n)
cg(n) << f(n)
f(n)
for
for all
all nn nn00 }}
EXAMPLE:
September 12, 2005
n = (lg n)
(n0 = 1+1/c)
L2.18
Solving recurrences
The analysis of merge sort from Lecture 1
required us to solve a recurrence.
Recurrences are like solving integrals,
differential equations, etc.
o Learn a few tricks.
Lecture 3: Applications of recurrences to
divide-and-conquer algorithms.
September 12, 2005
L2.19
Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
September 12, 2005
L2.20
Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
EXAMPLE: T(n) = 4T(n/2) + n
[Assume that T(1) = (1).]
Guess O(n3) . (Prove O and separately.)
Assume that T(k) ck3 for k < n .
Prove T(n) cn3 by induction.
September 12, 2005
L2.21
Example of substitution
T (n) = 4T (n / 2) + n
4c ( n / 2 ) 3 + n
= ( c / 2) n 3 + n
desired residual
= cn3 ((c / 2)n3 n)
cn3 desired
whenever (c/2)n3 n 0, for example,
if c 2 and n 1.
residual
September 12, 2005
L2.22
Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have (1) cn3, if we
pick c big enough.
September 12, 2005
L2.23
Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have (1) cn3, if we
pick c big enough.
This bound is not tight!
September 12, 2005
L2.24
A tighter upper bound?

We shall prove that T(n) = O(n2).
September 12, 2005
L2.25

Assume that T(k) ck2 for k < n:
T (n) = 4T (n / 2) + n
4 c ( n / 2) 2 + n
= cn 2 + n
2
= O(n )
September 12, 2005
L2.26

T (n) = 4T (n / 2) + n
4c ( n / 2) 2 + n
= cn 2 + n
2
= O(n ) Wrong! We must prove the I.H.
September 12, 2005
L2.27

T (n) = 4T (n / 2) + n
4c ( n / 2) 2 + n
= cn 2 + n
2
= O(n ) Wrong! We must prove the I.H.
= cn 2 ( n) [ desired residual ]
cn 2 for no choice of c > 0. Lose!
September 12, 2005
L2.28
A tighter upper bound!

IDEA: Strengthen the inductive hypothesis.
Subtract a low-order term.
Inductive hypothesis: T(k) c1k2 c2k for k < n.
September 12, 2005
L2.29

T(n) = 4T(n/2) + n
= 4(c1(n/2)2 c2(n/2)) + n
= c1n2 2c2n + n
= c1n2 c2n (c2n n)
c1n2 c2n if c2 1.
September 12, 2005
L2.30

T(n) = 4T(n/2) + n
= 4(c1(n/2)2 c2(n/2)) + n
= c1n2 2c2n + n
= c1n2 c2n (c2n n)
c1n2 c2n if c2 1.
Pick c1 big enough to handle the initial conditions.

September 12, 2005
L2.31
Recursion-tree method
A recursion tree models the costs (time) of a
recursive execution of an algorithm.
The recursion-tree method can be unreliable,
just like any method that uses ellipses ().
The recursion-tree method promotes intuition,
however.
The recursion tree method is good for
generating guesses for the substitution method.
September 12, 2005
L2.32
Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
September 12, 2005
L2.33

Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)
September 12, 2005
L2.34

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4)
September 12, 2005
T(n/2)
L2.35

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
T(n/16)
September 12, 2005
T(n/8)
(n/2)2
T(n/8)
T(n/4)
L2.36

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/8)2
(n/4)2
(n/16)2
(n/2)2
(1)
September 12, 2005
L2.37

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/2)2
(n/8)2
(n/4)2
(n/16)2
n2
(1)
September 12, 2005
L2.38

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/8)2
5 n2
16
(n/4)2
(n/16)2
(n/2)2
n2
(1)
September 12, 2005
L2.39

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/4)2
5 n2
16
25 n 2
256
(n/8)2
(n/16)2
(n/2)2
n2
(1)
September 12, 2005
L2.40

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/2)2
(n/8)2
(n/8)2
(1)
September 12, 2005
(n/4)2
( ) +( )
2
5
5
1 + 16 + 16
Total = n
= (n2)
5 n2
16
25 n 2
256
(n/4)2
(n/16)2
n2
5 3
16
+L
geometric series
L2.41
The master method

The master method applies to recurrences of
the form
T(n) = a T(n/b) + f (n) ,
where a 1, b > 1, and f is asymptotically
positive.
September 12, 2005
L2.42
Three common cases

Compare f (n) with nlogba:
1. f (n) = O(nlogba ) for some constant > 0.
f (n) grows polynomially slower than nlogba
(by an n factor).
Solution: T(n) = (nlogba) .
September 12, 2005
L2.43
Three common cases

1. f (n) = O(nlogba ) for some constant > 0.
f (n) grows polynomially slower than nlogba
(by an n factor).
Solution: T(n) = (nlogba) .
2. f (n) = (nlogba lgkn) for some constant k 0.
f (n) and nlogba grow at similar rates.
Solution: T(n) = (nlogba lgk+1n) .
September 12, 2005
L2.44
Three common cases (cont.)

3. f (n) = (nlogba + ) for some constant > 0.
f (n) grows polynomially faster than nlogba (by
an n factor),
and f (n) satisfies the regularity condition that
a f (n/b) c f (n) for some constant c < 1.
Solution: T(n) = ( f (n)) .
September 12, 2005
L2.45
Examples
EX. T(n) = 4T(n/2) + n
a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 ) for = 1.
T(n) = (n2).
September 12, 2005
L2.46
Examples
EX. T(n) = 4T(n/2) + n
a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 ) for = 1.
T(n) = (n2).
EX. T(n) = 4T(n/2) + n2
a = 4, b = 2 nlogba = n2; f (n) = n2.
CASE 2: f (n) = (n2lg0n), that is, k = 0.
T(n) = (n2lg n).
September 12, 2005
L2.47
Examples
EX. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = (n2 + ) for = 1
and 4(n/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = (n3).
September 12, 2005
L2.48
Examples
EX. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = (n2 + ) for = 1
and 4(n/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = (n3).
EX. T(n) = 4T(n/2) + n2/lg n
a = 4, b = 2 nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant > 0, we have n = (lg n).
September 12, 2005
L2.49
Idea of master theorem

Recursion tree:
f (n)
a
f (n/b) f (n/b) f (n/b)
a
f (n/b2) f (n/b2) f (n/b2)
(1)
September 12, 2005
L2.50

Recursion tree:
f (n)
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
f (n/b2) f (n/b2) f (n/b2)
f (n)
(1)
September 12, 2005
L2.51

Recursion tree:
f (n)
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
f (n)
(1)
September 12, 2005
L2.52

f (n)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
(1)
September 12, 2005
#leaves = ah
= alogbn
= nlogba
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
L2.53

f (n)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 1:
1: The
The weight
weight increases
increases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
September 12, 2005
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
(nlogba)
L2.54

Recursion tree:
f (n)
f (n)
(1)
September 12, 2005
C
CASE
ASE 2:
2: (k
(k == 0)
0) The
The weight
weight
isis approximately
approximately the
the same
same on
on
each
each of
of the
the log
logbbnn levels.
levels.
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
nlogba (1)
(nlogbalg n)
L2.55

f (n)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 3:
3: The
The weight
weight decreases
decreases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
September 12, 2005
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
( f (n))
L2.56
Appendix: geometric series

n +1
x
1
for x 1
1 + x + x2 + L + xn =
1 x
1
1+ x + x +L =
for |x| < 1
1 x
2
Return to last
slide viewed.
September 12, 2005
L2.57

Introduction To Algorithms: Asymptotic Notation Recurrences

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Algorithms: Asymptotic Notation Recurrences

Uploaded by

Copyright:

Available Formats

Introduction to Algorithms

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Set definition of O-notation

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Set definition of O-notation

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Set definition of O-notation

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

-notation (lower bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation (lower bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation (lower bounds)

-notation (tight bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation (tight bounds)

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation and -notation

EXAMPLE: 2n2 = o(n3)

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

-notation and -notation

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound?

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound!

September 12, 2005

Copyright 2001-5 Erik D. Demaine and Charles E. Leiserson

A tighter upper bound!