Asymptotics and Recurrence

Equations
Prepared by
John Reif, Ph.D.
Analysis of Algorithms
Asymptotics and Recurrence
Equations
a) Computational Complexity of a
Program
b) Worst Case and Expected Bounds
c) Definition of Asymptotic Equations
d) Solution of Recurrence Notation
Readings
Main Reading Selection:
CLR, Chapter 3, 4 and Appendix A
Goal
To estimate and compare growth rates
of functions

Ignore constant factors of growth

f(n) is asymptotically equal to g(n)
1
g(n)
f(n)
lim
if
g(n) ~ f(n)
n
=

f(n) is little o g(n)
f(n) is o(g(n)) if
0
g(n)
f(n)
lim =
n
f(n) is big O g(n)
f(n) is O(g(n)) if
c
g(n)
f(n)
sup lim s
n
Example of f(n) is O(g(n))
0
0
c,n 0
s.t. f(n) c g(n)
for all n n
- >
s
>
f(n) is order at least g(n)
f(n) is O(g(n)) if
c
g(n)
f(n)
inf lim >
n
0
0
n n all for
g(n) c f(n) s.t. 0 c , n
>
> > -
Example of f(n) is O(g(n))
f(n) is order tight with g(n)
f(n) is u(g(n)) if
g(n) c' f(n) g(n) c s.t. c' c, , n
0
s s -
Suppose my algorithm runs in time
O(n)
Dont say:
his runs in time O(n
2
) so is worse
But prove:
his runs in time O(n
2
) so is worse

Must find a worst case input of length
n for which his algorithm takes time >
cn
2
for all n > n
0

Use of O Notation
N is O (n
2
)
sometimes written
n = O (n
2
)

But n
2
is not O(n) so cant use
identities!

The two sides of the equality do not
play a symmetric role
Use of Asymptotic Notation
Write

As

Example
o(h(n))" is g(n) f(n) "
o(h(n)) g(n) f(n) + =
+ =
|
.
|

\
|
+ + =
|
.
|

\
|
+ + =
n as ) 1 ( o 1
n
1
o
n
1
1
n
1
O
n
1
1
1 - n
n
2
Convergent Power Sum
A polynomial is asymptotically equal to
its leading term as x
1 0 for , ) 1 (
1
1
0
< < s

=
x O
x
x
i
i
( )
( )
d
d
d
0 i
i
i
1 d
d
0 i
i
i
d
d
0 i
i
i
x a ~ x a
x o x a
x x a

=
+
=
=
=
=
Sums of Powers
for n


Or equivalently
1 d
n
1 i
d
n
1 d
1
~ i
+
=
+

( )
1 d 1 d
n
1 i
d
n o n
1 d
1
i
+ +
=
+
+
=

Examples
2
nd
order asymptotic expansion

\
|

=
=

3
n
~ i
2
n
~ i
3
2
n
1 i
2
n
1 i
( )
1 d d 1 d
n
1 i
d
n o n
2
1
n
1 d
1
i
+
=
+ +
+
=

Asymptotic Expansion of f(n) as

n n
0
If


and
(n) g c ~ f(n)
i
1 i
i

=
1 i all for
(n)) o(g (n) (1)g
i 1 i
>
=
+
1 k all for
(n)) o(g (n) g c f(n) (2)
k i
k
1 i
i
>
+ =

=
Bounding Sums by Integrals

}
= =
+
+ s s
n
1 k
n
1 k
1 n
1
1) (k f dx f(x) f(k)
Bounding Sums by Integrals (contd)

So


Example then


}
= =
+
+ s s
n
1 k
n
1 k
1 n
1
1) (k f dx f(x) f(k)
}

}
+
=
+
s s + +
1 n
1
n
1 k
1 n
1
dx f(x) f(k) f(1) 1) f(n - dx f(x)
}
= x xln(x) (x)dx ln
ln(x) f(x) if =
Bounding Sums by Integrals (contd)

So


Since


So

=
+ + + =
n
1 k
(ln(n)) n - 1) (n ln 1) (n k ln

2 ln
n ln
log(n) =

=
+ + + =
n
1 k
n) (log
2 ln
n
- 1) (n log 1) (n k log
Other Approximations Derived from
Integrals
Harmonic Numbers

=
+
+
+
+
=
n
1 k
2 2
n) log (n
2 ln 4
1) (n
- 1) (n log
2
1) (n
k log k

=
=
n
1 k
n
k
1
H

\
|
=
|
.
|

\
|
+ + =
.577... constant s Euler'
n
1
O ln(n) H
n
Stirlings Approximation for Factorial
Factorial n! = 1 2 3 (n-1) n




So
n as e n n 2 ~ n!
-n n
t
(n) - n log n
(1) n) (2 log
2
1
e log n - n log n ) log(n!
=
+ + = t
Recurrence Equations
(over integers)
Homogenous of degree d

n > d



Given constant coefficients a
1
, , a
d
initial values x
1
, x
2
,

, x
d

d n d 2 n 2 1 n 1 n
x a ... x a x a x

+ + + =
Example: Fibonacci Sequence
n > 2

F
n
= F
n-1
+ F
n-2


F
0
= 0, F
1
=1


Solution of Fibonacci Sequence
golden ratio







where



n
2 2
n
1 1 n
2
1
r c r c F
) 5 1 (
2
1
r
1.618... ) 5 1 (
2
1
r
+ =
=
= + =
1 r c r c F
0 c c F
2 2 1 1 1
2 1 0
= + =
= + =
Solution of Fibonacci Sequence
(contd)
Hence
n
n
n n
n
2
5 1

5
1
~ F
2
5 1
2
5 1
5
1
F
|
|
.
|

\
|
+

(
(

|
|
.
|

\
|

|
|
.
|

\
|
+
=
A Useful Theorem
c > 0, d > 0


If




then
0
d
c n=1
T (n) =
n
aT + cn n>1
b

| |
|

\ .

( )
( )
( )
b
log a d
d d
b
d d
n a > b
T (n) = n log n a = b
n a < b

Proof

Is solution
b
log n d
T (n) = cn g(n) + a d
b
2 log n-1
d d d
a a a
g(n) = 1 + + + ... +
b b b
| | | |
| |
\ . \ .
Cases
( ) ( )
b
b b
log n-1
d
d
log n log a
a
(1) a > b g(n) ~
b
is last term so
T(n) = a d = n
| |

|
\ .
( )
d
b
d
b
(2) a = b g(n) = log n
so T(n) = n log n

( )
d
d
(3) a < b g(n) upper bound by 0(1)
so T(n) = n

Example: Mergesort
input list L of length N
if N=1 then return L
else do
let L
1
be the first
elements of L

let L
2
be the last elements
of L

M
1
Mergesort (L
1
)
M
2
Mergesort (L
2
)

return Merge (M
1
, M
2
)


N
2
(
N
2
Time Bounds of Mergesort
Initial Value T(1) = c
1


for N > 1

for some constants c
1
, c
2
> 1
2
T(N) T T c N
(

| |
| |
N N
s + +
|
|
|
|
2 2
\ .
\ .
Time Bound (contd)
N > 1


guess




Holds if

Solution



2
T(N) 2T c N
N
| |
s +
|
2
\ .
2
T(N) a N log N + b
N N
2 a log + b c N
2 2
s
| |
| |
s +
| |
\ .
\ .
1 2 1
a = c + c , b = c
( )
1 2 1
T(N) c + c N log N + c s
Time Bound (contd)
N > 1

Transform Variables




Recurrence equation:







2 1
T(N) 2T c N, T(1) = c
N
| |
s +
|
2
\ .
n
n = log N, N = 2
N
n 1 log N log2 log
2
| |
= =
|
\ .
( )
( )
n n
n n-1 2
0
0 1
X T 2 X + c
X T = T(1) = c
= 2 = 2
= 2
Solve by usual methods for recurrence
equations
( )
( )
n
n
X O n
so T N = O (N log N)
= 2
Advanced Material
Exact Solution of Recurrence
Relations
Homogenous Recurrence Relations
(no constant additive term)
Solve:

try

Multiply by


Get characteristic equation:


d n d 2 n 2 1 n 1 n
x a ... x a x a x

+ + + =
n
n
r x =
n
d
r
r
0 a ... r a r a r
d
2 - d
2
1 - d
1
d
=
Case of Distinct Roots
Distinct Roots
r
1
, r
2
, , r
d







Where r
i
is dominant root
n
i i n
n
i
d
1 i
i n
r c ~ x
r c x

=
=
i j r r
j i
= >
Other Case
Roots are not distinct
r
1
= r
2
= r
3


Then solutions not independent,
so additional terms:
n
i
d
4 i
i
n
1
2
3
n
1 2
n
1 1 n
r c r n c r n c r c x

=
+ + + =
Inhomogenous Recurrence
Equations

Nonzero constant term

Solution Method

(1) Solve homogenous equation
0
a 0 =
n 1 n 1 2 n 2 n n d
Y a Y a Y ... a Y

= + + +
n 1 n 1 2 n 2 d n d 0
x a x a x ... a x a

= + + + +
Solution Method
1) Solve homogenous equation



2) Case

, add particular solution




n 1 n 1 2 n 2 n n d
Y a Y a Y ... a Y

= + + +
i
a 1 =

Solution Method (contd)

Case

, add particular solution





3) Add particular and homogeneous
solutions, and solve for constants

This is all we usually need!!



i
a 1 =

0
n
i
a
x cn= n
ia
| |
=
|
|
\ .

Asymptotics and Recurrence

Equations
Prepared by
John Reif, Ph.D.
Analysis of Algorithms