6 June 2014 Comp 122, Spring 2004

Asymptotic Notation,
Review of Functions &
Summations
asymp - 1 Comp 122
Asymptotic Complexity
Running time of an algorithm as a function of
input size n for large n.
Expressed using only the highest-order term in
the expression for the exact running time.
Instead of exact running time, say O(n
2
).
Describes behavior of function in the limit.
Written using Asymptotic Notation.


asymp - 2 Comp 122
Asymptotic Notation
O, O, O, o, e
Defined for functions over the natural numbers.
Ex: f(n) = O(n
2
).
Describes how f(n) grows in comparison to n
2
.
Define a set of functions; in practice used to compare
two function sizes.
The notations describe different rate-of-growth
relations between the defining function and the
defined set of functions.
asymp - 3 Comp 122
O-notation
O(g(n)) = {f(n) :
- positive constants c
1
, c
2
, and n
0,

such that n > n
0
,
we have 0 s c
1
g(n) s f(n) s c
2
g(n)
}
For function g(n), we define O(g(n)),
big-Theta of n, as the set:
g(n) is an asymptotically tight bound for f(n).
I ntuitively: Set of all functions that
have the same rate of growth as g(n).
asymp - 4 Comp 122
O-notation
O(g(n)) = {f(n) :
- positive constants c
1
, c
2
, and n
0,

such that n > n
0
,
we have 0 s c
1
g(n) s f(n) s c
2
g(n)
}
For function g(n), we define O(g(n)),
big-Theta of n, as the set:
Technically, f(n) e O(g(n)).
Older usage, f(n) = O(g(n)).
Ill accept either
f(n) and g(n) are nonnegative, for large n.
asymp - 5 Comp 122
Example
10n
2
- 3n = O(n
2
)
What constants for n
0
, c
1
, and c
2
will work?
Make c
1
a little smaller than the leading
coefficient, and c
2
a little bigger.
To compare orders of growth, look at the
leading term.
Exercise: Prove that n
2
/2-3n= O(n
2
)

O(g(n)) = {f(n) : - positive constants c
1
, c
2
, and n
0
,
such that n > n
0
, 0 s c
1
g(n) s f(n) s c
2
g(n)}
asymp - 6 Comp 122
Example
Is 3n
3
e O(n
4
) ??
How about 2
2n
e O(2
n
)??

O(g(n)) = {f(n) : - positive constants c
1
, c
2
, and n
0
,
such that n > n
0
, 0 s c
1
g(n) s f(n) s c
2
g(n)}
asymp - 7 Comp 122
O-notation
O(g(n)) = {f(n) :
- positive constants c and n
0,

such that n > n
0
,
we have 0 s f(n) s cg(n) }
For function g(n), we define O(g(n)),
big-O of n, as the set:
g(n) is an asymptotic upper bound for f(n).
I ntuitively: Set of all functions
whose rate of growth is the same as
or lower than that of g(n).
f(n) = O(g(n)) f(n) = O(g(n)).
O(g(n)) c O(g(n)).
asymp - 8 Comp 122
Examples
Any linear function an + b is in O(n
2
). How?
Show that 3n
3
=O(n
4
) for appropriate c and n
0
.

O(g(n)) = {f(n) : - positive constants c and n
0
,
such that n > n
0
, we have 0 s f(n) s cg(n) }
asymp - 9 Comp 122
O -notation
g(n) is an asymptotic lower bound for f(n).
I ntuitively: Set of all functions
whose rate of growth is the same
as or higher than that of g(n).
f(n) = O(g(n)) f(n) = O(g(n)).
O(g(n)) c O(g(n)).
O(g(n)) = {f(n) :
- positive constants c and n
0,

such that n > n
0
,
we have 0 s cg(n) s f(n)}
For function g(n), we define O(g(n)),
big-Omega of n, as the set:
asymp - 10 Comp 122
Example


\n = O(lg n). Choose c and n
0
.
O(g(n)) = {f(n) : - positive constants c and n
0
, such
that n > n
0
, we have 0 s cg(n) s f(n)}
asymp - 11 Comp 122
Relations Between O, O, O
asymp - 12 Comp 122
Relations Between O, O, O
I.e., O(g(n)) = O(g(n)) O(g(n))
In practice, asymptotically tight bounds are
obtained from asymptotic upper and lower bounds.
Theorem : For any two functions g(n) and f(n),
f(n) = O(g(n)) iff
f(n) = O(g(n)) and f(n) = O(g(n)).
asymp - 13 Comp 122
Running Times
Running time is O(f(n)) Worst case is O(f(n))
O(f(n)) bound on the worst-case running time
O(f(n)) bound on the running time of every input.
O(f(n)) bound on the worst-case running time
O(f(n)) bound on the running time of every input.
Running time is O(f(n)) Best case is O(f(n))
Can still say Worst-case running time is O(f(n))
Means worst-case running time is given by some
unspecified function g(n) e O(f(n)).
asymp - 14 Comp 122
Example
I nsertion sort takes O(n
2
) in the worst case, so
sorting (as a problem) is O(n
2
). Why?
Any sort algorithm must look at each item, so
sorting is O(n).
In fact, using (e.g.) merge sort, sorting is O(n lg n)
in the worst case.
Later, we will prove that we cannot hope that any
comparison sort to do better in the worst case.
asymp - 15 Comp 122
Asymptotic Notation in Equations
Can use asymptotic notation in equations to
replace expressions containing lower-order terms.
For example,
4n
3
+ 3n
2
+ 2n + 1 = 4n
3
+ 3n
2
+ O(n)
= 4n
3
+ O(n
2
) = O(n
3
). How to interpret?
In equations, O(f(n)) always stands for an
anonymous function g(n) e O(f(n))
In the example above, O(n
2
) stands for
3n
2
+ 2n + 1.
asymp - 16 Comp 122
o-notation
f(n) becomes insignificant relative to g(n) as n
approaches infinity:
lim [f(n) / g(n)] = 0

n

g(n) is an upper bound for f(n) that is not
asymptotically tight.
Observe the difference in this definition from previous
ones. Why?
o(g(n)) = {f(n): c > 0, - n
0
> 0 such that
n >

n
0
, we have

0 s f(n) < cg(n)}.
For a given function g(n), the set little-o:
asymp - 17 Comp 122
e(g(n)) = {f(n): c > 0, - n
0
> 0 such that
n >

n
0
, we have

0 s cg(n) < f(n)}.
e -notation
f(n) becomes arbitrarily large relative to g(n) as n
approaches infinity:
lim [f(n) / g(n)] = .

n

g(n) is a lower bound for f(n) that is not
asymptotically tight.
For a given function g(n), the set little-omega:
asymp - 18 Comp 122
Comparison of Functions
f g ~ a b

f (n) = O(g(n)) ~ a s b
f (n) = O(g(n)) ~ a > b
f (n) = O(g(n)) ~ a = b
f (n) = o(g(n)) ~ a < b
f (n) = e (g(n)) ~ a > b
asymp - 19 Comp 122
Limits
lim [f(n) / g(n)] = 0 f(n) e o(g(n))

n

lim [f(n) / g(n)] < f(n) e O(g(n))

n

0 < lim [f(n) / g(n)] < f(n) e O(g(n))

n

0 < lim [f(n) / g(n)] f(n) e O(g(n))

n

lim [f(n) / g(n)] = f(n) e e(g(n))

n

lim [f(n) / g(n)] undefined cant say

n

asymp - 20 Comp 122
Properties
Transitivity
f(n) = O(g(n)) & g(n) = O(h(n)) f(n) = O(h(n))
f(n) = O(g(n)) & g(n) = O(h(n)) f(n) = O(h(n))
f(n) = O(g(n)) & g(n) = O(h(n)) f(n) = O(h(n))
f(n) = o (g(n)) & g(n) = o (h(n)) f(n) = o (h(n))
f(n) = e(g(n)) & g(n) = e(h(n)) f(n) = e(h(n))

Reflexivity
f(n) = O(f(n))
f(n) = O(f(n))
f(n) = O(f(n))


asymp - 21 Comp 122
Properties
Symmetry
f(n) = O(g(n)) iff g(n) = O(f(n))

Complementarity
f(n) = O(g(n)) iff g(n) = O(f(n))
f(n) = o(g(n)) iff g(n) = e((f(n))
6 June 2014 Comp 122, Spring 2004
Common Functions
asymp - 23 Comp 122
Monotonicity
f(n) is
monotonically increasing if m s n f(m) s f(n).
monotonically decreasing if m > n f(m) > f(n).
strictly increasing if m < n f(m) < f(n).
strictly decreasing if m > n f(m) > f(n).

asymp - 24 Comp 122
Exponentials
Useful Identities:




Exponentials and polynomials

n m n m
mn n m
a a a
a a
a
a
+

=
=
=
) (
1
1
) (
0 lim
n b
n
b
n
a o n
a
n
=
=

asymp - 25 Comp 122
Logarithms
x = log
b
a is the
exponent for a = b
x
.

Natural log: ln a = log
e
a
Binary log: lg a = log
2
a

lg
2
a = (lg a)
2
lg lg a

=

lg (lg a)


a c
a
b
b b
c
c
b
b
n
b
c c c
a
b b
b
c a
b
a
a a
b
a
a
a n a
b a ab
b a
log log
log
log
1
log
log ) / 1 ( log
log
log
log
log log
log log ) ( log
=
=
=
=
=
+ =
=
asymp - 26 Comp 122
Logarithms and exponentials Bases
If the base of a logarithm is changed from one
constant to another, the value is altered by a
constant factor.
Ex: log
10
n * log
2
10 = log
2
n.
Base of logarithm is not an issue in asymptotic
notation.
Exponentials with different bases differ by a
exponential factor (not a constant factor).
Ex: 2
n
= (2/3)
n
*3
n
.
asymp - 27 Comp 122
Polylogarithms
For a > 0, b > 0, lim
n
( lg
a
n / n
b
) = 0,
so lg
a
n = o(n
b
), and n
b
= e(lg
a
n )
Prove using LHopitals rule repeatedly

lg(n!) = O(n lg n)
Prove using Stirlings approximation (in the text) for lg(n!).
asymp - 28 Comp 122
Exercise
Express functions in A in asymptotic notation using functions in B.
A B
5n
2
+ 100n 3n
2
+ 2
A e O(n
2
), n
2
e O(B) A e O(B)
log
3
(n
2
) log
2
(n
3
)
log
b
a = log
c
a / log
c
b; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
n
lg4

3
lg n
a
log b
=

b
log a
; B =3
lg n
=n
lg 3
; A/B =n
lg(4/3)

as n
lg
2
n

n
1/2
lim ( lg
a
n / n
b
) = 0 (here a = 2 and b = 1/2) A e o (B)

n

A e O(B)
A e O(B)
A e e(B)
A e o (B)
6 June 2014 Comp 122, Spring 2004
Summations Review
asymp - 30 Comp 122
Review on Summations
Why do we need summation formulas?
For computing the running times of iterative
constructs (loops). (CLRS Appendix A)
Example: Maximum Subvector
Given an array A[1n] of numeric values (can be
positive, zero, and negative) determine the
subvector A[ij] (1s i s j s n) whose sum of
elements is maximum over all subvectors.

1 -2 2 2
asymp - 31 Comp 122
Review on Summations
MaxSubvector(A, n)
maxsum 0;
for i 1 to n
do for j = i to n
sum 0
for k i to j
do sum += A[k]
maxsum max(sum, maxsum)
return maxsum

n n j
T(n) = 1

i=1 j=i k=i
NOTE: This is not a simplified solution. What is the final answer?

asymp - 32 Comp 122
Review on Summations
Constant Series: For integers a and b, a s b,







Linear Series (Arithmetic Series): For n > 0,



Quadratic Series: For n > 0,

=
+ =
b
a i
a b 1 1
2
) 1 (
2 1
1
+
= + + + =

=
n n
n i
n
i

=
+ +
= + + + =
n
i
n n n
n i
1
2 2 2 2
6
) 1 2 )( 1 (
2 1
asymp - 33 Comp 122
Review on Summations
Cubic Series: For n > 0,




Geometric Series: For real x = 1,



For |x| < 1,

=
+
= + + + =
n
i
n n
n i
1
2 2
3 3 3 3
4
) 1 (
2 1

=
+

= + + + + =
n
k
n
n k
x
x
x x x x
0
1
2
1
1
1

=

=
0
1
1
k
k
x
x
asymp - 34 Comp 122
Review on Summations

Linear-Geometric Series: For n > 0, real c = 1,




Harmonic Series: nth harmonic number, neI
+
,

=
+ +

+ + +
= + + + =
n
i
n n
n i
c
c nc c n
nc c c ic
1
2
2 1
2
) 1 (
) 1 (
2
n
H
n
1
3
1
2
1
1 + + + + =

=
+ = =
n
k
O n
k
1
) 1 ( ) ln(
1
asymp - 35 Comp 122
Review on Summations
Telescoping Series:



Differentiating Series: For |x| < 1,

=

=
n
k
n k k
a a a a
1
0 1
( )

=

=
0
2
1
k
k
x
x
kx
asymp - 36 Comp 122
Review on Summations
Approximation by integrals:
For monotonically increasing f(n)


For monotonically decreasing f(n)


How?
}

}

=
+
s s
n
m
n
m k
n
m
dx x f k f dx x f
1
1
) ( ) ( ) (
}

}
+
=

s s
1
1
) ( ) ( ) (
n
m
n
m k
n
m
dx x f k f dx x f
asymp - 37 Comp 122
Review on Summations
nth harmonic number

}
=
+
+ = >
n
k
n
n
x
dx
k
1
1
1
) 1 ln(
1

}
=
= s
n
k
n
n
x
dx
k
2
1
ln
1

=
+ s
n
k
n
k
1
1 ln
1
asymp - 38 Comp 122
Reading Assignment
Chapter 4 of CLRS.