Analysis & Design of

Algorithms
(CSCE 2202)

Prof. Amr Goneid

Department of Computer Science, AUC

Part 2. Types of Complexities

Prof. Amr Goneid, AUC 1

Types of Complexities

Prof. Amr Goneid, AUC 2

Types of Complexities
 Rules for Upper Bound
 Comparing Complexities
 Types of Complexities

Prof. Amr Goneid, AUC 3

1. Rules for Upper Bound
 If (k) is a constant, then:
O(k)  O(n) and
O(k f(n)) = O(f(n))
 e.g. an2 and bn2 are T(n) O(kn)
both O(n2) O(n)
O(k)

n
Prof. Amr Goneid, AUC 4
Rules for Big O
 The growth rate of a sum of terms of is the
growth rate of its fastest growing term
O(f(n)) + O(g(n)) = max(O(f(n)) ,O(g(n)))
 e.g. f(n) = 2n = O(n), g(n) = 0.1 n3 = O(n3)
T(n) = max(O(n) , O(n3)) = O(n3)

2n
O(n3)
0.1 n3

Prof. Amr Goneid, AUC 5

Rules for Big O
 The product of big-O’s is the big-O of the
products:
O(f(n)) * O(g(n)) = O(f(n) * g(n))
 e.g. T1(n) = O(n), T2(n) = O(n2)
T(n) = T1 (n) * T2 (n) = O(n3)
Repeat n times
O(n2) O(n3)

Prof. Amr Goneid, AUC 6

Rules for Big O
 Logarithms grow slower than powers:
O(loga n)  O(nk) for all a > 1 and k > 0

 e.g. O(log n) < O(n)

 Claim:
For all n ≥ 1, log n < n.
Prove by induction on n.

Prof. Amr Goneid, AUC 7

Rules for Big O
 All logarithms grow at the same rate:
loga n =  (logb n) for all a , b > 1

 e.g. log2 n =  (log3 n)

 Proof:

n  b logb n
loga n  loga b logb n  (loga b )logb n  c logb n   (logb n )

Prof. Amr Goneid, AUC 8

Rules for Big O
 For a polynomial of degree m,
2 m
T ( n )  Pm ( n )  a0  a1n  a 2 n  ...  a m n
m
Then T ( n )  O( n )
2 3 3
e.g.T ( n )  2  4n  3n  2n  O( n )
 Prove!
 O(nm-1)  O(nm) follows from above

Prof. Amr Goneid, AUC 9

Rules for Big O
 Exponential functions grow faster than powers
O(nk) < O(bn) for all k ≥ 0 and b > 1

 e.g. O(n3) < O(2n)

 Proof:
log n k  k log n and log b n  n log b
Since log n  n then
O( n k )  O( b n ) for all k  0 and b  1

Prof. Amr Goneid, AUC 10

Summary of Rules for Big-O
Rule Example
For constant k, O(k) < O(n) O(7) = O(1) < O(n)

For constant k, O(kf) = O(f) O(2n) = O(n)

O(|f|+|g|) = O(|f|) + O(|g|) =

O(6n2+n) = O(6n2) + O(n) = O(n2)
Max (O(|f|) , O(|g|)
Nesting of loop O(g) within a
O(n4 * n2) = O(n6)
loop O(f) gives O(f * g)

O(nm-1) < O(nm) O(n2) < O(n3)

O((log n)k) < O(n) O(log n) < O(n)

Prof. Amr Goneid, AUC 11

Exercises
1. Prove by induction on n that n2 < 2n
2. Prove by induction on n that 2n < n! for n ≥ 4
3. Prove by induction on n that for n > 0
n
S ( n)   ( 2i  1)  n
i 1
2

n
S ( n)   i  n( n  1) / 2
i 1

Prof. Amr Goneid, AUC 12

2. Comparing Complexities
Dominance:
 If lim(n->) f(n)/g(n) = 
then f(n) dominates (i.e. grows faster)
 If lim(n->) f(n)/g(n) = 0
then g(n) dominates.
 If lim(n->) f(n)/g(n) = constant
then both grow at the same rate.

Prof. Amr Goneid, AUC 13

Comparing Complexities
Examples:
 if a > b then na dominates nb
Why?
 n2 dominates (3n+2)
Why?
 n2 dominates (n log n)
Why?

Prof. Amr Goneid, AUC 14

Using l’Hopital’s Rule (1696)
If lim f ( n )   and lim g( n )  
n  n 

f (n) f '(n )
then lim  lim '
n  g( n ) n  g ( n )

d
where the prime means
dn
Example: Show that
f(n) = n(k+α) + nk (log n)2 and
g(n) = k n(k+α)
grow at the same rate.

Prof. Amr Goneid, AUC 15

Comparing Complexities
Which grows faster:
 f(n) = n3 or g(n) = n log n
 f(n) = n 0.001 or g(n) = log n
 f(n) = 2n+1 or g(n) = 2n
 f(n) = 2n or g(n) = 22n

Prof. Amr Goneid, AUC 16

Exercises
Which function has smaller complexity ?
 f = 100 n4 g = n5
 f = log(log n3) g = log n
 f = n2 g = n log n
 f = 50 n5 + n2 + n g = n5
 f = en g = n!

Prof. Amr Goneid, AUC 17

3. Types of Complexities
 Constant Complexity
 T(n) = constant independent of (n)
 Runs in constant amount of time  O(1)
 Example: cout << a[0][0]

Prof. Amr Goneid, AUC 18

Types of Complexities
 Logarithmic Complexity
 Log2n = m is equivalent to n=2m
 Reduces the problem to half  O(log2n)
 Example: Binary Search
T(n) = O(log2n)
Much faster than Linear Search which has
T(n) = O(n)

Prof. Amr Goneid, AUC 19

Linear vs Logarithmic Complexities
16 T(n)
14

12

10 O(n)
x
8
logx

4
O(log2n)
2

0
n
1 2 3 4 5 6 7 8 9 10 11 12 13 14

Prof. Amr Goneid, AUC 20

Types of Complexities
Polynomial Complexity
 T(n) = amnm+…+ a2n2 + a1n1 + a0
 If m=1, then O(a n+a )  O(n)
1 0
 If m > 1, then  O(nm) as nm dominates

Prof. Amr Goneid, AUC 21

Polynomials Complexities
O(n3)

O(n2)
Log T(n)

O(n)

Prof. Amr Goneid, AUC 22

Types of Complexities
 Exponential
 Example: List all the subsets of a set of n
elements {a,b,c}
{a,b,c}, {a,b},{a,c},{b,c},{a},{b},{c},{}
 Number of operations T(n) = O(2n)
 Exponential expansion of the problem 
O(an) where a is a constant greater than 1

Prof. Amr Goneid, AUC 23

Exponential VS Polynomial

O(2n)
Log T(n)

O(n3)

O(n)
n

Prof. Amr Goneid, AUC 24

Types of Complexities
 Factorial time Algorithms
 Example:
Traveling Salesperson Problem (TSP):
Find the best route to take in visiting n
cities away from home. What are the
number of possible routes? For 3 cities:
(A,B,C)

Prof. Amr Goneid, AUC 25

Possible routes in a TSP
Hom e

Amsterdam NY M ontreal

NY M ontreal Amsterdam M ontreal Amsterdam NY

M ontreal NY M ontreal Am sterdam NY Am sterdam

Number of operations = 3!=6, Hence T(n) = n!
Expansion of the problem  O(n!)

Prof. Amr Goneid, AUC 26

Exponential Vs Factorial

O(nn)
Log T(n)

O(n!)

O(2n)

Prof. Amr Goneid, AUC 27

Execution Time Example
 Example:
 For the exponential algorithm of listing all
subsets of a given set, assume the set size
to be of 1024 elements
 Number of operations is 21024 about
1.8*10308
 If we can list a subset every nanosecond
the process will take 5.7 * 10291 yr!!!

Prof. Amr Goneid, AUC 28

P and NP – Times
 P (Polynomial) Times O(n) ,  ≥ 0:
O(1), O(log n), O(log n)2, O(n), O(n log n),
O(n2), O(n3), ….

 NP (Non-Polynomial) Times:
O(2n) , O(en) , O(n!) , O(nn) , …..
Intractable Algorithms
Prof. Amr Goneid, AUC 29
P and NP – Times
 Polynomial time is

GOOD
Try to reduce the polynomial
power
 NP (e.g. Exponential) Time is

BAD
Prof. Amr Goneid, AUC 30